langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-27 03:01:54 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

When the vector library switches to milvus: Fail connecting to server on 127.0.0.1:19530 #4307

New Issue
Closed
opened 2026-02-21 18:05:41 -05:00 by yindo · 4 comments
yindo commented 2026-02-21 18:05:41 -05:00
Owner
Copy Link
Copy Source

Originally created by @0sengseng0 on GitHub (Jun 28, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.11

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

When I switched the vector library to milvus, I added parts of the docker-compose.milvus.yaml file to docker-compose.yaml and removed the weaviate part of docker-compose.yaml. When I vectorized the document, I got an error: Fail connecting to server on 127.0.0.1:19530

Also, I was able to open the milvus web page and successfully create the collection.
image

docker-compose.yaml

services:
api:
image: langgenius/dify-api:0.6.11
restart: always
environment:
***
VECTOR_STORE: milvus
MILVUS_HOST: 127.0.0.1
MILVUS_PORT: 19530
MILVUS_USER: root
MILVUS_PASSWORD: Milvus
MILVUS_SECURE: 'false'
***
worker:
image: langgenius/dify-api:0.6.11
restart: always
environment:
****
VECTOR_STORE: milvus
MILVUS_HOST: 127.0.0.1
MILVUS_PORT: 19530
MILVUS_USER: root
MILVUS_PASSWORD: Milvus
MILVUS_SECURE: 'false'

etcd:
container_name: milvus-etcd
image: quay.io/coreos/etcd:v3.5.5
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
healthcheck:
test: ["CMD", "etcdctl", "endpoint", "health"]
interval: 30s
timeout: 20s
retries: 3

minio:
container_name: milvus-minio
image: minio/minio:RELEASE.2023-03-20T20-16-18Z
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
ports:
- "9001:9001"
- "9000:9000"
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
command: minio server /minio_data --console-address ":9001"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3

milvus-standalone:
container_name: milvus-standalone
image: milvusdb/milvus:v2.3.1
command: ["milvus", "run", "standalone"]
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
common.security.authorizationEnabled: false
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
interval: 30s
start_period: 90s
timeout: 20s
retries: 3
ports:
- "19530:19530"
- "9091:9091"
depends_on:
- "etcd"
- "minio"

Err log - MILVUS

[2024/06/28 08:28:28.235 +00:00] [INFO] [rootcoord/root_coord.go:1130] ["done to describe collection"] [traceID=b92cdf574d50e672ee60f4ced62e626e] [collectionName=t1] [dbName=default] [id=0] [ts=18446744073709551615] [allowUnavailable=false] [collection_id=450770507019846953]
[2024/06/28 08:28:28.235 +00:00] [INFO] [datacoord/index_service.go:521] ["receive DescribeIndex request"] [traceID=b92cdf574d50e672ee60f4ced62e626e] [collectionID=450770507019846953] [indexName=] [timestamp=0]
[2024/06/28 08:28:28.236 +00:00] [WARN] [datacoord/index_service.go:539] ["DescribeIndex fail"] [traceID=b92cdf574d50e672ee60f4ced62e626e] [collectionID=450770507019846953] [indexName=] ["fail reason"="there is no index on collection: 450770507019846953 with the index name: "]
[2024/06/28 08:28:28.236 +00:00] [WARN] [proxy/task_scheduler.go:460] ["Failed to execute task: "] [traceID=b92cdf574d50e672ee60f4ced62e626e] [error="index doesn't exist, collectionID 450770507019846953"] [errorVerbose="index doesn't exist, collectionID 450770507019846953\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*describeIndexTask).Execute\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_index.go:497\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:457\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).definitionLoop\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:484\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) index doesn't exist, collectionID 450770507019846953\nError types: (1) *withstack.withStack (2) *errutil.leafError"]
[2024/06/28 08:28:28.236 +00:00] [WARN] [proxy/impl.go:1769] ["DescribeIndex failed to WaitToFinish"] [traceID=b92cdf574d50e672ee60f4ced62e626e] [role=proxy] [db=default] [collection=t1] [field=] ["index name"=] [error="index doesn't exist, collectionID 450770507019846953"] [errorVerbose="index doesn't exist, collectionID 450770507019846953\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*describeIndexTask).Execute\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_index.go:497\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:457\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).definitionLoop\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:484\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) index doesn't exist, collectionID 450770507019846953\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [BeginTs=450773203870285833] [EndTs=450773203870285833]
[2024/06/28 08:28:28.240 +00:00] [INFO] [rootcoord/root_coord.go:1420] ["received request to show partitions"] [traceID=fe472c50351b961347b41b2af3d08e06] [collection=t1] [collection_id=0] [partitions="[]"] [allowUnavailable=false]
[2024/06/28 08:28:28.241 +00:00] [INFO] [rootcoord/root_coord.go:1451] ["done to show partitions"] [traceID=fe472c50351b961347b41b2af3d08e06] [collection=t1] [collection_id=0] [partitions="[]"] [allowUnavailable=false] [partitions="[_default]"]
[2024/06/28 08:28:28.241 +00:00] [INFO] [proxy/meta_cache.go:791] ["no shard cache for collection, try to get shard leaders from QueryCoord"] [traceID=fe472c50351b961347b41b2af3d08e06] [collectionName=t1] [collectionID=450770507019846953]
[2024/06/28 08:28:28.242 +00:00] [INFO] [querycoordv2/services.go:840] ["get shard leaders request received"] [traceID=fe472c50351b961347b41b2af3d08e06] [collectionID=450770507019846953]
[2024/06/28 08:28:28.242 +00:00] [WARN] [querycoordv2/services.go:856] ["failed to GetShardLeaders"] [traceID=fe472c50351b961347b41b2af3d08e06] [collectionID=450770507019846953] [error="collection=450770507019846953: collection not loaded"] [errorVerbose="collection=450770507019846953: collection not loaded\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.wrapWithField\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:531\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrCollectionNotLoaded\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:258\n | github.com/milvus-io/milvus/internal/querycoordv2.(*Server).GetShardLeaders\n | \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/services.go:855\n | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).GetShardLeaders\n | \t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:384\n | github.com/milvus-io/milvus/internal/proto/querypb._QueryCoord_GetShardLeaders_Handler.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/proto/querypb/query_coord.pb.go:5555\n | github.com/milvus-io/milvus/pkg/util/interceptor.ServerIDValidationUnaryServerInterceptor.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/interceptor/server_id_interceptor.go:54\n | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n | github.com/milvus-io/milvus/pkg/util/interceptor.ClusterValidationUnaryServerInterceptor.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/interceptor/cluster_interceptor.go:48\n | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n | github.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/grpc_interceptor.go:23\n | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n | go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n | \t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.38.0/interceptor.go:342\n | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\n | github.com/milvus-io/milvus/internal/proto/querypb._QueryCoord_GetShardLeaders_Handler\n | \t/go/src/github.com/milvus-io/milvus/internal/proto/querypb/query_coord.pb.go:5557\n | google.golang.org/grpc.(*Server).processUnaryRPC\n | \t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345\n | google.golang.org/grpc.(*Server).handleStream\n | \t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722\n | google.golang.org/grpc.(*Server).serveStreams.func1.2\n | \t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) collection=450770507019846953\nWraps: (3) collection not loaded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/06/28 08:28:28.242 +00:00] [ERROR] [retry/retry.go:42] ["retry func failed"] [traceID=fe472c50351b961347b41b2af3d08e06] ["retry time"=0] [error="fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error"] [stack="github.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/proxy.(*MetaCache).GetShards\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/meta_cache.go:806\ngithub.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:194\ngithub.com/milvus-io/milvus/internal/proxy.(*queryTask).Execute\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:408\ngithub.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:457"]
[2024/06/28 08:28:28.243 +00:00] [WARN] [proxy/lb_policy.go:196] ["failed to get shards"] [traceID=fe472c50351b961347b41b2af3d08e06] [error="attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error"]
[2024/06/28 08:28:28.243 +00:00] [WARN] [proxy/task_query.go:416] ["fail to execute query"] [traceID=fe472c50351b961347b41b2af3d08e06] [collection=450770507019846953] [partitionIDs="[]"] [requestType=query] [error="attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error"]
[2024/06/28 08:28:28.243 +00:00] [WARN] [proxy/task_scheduler.go:460] ["Failed to execute task: "] [traceID=fe472c50351b961347b41b2af3d08e06] [error="failed to query: attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error"] [errorVerbose="failed to query: attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*queryTask).Execute\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:417\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:457\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) failed to query\nWraps: (3) attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.multiErrors"]
[2024/06/28 08:28:28.244 +00:00] [WARN] [proxy/impl.go:2706] ["Query failed to WaitToFinish"] [traceID=fe472c50351b961347b41b2af3d08e06] [role=proxy] [db=default] [collection=t1] [partitions="[]"] [error="failed to query: attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error"] [errorVerbose="failed to query: attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*queryTask).Execute\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:417\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:457\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) failed to query\nWraps: (3) attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.multiErrors"]
[2024/06/28 08:28:28.250 +00:00] [INFO] [datacoord/services.go:311] ["received request to get collection statistics"] [traceID=c97290dc9f398ea2c906606ba2a98039] [collectionID=450770507019846953]
[2024/06/28 08:28:28.250 +00:00] [INFO] [datacoord/services.go:324] ["success to get collection statistics"] [traceID=c97290dc9f398ea2c906606ba2a98039] [collectionID=450770507019846953] [response="stats:<key:"row_count" value:"0" > status:<> "]
[2024/06/28 08:28:30.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"]
[2024/06/28 08:28:37.444 +00:00] [INFO] [gc/gc_tuner.go:90] ["GC Tune done"] ["previous GOGC"=200] ["heapuse "=63] ["total memory"=525] ["next GC"=71] ["new GOGC"=200] [gc-pause=257.945µs] [gc-pause-end=1719563317439625899]
[2024/06/28 08:28:40.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"]
[2024/06/28 08:28:41.244 +00:00] [INFO] [datacoord/channel_manager.go:281] ["ChannelManager is not silent, skip channel balance this round"]
[2024/06/28 08:28:42.833 +00:00] [INFO] [datacoord/meta.go:1284] ["UpdateChannelCheckpoint done"] [vChannel=by-dev-rootcoord-dml_0_450770507019846953v0] [ts=450773207658266625] [msgID="\u0005\u0006Ph\ufffdwA\u0006"] [time=2024/06/28 08:28:42.671 +00:00]
[2024/06/28 08:28:42.833 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:145] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_0_450770507019846953v0] [cpTs=450773207658266625] [cpTime=2024/06/28 08:28:42.671 +00:00]
[2024/06/28 08:28:50.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"]
[2024/06/28 08:29:00.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"]
[2024/06/28 08:29:10.418 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"]
[2024/06/28 08:29:11.130 +00:00] [INFO] [gc/gc_tuner.go:90] ["GC Tune done"] ["previous GOGC"=200] ["heapuse "=66] ["total memory"=528] ["next GC"=70] ["new GOGC"=200] [gc-pause=280.14µs] [gc-pause-end=1719563351124284505]
[2024/06/28 08:29:20.418 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"]
[2024/06/28 08:29:30.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"]
[2024/06/28 08:29:40.418 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"]
[2024/06/28 08:29:43.025 +00:00] [INFO] [datacoord/meta.go:1284] ["UpdateChannelCheckpoint done"] [vChannel=by-dev-rootcoord-dml_0_450770507019846953v0] [ts=450773223439073281] [msgID="2\u0007Ph\ufffdwA\u0006"] [time=2024/06/28 08:29:42.870 +00:00]
[2024/06/28 08:29:43.026 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:145] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_0_450770507019846953v0] [cpTs=450773223439073281] [cpTime=2024/06/28 08:29:42.870 +00:00]
[2024/06/28 08:29:44.430 +00:00] [INFO] [gc/gc_tuner.go:90] ["GC Tune done"] ["previous GOGC"=200] ["heapuse "=66] ["total memory"=536] ["next GC"=70] ["new GOGC"=200] [gc-pause=1.308523ms] [gc-pause-end=1719563384425332667]
[2024/06/28 08:29:50.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"]
[2024/06/28 08:30:00.418 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"]
[2024/06/28 08:30:10.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"]
[2024/06/28 08:30:17.231 +00:00] [INFO] [gc/gc_tuner.go:90] ["GC Tune done"] ["previous GOGC"=200] ["heapuse "=65] ["total memory"=537] ["next GC"=70] ["new GOGC"=200] [gc-pause=1.403646ms] [gc-pause-end=1719563417225867954]
[2024/06/28 08:30:20.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"]

✔️ Expected Behavior

I was able to use milvus successfully.

❌ Actual Behavior

milvus works fine on its own, but when dify vectorizes the document, it throws an error: Fail connecting to server on 127.0.0.1:19530

Originally created by @0sengseng0 on GitHub (Jun 28, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.6.11 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce When I switched the vector library to milvus, I added parts of the docker-compose.milvus.yaml file to docker-compose.yaml and removed the weaviate part of docker-compose.yaml. When I vectorized the document, I got an error: Fail connecting to server on 127.0.0.1:19530 Also, I was able to open the milvus web page and successfully create the collection. ![image](https://github.com/langgenius/dify/assets/73268510/b0ad51af-32f1-4ad8-bf3a-35f823bb36c8) # docker-compose.yaml services: api: image: langgenius/dify-api:0.6.11 restart: always environment: *** VECTOR_STORE: milvus MILVUS_HOST: 127.0.0.1 MILVUS_PORT: 19530 MILVUS_USER: root MILVUS_PASSWORD: Milvus MILVUS_SECURE: 'false' *** worker: image: langgenius/dify-api:0.6.11 restart: always environment: **** VECTOR_STORE: milvus MILVUS_HOST: 127.0.0.1 MILVUS_PORT: 19530 MILVUS_USER: root MILVUS_PASSWORD: Milvus MILVUS_SECURE: 'false' **** etcd: container_name: milvus-etcd image: quay.io/coreos/etcd:v3.5.5 environment: - ETCD_AUTO_COMPACTION_MODE=revision - ETCD_AUTO_COMPACTION_RETENTION=1000 - ETCD_QUOTA_BACKEND_BYTES=4294967296 - ETCD_SNAPSHOT_COUNT=50000 volumes: - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd healthcheck: test: ["CMD", "etcdctl", "endpoint", "health"] interval: 30s timeout: 20s retries: 3 minio: container_name: milvus-minio image: minio/minio:RELEASE.2023-03-20T20-16-18Z environment: MINIO_ACCESS_KEY: minioadmin MINIO_SECRET_KEY: minioadmin ports: - "9001:9001" - "9000:9000" volumes: - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data command: minio server /minio_data --console-address ":9001" healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 milvus-standalone: container_name: milvus-standalone image: milvusdb/milvus:v2.3.1 command: ["milvus", "run", "standalone"] environment: ETCD_ENDPOINTS: etcd:2379 MINIO_ADDRESS: minio:9000 common.security.authorizationEnabled: false volumes: - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"] interval: 30s start_period: 90s timeout: 20s retries: 3 ports: - "19530:19530" - "9091:9091" depends_on: - "etcd" - "minio" # Err log - MILVUS [2024/06/28 08:28:28.235 +00:00] [INFO] [rootcoord/root_coord.go:1130] ["done to describe collection"] [traceID=b92cdf574d50e672ee60f4ced62e626e] [collectionName=t1] [dbName=default] [id=0] [ts=18446744073709551615] [allowUnavailable=false] [collection_id=450770507019846953] [2024/06/28 08:28:28.235 +00:00] [INFO] [datacoord/index_service.go:521] ["receive DescribeIndex request"] [traceID=b92cdf574d50e672ee60f4ced62e626e] [collectionID=450770507019846953] [indexName=] [timestamp=0] [2024/06/28 08:28:28.236 +00:00] [WARN] [datacoord/index_service.go:539] ["DescribeIndex fail"] [traceID=b92cdf574d50e672ee60f4ced62e626e] [collectionID=450770507019846953] [indexName=] ["fail reason"="there is no index on collection: 450770507019846953 with the index name: "] [2024/06/28 08:28:28.236 +00:00] [WARN] [proxy/task_scheduler.go:460] ["Failed to execute task: "] [traceID=b92cdf574d50e672ee60f4ced62e626e] [error="index doesn't exist, collectionID 450770507019846953"] [errorVerbose="index doesn't exist, collectionID 450770507019846953\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*describeIndexTask).Execute\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_index.go:497\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:457\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).definitionLoop\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:484\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) index doesn't exist, collectionID 450770507019846953\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [2024/06/28 08:28:28.236 +00:00] [WARN] [proxy/impl.go:1769] ["DescribeIndex failed to WaitToFinish"] [traceID=b92cdf574d50e672ee60f4ced62e626e] [role=proxy] [db=default] [collection=t1] [field=] ["index name"=] [error="index doesn't exist, collectionID 450770507019846953"] [errorVerbose="index doesn't exist, collectionID 450770507019846953\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*describeIndexTask).Execute\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_index.go:497\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:457\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).definitionLoop\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:484\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) index doesn't exist, collectionID 450770507019846953\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [BeginTs=450773203870285833] [EndTs=450773203870285833] [2024/06/28 08:28:28.240 +00:00] [INFO] [rootcoord/root_coord.go:1420] ["received request to show partitions"] [traceID=fe472c50351b961347b41b2af3d08e06] [collection=t1] [collection_id=0] [partitions="[]"] [allowUnavailable=false] [2024/06/28 08:28:28.241 +00:00] [INFO] [rootcoord/root_coord.go:1451] ["done to show partitions"] [traceID=fe472c50351b961347b41b2af3d08e06] [collection=t1] [collection_id=0] [partitions="[]"] [allowUnavailable=false] [partitions="[_default]"] [2024/06/28 08:28:28.241 +00:00] [INFO] [proxy/meta_cache.go:791] ["no shard cache for collection, try to get shard leaders from QueryCoord"] [traceID=fe472c50351b961347b41b2af3d08e06] [collectionName=t1] [collectionID=450770507019846953] [2024/06/28 08:28:28.242 +00:00] [INFO] [querycoordv2/services.go:840] ["get shard leaders request received"] [traceID=fe472c50351b961347b41b2af3d08e06] [collectionID=450770507019846953] [2024/06/28 08:28:28.242 +00:00] [WARN] [querycoordv2/services.go:856] ["failed to GetShardLeaders"] [traceID=fe472c50351b961347b41b2af3d08e06] [collectionID=450770507019846953] [error="collection=450770507019846953: collection not loaded"] [errorVerbose="collection=450770507019846953: collection not loaded\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.wrapWithField\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:531\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrCollectionNotLoaded\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:258\n | github.com/milvus-io/milvus/internal/querycoordv2.(*Server).GetShardLeaders\n | \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/services.go:855\n | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).GetShardLeaders\n | \t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:384\n | github.com/milvus-io/milvus/internal/proto/querypb._QueryCoord_GetShardLeaders_Handler.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/proto/querypb/query_coord.pb.go:5555\n | github.com/milvus-io/milvus/pkg/util/interceptor.ServerIDValidationUnaryServerInterceptor.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/interceptor/server_id_interceptor.go:54\n | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n | github.com/milvus-io/milvus/pkg/util/interceptor.ClusterValidationUnaryServerInterceptor.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/interceptor/cluster_interceptor.go:48\n | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n | github.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/grpc_interceptor.go:23\n | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n | go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n | \t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.38.0/interceptor.go:342\n | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\n | github.com/milvus-io/milvus/internal/proto/querypb._QueryCoord_GetShardLeaders_Handler\n | \t/go/src/github.com/milvus-io/milvus/internal/proto/querypb/query_coord.pb.go:5557\n | google.golang.org/grpc.(*Server).processUnaryRPC\n | \t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345\n | google.golang.org/grpc.(*Server).handleStream\n | \t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722\n | google.golang.org/grpc.(*Server).serveStreams.func1.2\n | \t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) collection=450770507019846953\nWraps: (3) collection not loaded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [2024/06/28 08:28:28.242 +00:00] [ERROR] [retry/retry.go:42] ["retry func failed"] [traceID=fe472c50351b961347b41b2af3d08e06] ["retry time"=0] [error="fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error"] [stack="github.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/proxy.(*MetaCache).GetShards\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/meta_cache.go:806\ngithub.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:194\ngithub.com/milvus-io/milvus/internal/proxy.(*queryTask).Execute\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:408\ngithub.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:457"] [2024/06/28 08:28:28.243 +00:00] [WARN] [proxy/lb_policy.go:196] ["failed to get shards"] [traceID=fe472c50351b961347b41b2af3d08e06] [error="attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error"] [2024/06/28 08:28:28.243 +00:00] [WARN] [proxy/task_query.go:416] ["fail to execute query"] [traceID=fe472c50351b961347b41b2af3d08e06] [collection=450770507019846953] [partitionIDs="[]"] [requestType=query] [error="attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error"] [2024/06/28 08:28:28.243 +00:00] [WARN] [proxy/task_scheduler.go:460] ["Failed to execute task: "] [traceID=fe472c50351b961347b41b2af3d08e06] [error="failed to query: attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error"] [errorVerbose="failed to query: attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*queryTask).Execute\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:417\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:457\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) failed to query\nWraps: (3) attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.multiErrors"] [2024/06/28 08:28:28.244 +00:00] [WARN] [proxy/impl.go:2706] ["Query failed to WaitToFinish"] [traceID=fe472c50351b961347b41b2af3d08e06] [role=proxy] [db=default] [collection=t1] [partitions="[]"] [error="failed to query: attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error"] [errorVerbose="failed to query: attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*queryTask).Execute\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:417\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:457\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) failed to query\nWraps: (3) attempt #0: fail to get shard leaders from QueryCoord: collection=450770507019846953: collection not loaded: unrecoverable error\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.multiErrors"] [2024/06/28 08:28:28.250 +00:00] [INFO] [datacoord/services.go:311] ["received request to get collection statistics"] [traceID=c97290dc9f398ea2c906606ba2a98039] [collectionID=450770507019846953] [2024/06/28 08:28:28.250 +00:00] [INFO] [datacoord/services.go:324] ["success to get collection statistics"] [traceID=c97290dc9f398ea2c906606ba2a98039] [collectionID=450770507019846953] [response="stats:<key:\"row_count\" value:\"0\" > status:<> "] [2024/06/28 08:28:30.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"] [2024/06/28 08:28:37.444 +00:00] [INFO] [gc/gc_tuner.go:90] ["GC Tune done"] ["previous GOGC"=200] ["heapuse "=63] ["total memory"=525] ["next GC"=71] ["new GOGC"=200] [gc-pause=257.945µs] [gc-pause-end=1719563317439625899] [2024/06/28 08:28:40.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"] [2024/06/28 08:28:41.244 +00:00] [INFO] [datacoord/channel_manager.go:281] ["ChannelManager is not silent, skip channel balance this round"] [2024/06/28 08:28:42.833 +00:00] [INFO] [datacoord/meta.go:1284] ["UpdateChannelCheckpoint done"] [vChannel=by-dev-rootcoord-dml_0_450770507019846953v0] [ts=450773207658266625] [msgID="\u0005\u0006Ph\ufffdwA\u0006"] [time=2024/06/28 08:28:42.671 +00:00] [2024/06/28 08:28:42.833 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:145] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_0_450770507019846953v0] [cpTs=450773207658266625] [cpTime=2024/06/28 08:28:42.671 +00:00] [2024/06/28 08:28:50.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"] [2024/06/28 08:29:00.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"] [2024/06/28 08:29:10.418 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"] [2024/06/28 08:29:11.130 +00:00] [INFO] [gc/gc_tuner.go:90] ["GC Tune done"] ["previous GOGC"=200] ["heapuse "=66] ["total memory"=528] ["next GC"=70] ["new GOGC"=200] [gc-pause=280.14µs] [gc-pause-end=1719563351124284505] [2024/06/28 08:29:20.418 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"] [2024/06/28 08:29:30.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"] [2024/06/28 08:29:40.418 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"] [2024/06/28 08:29:43.025 +00:00] [INFO] [datacoord/meta.go:1284] ["UpdateChannelCheckpoint done"] [vChannel=by-dev-rootcoord-dml_0_450770507019846953v0] [ts=450773223439073281] [msgID="2\u0007Ph\ufffdwA\u0006"] [time=2024/06/28 08:29:42.870 +00:00] [2024/06/28 08:29:43.026 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:145] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_0_450770507019846953v0] [cpTs=450773223439073281] [cpTime=2024/06/28 08:29:42.870 +00:00] [2024/06/28 08:29:44.430 +00:00] [INFO] [gc/gc_tuner.go:90] ["GC Tune done"] ["previous GOGC"=200] ["heapuse "=66] ["total memory"=536] ["next GC"=70] ["new GOGC"=200] [gc-pause=1.308523ms] [gc-pause-end=1719563384425332667] [2024/06/28 08:29:50.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"] [2024/06/28 08:30:00.418 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"] [2024/06/28 08:30:10.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"] [2024/06/28 08:30:17.231 +00:00] [INFO] [gc/gc_tuner.go:90] ["GC Tune done"] ["previous GOGC"=200] ["heapuse "=65] ["total memory"=537] ["next GC"=70] ["new GOGC"=200] [gc-pause=1.403646ms] [gc-pause-end=1719563417225867954] [2024/06/28 08:30:20.417 +00:00] [INFO] [proxy/meta_cache.go:905] ["expire all shard leader cache"] [database=default] [collections="[t1]"] ### ✔️ Expected Behavior I was able to use milvus successfully. ### ❌ Actual Behavior milvus works fine on its own, but when dify vectorizes the document, it throws an error: Fail connecting to server on 127.0.0.1:19530
yindo added the 👻 feat:rag label 2026-02-21 18:05:41 -05:00
yindo closed this issue 2026-02-21 18:05:41 -05:00
yindo commented 2026-02-21 18:05:41 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jun 28, 2024):

To resolve the "Fail connecting to server on 127.0.0.1:19530" error when switching to Milvus in your self-hosted Dify setup, you need to ensure that the server address configured by Dify uses the host LAN IP address instead of 127.0.0.1. This is because 127.0.0.1 is the internal address of the container and not accessible from outside the container.

Steps to Resolve:

  1. Modify the Environment Variable:
    Update the VECTOR_STORE environment variable in the docker-compose.yaml file to use Milvus.

    # The type of vector store to use. Supported values are `weaviate`, `qdrant`, `milvus`.
VECTOR_STORE: milvus

  2. Update the Server Address:
    Ensure that the server address for Milvus is set to the host LAN IP address instead of 127.0.0.1.

  3. Update the Docker Compose Configuration:
    Make sure your docker-compose.milvus.yaml file is correctly configured. Here is an example configuration:

    version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  milvus-standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.3.1
    command: ["milvus", "run", "standalone"]
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
      common.security.authorizationEnabled: true
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"

networks:
  default:
    name: milvus

  4. Update Milvus Configuration in Code:
    Ensure that the Milvus configuration in your code is set to use the correct host and port. For example, in your test file test_milvus.py, the configuration should look like this:

    self.vector = MilvusVector(
    collection_name=self.collection_name,
    config=MilvusConfig(
        host='localhost',  # Change this to the host LAN IP if necessary
        port=19530,
        user='root',
        password='Milvus',
    )
)

  5. Run the Migration Command:
    Execute the migration command to migrate the data to Milvus.

    flask vdb-migrarte # or docker exec -it docker-api-1 flask vdb-migrarte

By following these steps, you should be able to resolve the connection error and successfully switch to Milvus in your self-hosted Dify setup [1][2][3].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Jun 28, 2024): <!-- Answer --> To resolve the "Fail connecting to server on 127.0.0.1:19530" error when switching to Milvus in your self-hosted Dify setup, you need to ensure that the server address configured by Dify uses the host LAN IP address instead of `127.0.0.1`. This is because `127.0.0.1` is the internal address of the container and not accessible from outside the container. ### Steps to Resolve: 1. **Modify the Environment Variable:** Update the `VECTOR_STORE` environment variable in the `docker-compose.yaml` file to use Milvus. ```yaml # The type of vector store to use. Supported values are `weaviate`, `qdrant`, `milvus`. VECTOR_STORE: milvus ``` 2. **Update the Server Address:** Ensure that the server address for Milvus is set to the host LAN IP address instead of `127.0.0.1`. 3. **Update the Docker Compose Configuration:** Make sure your `docker-compose.milvus.yaml` file is correctly configured. Here is an example configuration: ```yaml version: '3.5' services: etcd: container_name: milvus-etcd image: quay.io/coreos/etcd:v3.5.5 environment: - ETCD_AUTO_COMPACTION_MODE=revision - ETCD_AUTO_COMPACTION_RETENTION=1000 - ETCD_QUOTA_BACKEND_BYTES=4294967296 - ETCD_SNAPSHOT_COUNT=50000 volumes: - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd healthcheck: test: ["CMD", "etcdctl", "endpoint", "health"] interval: 30s timeout: 20s retries: 3 minio: container_name: milvus-minio image: minio/minio:RELEASE.2023-03-20T20-16-18Z environment: MINIO_ACCESS_KEY: minioadmin MINIO_SECRET_KEY: minioadmin ports: - "9001:9001" - "9000:9000" volumes: - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data command: minio server /minio_data --console-address ":9001" healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 milvus-standalone: container_name: milvus-standalone image: milvusdb/milvus:v2.3.1 command: ["milvus", "run", "standalone"] environment: ETCD_ENDPOINTS: etcd:2379 MINIO_ADDRESS: minio:9000 common.security.authorizationEnabled: true volumes: - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"] interval: 30s start_period: 90s timeout: 20s retries: 3 ports: - "19530:19530" - "9091:9091" depends_on: - "etcd" - "minio" networks: default: name: milvus ``` 4. **Update Milvus Configuration in Code:** Ensure that the Milvus configuration in your code is set to use the correct host and port. For example, in your test file `test_milvus.py`, the configuration should look like this: ```py self.vector = MilvusVector( collection_name=self.collection_name, config=MilvusConfig( host='localhost', # Change this to the host LAN IP if necessary port=19530, user='root', password='Milvus', ) ) ``` 5. **Run the Migration Command:** Execute the migration command to migrate the data to Milvus. ```bash flask vdb-migrarte # or docker exec -it docker-api-1 flask vdb-migrarte ``` By following these steps, you should be able to resolve the connection error and successfully switch to Milvus in your self-hosted Dify setup <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/en/getting-started/install-self-hosted/install-faq.md)</sup><sup>[[2]](https://github.com/langgenius/dify/blob/main/docker/docker-compose.milvus.yaml)</sup><sup>[[3]](https://github.com/langgenius/dify/blob/main/api/tests/integration_tests/vdb/milvus/test_milvus.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:05:41 -05:00
Author
Owner
Copy Link
Copy Source

@0sengseng0 commented on GitHub (Jun 28, 2024):

If I execute docker-compose.middleware.yaml file separately: docker-compose-f docker-compose.middleware.yaml up -d. milvus gives an error:

milvusdb/milvus:v2.3.1 ERROR LOG

image

[2024/06/28 09:16:39.628 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:16:39.628 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:16:39.829 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:40.230 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:40.231 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:16:40.231 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:16:40.432 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:40.886 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:40.887 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:16:40.888 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:16:40.984 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:40.985 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:16:40.985 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:16:40.989 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:40.990 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:16:40.990 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:16:41.088 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:41.186 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""]
[2024/06/28 09:16:41.191 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:41.233 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:41.234 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:16:41.234 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:16:41.436 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""]
[2024/06/28 09:16:43.037 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""]
[2024/06/28 09:16:43.038 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:16:43.038 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:16:43.239 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:43.239 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=4] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196\ngithub.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:254\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:164\ngithub.com/milvus-io/milvus/internal/datacoord.(*Cluster).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initServiceDiscovery\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:490\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:347\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:316\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:108\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:229\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/06/28 09:16:43.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"]
[2024/06/28 09:16:43.813 +00:00] [WARN] [grpcclient/client.go:499] ["ClientBase Call grpc call get error"] [role=rootcoord] [address=] [error="stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:485 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel: attempt #0: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #1: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout": attempt #2: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #3: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #4: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout": context done during sleep after run#4: context deadline exceeded"]
[2024/06/28 09:16:43.819 +00:00] [ERROR] [datacoord/handler.go:411] ["datacoord ServerHandler HasCollection finally failed"] [collectionID=450770507019846953] [error="attempt #0: stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:485 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel: attempt #0: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #1: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #2: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #3: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #4: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: context done during sleep after run#4: context deadline exceeded: attempt #1: stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:485 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel: attempt #0: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #1: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout": attempt #2: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #3: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #4: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout": context done during sleep after run#4: context deadline exceeded: context done during sleep after run#1: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:254\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:164\ngithub.com/milvus-io/milvus/internal/datacoord.(*Cluster).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initServiceDiscovery\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:490\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:347\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:316\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:108\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:229\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/handler.go:416] ["datacoord ServerHandler CheckShouldDropChannel hasCollection"] [shouldDropChannel=false] [channel=by-dev-rootcoord-dml_0_450770507019846953v0]
[2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/channel_manager.go:171] ["starting etcd states checker"]
[2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/channel_manager.go:176] ["starting background balance checker"]
[2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/channel_manager.go:179] ["cluster start up"] [nodes="[14]"] [oNodes="[14]"] ["old onlines"="[14]"] ["new onlines"="[]"] [offLines="[]"]
[2024/06/28 09:16:43.823 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-15] [address=172.169.11.4:21121]
[2024/06/28 09:16:43.823 +00:00] [DEBUG] [datacoord/indexnode_manager.go:83] ["add IndexNode"] [nodeID=15] ["node address"=172.169.11.4:21121]
[2024/06/28 09:16:43.823 +00:00] [DEBUG] [datacoord/indexnode_manager.go:56] ["set IndexNode client"] [nodeID=15]
[2024/06/28 09:16:43.823 +00:00] [DEBUG] [datacoord/indexnode_manager.go:61] ["IndexNode IndexNodeManager setClient success"] [nodeID=15] ["IndexNode num"=1]
[2024/06/28 09:16:43.825 +00:00] [INFO] [datacoord/garbage_collector.go:71] ["GC with option"] [enabled=true] [interval=1h0m0s] [missingTolerance=1h0m0s] [dropTolerance=3h0m0s]
[2024/06/28 09:16:43.825 +00:00] [DEBUG] [datacoord/service.go:232] ["DataCoord init done ..."]
[2024/06/28 09:16:43.825 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=indexcoord] [ServerID=15]
[2024/06/28 09:16:43.831 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/indexcoord] [value="{"ServerID":15,"ServerName":"indexcoord","Address":"172.169.11.4:13333","Exclusive":true,"TriggerKill":true,"Version":"2.3.1","LeaseID":7587879681157680761}"]
[2024/06/28 09:16:43.831 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=indexcoord] [serverID=15]
[2024/06/28 09:16:43.831 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=datacoord] [ServerID=15]
[2024/06/28 09:16:43.836 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/06/28 09:16:46.878 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=4] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/06/28 09:16:47.489 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:47.490 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:16:47.490 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:16:47.587 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""]
[2024/06/28 09:16:47.588 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:16:47.588 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:16:47.593 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:47.594 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:16:47.594 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:16:47.691 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:47.789 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:47.795 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:48.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"]
[2024/06/28 09:16:48.904 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=8] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/06/28 09:16:53.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"]
[2024/06/28 09:16:57.692 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:57.694 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:16:57.694 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:16:57.790 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:57.791 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:16:57.791 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:16:57.795 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:57.797 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:16:57.797 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:16:57.895 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:57.992 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:57.997 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:16:58.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"]
[2024/06/28 09:16:58.924 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=8] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/06/28 09:17:00.940 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=12] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/06/28 09:17:03.295 +00:00] [DEBUG] [datanode/services.go:70] ["DataNode current state"] [State=Abnormal]
[2024/06/28 09:17:03.295 +00:00] [INFO] [indexnode/indexnode.go:277] ["get IndexNode components states ..."]
[2024/06/28 09:17:03.296 +00:00] [INFO] [indexnode/indexnode.go:295] ["IndexNode Component states"] [State="nodeID:15 role:"indexnode" state_code:Healthy "] [Status=Success] [SubcomponentStates=null]
[2024/06/28 09:17:03.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"]
[2024/06/28 09:17:07.896 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:17:07.897 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:17:07.897 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:17:07.993 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:17:07.994 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:17:07.994 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:17:07.999 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:17:08.000 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:17:08.000 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:17:08.098 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:17:08.098 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=8] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:531\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:355\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/06/28 09:17:08.195 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:17:08.195 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=8] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:156\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:95\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/06/28 09:17:08.201 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:17:08.201 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=8] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:285\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:189\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/06/28 09:17:08.451 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"]
[2024/06/28 09:17:10.958 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=12] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/06/28 09:17:12.978 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=16] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/06/28 09:17:13.451 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"]
[2024/06/28 09:17:18.100 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:17:18.101 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:17:18.101 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:17:18.196 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:17:18.197 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:17:18.197 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:17:18.201 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:17:18.202 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100]
[2024/06/28 09:17:18.202 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14]
[2024/06/28 09:17:18.302 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:17:18.398 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"]
[2024/06/28 09:17:18.403 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""]
[2024/06/28 09:17:18.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"]
[2024/06/28 09:17:22.992 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=16] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/06/28 09:17:23.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"]
[2024/06/28 09:17:25.011 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=20] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]

@0sengseng0 commented on GitHub (Jun 28, 2024): If I execute docker-compose.middleware.yaml file separately: docker-compose-f docker-compose.middleware.yaml up -d. milvus gives an error: # milvusdb/milvus:v2.3.1 ERROR LOG ![image](https://github.com/langgenius/dify/assets/73268510/86bd02ed-4c98-437f-a62a-fccca886ad99) [2024/06/28 09:16:39.628 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:39.628 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:39.829 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.230 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.231 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:40.231 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:40.432 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.886 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.887 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:40.888 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:40.984 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.985 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:40.985 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:40.989 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.990 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:40.990 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:41.088 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:41.186 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = \"transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout\""] [2024/06/28 09:16:41.191 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:41.233 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:41.234 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:41.234 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:41.436 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = \"transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout\""] [2024/06/28 09:16:43.037 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = \"transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout\""] [2024/06/28 09:16:43.038 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:43.038 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:43.239 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:43.239 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=4] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = \"transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout\""] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196\ngithub.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:254\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:164\ngithub.com/milvus-io/milvus/internal/datacoord.(*Cluster).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initServiceDiscovery\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:490\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:347\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:316\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:108\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:229\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:43.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:16:43.813 +00:00] [WARN] [grpcclient/client.go:499] ["ClientBase Call grpc call get error"] [role=rootcoord] [address=] [error="stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:485 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel: attempt #0: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #1: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = \"transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout\": attempt #2: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #3: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #4: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = \"transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout\": context done during sleep after run#4: context deadline exceeded"] [2024/06/28 09:16:43.819 +00:00] [ERROR] [datacoord/handler.go:411] ["datacoord ServerHandler HasCollection finally failed"] [collectionID=450770507019846953] [error="attempt #0: stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:485 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel: attempt #0: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #1: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #2: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #3: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #4: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: context done during sleep after run#4: context deadline exceeded: attempt #1: stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:485 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel: attempt #0: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #1: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = \"transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout\": attempt #2: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #3: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #4: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = \"transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout\": context done during sleep after run#4: context deadline exceeded: context done during sleep after run#1: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:254\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:164\ngithub.com/milvus-io/milvus/internal/datacoord.(*Cluster).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initServiceDiscovery\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:490\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:347\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:316\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:108\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:229\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/handler.go:416] ["datacoord ServerHandler CheckShouldDropChannel hasCollection"] [shouldDropChannel=false] [channel=by-dev-rootcoord-dml_0_450770507019846953v0] [2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/channel_manager.go:171] ["starting etcd states checker"] [2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/channel_manager.go:176] ["starting background balance checker"] [2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/channel_manager.go:179] ["cluster start up"] [nodes="[14]"] [oNodes="[14]"] ["old onlines"="[14]"] ["new onlines"="[]"] [offLines="[]"] [2024/06/28 09:16:43.823 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-15] [address=172.169.11.4:21121] [2024/06/28 09:16:43.823 +00:00] [DEBUG] [datacoord/indexnode_manager.go:83] ["add IndexNode"] [nodeID=15] ["node address"=172.169.11.4:21121] [2024/06/28 09:16:43.823 +00:00] [DEBUG] [datacoord/indexnode_manager.go:56] ["set IndexNode client"] [nodeID=15] [2024/06/28 09:16:43.823 +00:00] [DEBUG] [datacoord/indexnode_manager.go:61] ["IndexNode IndexNodeManager setClient success"] [nodeID=15] ["IndexNode num"=1] [2024/06/28 09:16:43.825 +00:00] [INFO] [datacoord/garbage_collector.go:71] ["GC with option"] [enabled=true] [interval=1h0m0s] [missingTolerance=1h0m0s] [dropTolerance=3h0m0s] [2024/06/28 09:16:43.825 +00:00] [DEBUG] [datacoord/service.go:232] ["DataCoord init done ..."] [2024/06/28 09:16:43.825 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=indexcoord] [ServerID=15] [2024/06/28 09:16:43.831 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/indexcoord] [value="{\"ServerID\":15,\"ServerName\":\"indexcoord\",\"Address\":\"172.169.11.4:13333\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.3.1\",\"LeaseID\":7587879681157680761}"] [2024/06/28 09:16:43.831 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=indexcoord] [serverID=15] [2024/06/28 09:16:43.831 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=datacoord] [ServerID=15] [2024/06/28 09:16:43.836 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:46.878 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=4] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:47.489 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:47.490 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:47.490 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:47.587 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = \"transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout\""] [2024/06/28 09:16:47.588 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:47.588 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:47.593 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:47.594 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:47.594 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:47.691 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:47.789 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:47.795 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:48.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:16:48.904 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=8] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:53.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:16:57.692 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.694 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:57.694 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:57.790 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.791 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:57.791 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:57.795 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.797 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:57.797 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:57.895 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.992 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.997 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:58.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:16:58.924 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=8] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:00.940 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=12] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:03.295 +00:00] [DEBUG] [datanode/services.go:70] ["DataNode current state"] [State=Abnormal] [2024/06/28 09:17:03.295 +00:00] [INFO] [indexnode/indexnode.go:277] ["get IndexNode components states ..."] [2024/06/28 09:17:03.296 +00:00] [INFO] [indexnode/indexnode.go:295] ["IndexNode Component states"] [State="nodeID:15 role:\"indexnode\" state_code:Healthy "] [Status=Success] [SubcomponentStates=null] [2024/06/28 09:17:03.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:07.896 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:07.897 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:07.897 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:07.993 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:07.994 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:07.994 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:07.999 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:08.000 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:08.000 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:08.098 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:08.098 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=8] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:531\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:355\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:08.195 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:08.195 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=8] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:156\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:95\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:08.201 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:08.201 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=8] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:285\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:189\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:08.451 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:10.958 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=12] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:12.978 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=16] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:13.451 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:18.100 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.101 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:18.101 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:18.196 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.197 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:18.197 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:18.201 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.202 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:18.202 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:18.302 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.398 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.403 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = \"transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout\""] [2024/06/28 09:17:18.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:22.992 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=16] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:23.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:25.011 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=20] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
yindo commented 2026-02-21 18:05:41 -05:00
Author
Owner
Copy Link
Copy Source

@0sengseng0 commented on GitHub (Jun 28, 2024):

If I execute docker-compose.middleware.yaml file separately: docker-compose-f docker-compose.middleware.yaml up -d. milvus gives an error:

milvusdb/milvus:v2.3.1 ERROR LOG

image

[2024/06/28 09:16:39.628 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:39.628 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:39.829 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.230 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.231 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:40.231 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:40.432 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.886 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.887 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:40.888 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:40.984 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.985 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:40.985 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:40.989 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.990 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:40.990 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:41.088 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:41.186 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""] [2024/06/28 09:16:41.191 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:41.233 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:41.234 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:41.234 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:41.436 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""] [2024/06/28 09:16:43.037 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""] [2024/06/28 09:16:43.038 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:43.038 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:43.239 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:43.239 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=4] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196\ngithub.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:254\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:164\ngithub.com/milvus-io/milvus/internal/datacoord.(*Cluster).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initServiceDiscovery\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:490\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:347\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:316\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:108\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:229\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:43.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:16:43.813 +00:00] [WARN] [grpcclient/client.go:499] ["ClientBase Call grpc call get error"] [role=rootcoord] [address=] [error="stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:485 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel: attempt #0: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #1: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout": attempt #2: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #3: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #4: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout": context done during sleep after run#4: context deadline exceeded"] [2024/06/28 09:16:43.819 +00:00] [ERROR] [datacoord/handler.go:411] ["datacoord ServerHandler HasCollection finally failed"] [collectionID=450770507019846953] [error="attempt #0: stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:485 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel: attempt #0: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #1: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #2: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #3: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #4: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: context done during sleep after run#4: context deadline exceeded: attempt #1: stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:485 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel: attempt #0: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #1: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout": attempt #2: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #3: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #4: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout": context done during sleep after run#4: context deadline exceeded: context done during sleep after run#1: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:254\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:164\ngithub.com/milvus-io/milvus/internal/datacoord.(*Cluster).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initServiceDiscovery\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:490\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:347\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:316\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:108\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:229\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/handler.go:416] ["datacoord ServerHandler CheckShouldDropChannel hasCollection"] [shouldDropChannel=false] [channel=by-dev-rootcoord-dml_0_450770507019846953v0] [2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/channel_manager.go:171] ["starting etcd states checker"] [2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/channel_manager.go:176] ["starting background balance checker"] [2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/channel_manager.go:179] ["cluster start up"] [nodes="[14]"] [oNodes="[14]"] ["old onlines"="[14]"] ["new onlines"="[]"] [offLines="[]"] [2024/06/28 09:16:43.823 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-15] [address=172.169.11.4:21121] [2024/06/28 09:16:43.823 +00:00] [DEBUG] [datacoord/indexnode_manager.go:83] ["add IndexNode"] [nodeID=15] ["node address"=172.169.11.4:21121] [2024/06/28 09:16:43.823 +00:00] [DEBUG] [datacoord/indexnode_manager.go:56] ["set IndexNode client"] [nodeID=15] [2024/06/28 09:16:43.823 +00:00] [DEBUG] [datacoord/indexnode_manager.go:61] ["IndexNode IndexNodeManager setClient success"] [nodeID=15] ["IndexNode num"=1] [2024/06/28 09:16:43.825 +00:00] [INFO] [datacoord/garbage_collector.go:71] ["GC with option"] [enabled=true] [interval=1h0m0s] [missingTolerance=1h0m0s] [dropTolerance=3h0m0s] [2024/06/28 09:16:43.825 +00:00] [DEBUG] [datacoord/service.go:232] ["DataCoord init done ..."] [2024/06/28 09:16:43.825 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=indexcoord] [ServerID=15] [2024/06/28 09:16:43.831 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/indexcoord] [value="{"ServerID":15,"ServerName":"indexcoord","Address":"172.169.11.4:13333","Exclusive":true,"TriggerKill":true,"Version":"2.3.1","LeaseID":7587879681157680761}"] [2024/06/28 09:16:43.831 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=indexcoord] [serverID=15] [2024/06/28 09:16:43.831 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=datacoord] [ServerID=15] [2024/06/28 09:16:43.836 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:46.878 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=4] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:47.489 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:47.490 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:47.490 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:47.587 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""] [2024/06/28 09:16:47.588 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:47.588 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:47.593 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:47.594 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:47.594 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:47.691 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:47.789 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:47.795 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:48.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:16:48.904 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=8] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:53.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:16:57.692 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.694 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:57.694 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:57.790 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.791 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:57.791 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:57.795 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.797 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:57.797 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:57.895 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.992 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.997 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:58.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:16:58.924 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=8] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:00.940 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=12] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:03.295 +00:00] [DEBUG] [datanode/services.go:70] ["DataNode current state"] [State=Abnormal] [2024/06/28 09:17:03.295 +00:00] [INFO] [indexnode/indexnode.go:277] ["get IndexNode components states ..."] [2024/06/28 09:17:03.296 +00:00] [INFO] [indexnode/indexnode.go:295] ["IndexNode Component states"] [State="nodeID:15 role:"indexnode" state_code:Healthy "] [Status=Success] [SubcomponentStates=null] [2024/06/28 09:17:03.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:07.896 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:07.897 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:07.897 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:07.993 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:07.994 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:07.994 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:07.999 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:08.000 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:08.000 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:08.098 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:08.098 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=8] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:531\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:355\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:08.195 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:08.195 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=8] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:156\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:95\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:08.201 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:08.201 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=8] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:285\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:189\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:08.451 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:10.958 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=12] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:12.978 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=16] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:13.451 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:18.100 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.101 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:18.101 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:18.196 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.197 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:18.197 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:18.201 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.202 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:18.202 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:18.302 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.398 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.403 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""] [2024/06/28 09:17:18.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:22.992 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=16] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:23.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:25.011 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=20] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]

The file I executed was docker-compose.milvus.yaml instead of docker-compose.middleware.yaml. Clerical error

@0sengseng0 commented on GitHub (Jun 28, 2024): > If I execute docker-compose.middleware.yaml file separately: docker-compose-f docker-compose.middleware.yaml up -d. milvus gives an error: > > # milvusdb/milvus:v2.3.1 ERROR LOG > ![image](https://private-user-images.githubusercontent.com/73268510/344110888-86bd02ed-4c98-437f-a62a-fccca886ad99.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk1NjY3NDAsIm5iZiI6MTcxOTU2NjQ0MCwicGF0aCI6Ii83MzI2ODUxMC8zNDQxMTA4ODgtODZiZDAyZWQtNGM5OC00MzdmLWE2MmEtZmNjY2E4ODZhZDk5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MjglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjI4VDA5MjA0MFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTIxMDgwOTQxYjFiYjIwMmUxOWMwZjYwMWE2NzVkMzIwNmQ3OWUyOWVjNDlkZGNlNjFmY2U4NDEyNWQ1MGIzNGMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.-Pu1oXUznGGEyKH8DBPi_jGd8uCmAAR3hR9Y1MMOXhM) > > [2024/06/28 09:16:39.628 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:39.628 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:39.829 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.230 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.231 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:40.231 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:40.432 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.886 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.887 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:40.888 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:40.984 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.985 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:40.985 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:40.989 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:40.990 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:40.990 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:41.088 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:41.186 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""] [2024/06/28 09:16:41.191 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:41.233 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:41.234 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:41.234 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:41.436 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""] [2024/06/28 09:16:43.037 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""] [2024/06/28 09:16:43.038 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:43.038 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:43.239 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:43.239 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=4] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196\ngithub.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:254\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:164\ngithub.com/milvus-io/milvus/internal/datacoord.(*Cluster).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initServiceDiscovery\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:490\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:347\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:316\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:108\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:229\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:43.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:16:43.813 +00:00] [WARN] [grpcclient/client.go:499] ["ClientBase Call grpc call get error"] [role=rootcoord] [address=] [error="stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:485 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel: attempt #0: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #1: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout": attempt #2: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #3: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #4: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout": context done during sleep after run#4: context deadline exceeded"] [2024/06/28 09:16:43.819 +00:00] [ERROR] [datacoord/handler.go:411] ["datacoord ServerHandler HasCollection finally failed"] [collectionID=450770507019846953] [error="attempt #0: stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:485 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel: attempt #0: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #1: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #2: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #3: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #4: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: context done during sleep after run#4: context deadline exceeded: attempt #1: stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:485 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:135 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:372 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:371 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel: attempt #0: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #1: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout": attempt #2: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #3: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: attempt #4: empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout": context done during sleep after run#4: context deadline exceeded: context done during sleep after run#1: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:411\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:254\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:164\ngithub.com/milvus-io/milvus/internal/datacoord.(*Cluster).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initServiceDiscovery\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:490\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:347\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:316\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:108\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:229\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/handler.go:416] ["datacoord ServerHandler CheckShouldDropChannel hasCollection"] [shouldDropChannel=false] [channel=by-dev-rootcoord-dml_0_450770507019846953v0] [2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/channel_manager.go:171] ["starting etcd states checker"] [2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/channel_manager.go:176] ["starting background balance checker"] [2024/06/28 09:16:43.821 +00:00] [INFO] [datacoord/channel_manager.go:179] ["cluster start up"] [nodes="[14]"] [oNodes="[14]"] ["old onlines"="[14]"] ["new onlines"="[]"] [offLines="[]"] [2024/06/28 09:16:43.823 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-15] [address=172.169.11.4:21121] [2024/06/28 09:16:43.823 +00:00] [DEBUG] [datacoord/indexnode_manager.go:83] ["add IndexNode"] [nodeID=15] ["node address"=172.169.11.4:21121] [2024/06/28 09:16:43.823 +00:00] [DEBUG] [datacoord/indexnode_manager.go:56] ["set IndexNode client"] [nodeID=15] [2024/06/28 09:16:43.823 +00:00] [DEBUG] [datacoord/indexnode_manager.go:61] ["IndexNode IndexNodeManager setClient success"] [nodeID=15] ["IndexNode num"=1] [2024/06/28 09:16:43.825 +00:00] [INFO] [datacoord/garbage_collector.go:71] ["GC with option"] [enabled=true] [interval=1h0m0s] [missingTolerance=1h0m0s] [dropTolerance=3h0m0s] [2024/06/28 09:16:43.825 +00:00] [DEBUG] [datacoord/service.go:232] ["DataCoord init done ..."] [2024/06/28 09:16:43.825 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=indexcoord] [ServerID=15] [2024/06/28 09:16:43.831 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/indexcoord] [value="{"ServerID":15,"ServerName":"indexcoord","Address":"172.169.11.4:13333","Exclusive":true,"TriggerKill":true,"Version":"2.3.1","LeaseID":7587879681157680761}"] [2024/06/28 09:16:43.831 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=indexcoord] [serverID=15] [2024/06/28 09:16:43.831 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=datacoord] [ServerID=15] [2024/06/28 09:16:43.836 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:46.878 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=4] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:47.489 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:47.490 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:47.490 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:47.587 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""] [2024/06/28 09:16:47.588 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:47.588 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:47.593 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:47.594 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:47.594 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:47.691 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:47.789 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:47.795 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:48.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:16:48.904 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=8] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:16:53.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:16:57.692 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.694 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:57.694 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:57.790 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.791 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:57.791 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:57.795 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.797 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:16:57.797 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:16:57.895 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.992 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:57.997 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:16:58.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:16:58.924 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=8] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:00.940 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=12] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:03.295 +00:00] [DEBUG] [datanode/services.go:70] ["DataNode current state"] [State=Abnormal] [2024/06/28 09:17:03.295 +00:00] [INFO] [indexnode/indexnode.go:277] ["get IndexNode components states ..."] [2024/06/28 09:17:03.296 +00:00] [INFO] [indexnode/indexnode.go:295] ["IndexNode Component states"] [State="nodeID:15 role:"indexnode" state_code:Healthy "] [Status=Success] [SubcomponentStates=null] [2024/06/28 09:17:03.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:07.896 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:07.897 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:07.897 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:07.993 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:07.994 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:07.994 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:07.999 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:08.000 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:08.000 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:08.098 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:08.098 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=8] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:531\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:355\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:08.195 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:08.195 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=8] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:156\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:95\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:08.201 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:08.201 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=8] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:285\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:189\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:08.451 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:10.958 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=12] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:12.978 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=16] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:13.451 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:18.100 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.101 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:18.101 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:18.196 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.197 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:18.197 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:18.201 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.202 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.169.9.8:53100] [2024/06/28 09:17:18.202 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.169.9.8:53100] [serverID=14] [2024/06/28 09:17:18.302 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.398 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded"] [2024/06/28 09:17:18.403 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.169.9.8:53100, reason: context deadline exceeded: connection error: desc = "transport: error while dialing: dial tcp 172.169.9.8:53100: i/o timeout""] [2024/06/28 09:17:18.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:22.992 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=16] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] [2024/06/28 09:17:23.450 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] [2024/06/28 09:17:25.011 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=20] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] The file I executed was docker-compose.milvus.yaml instead of docker-compose.middleware.yaml. Clerical error
yindo commented 2026-02-21 18:05:41 -05:00
Author
Owner
Copy Link
Copy Source

@Marvin-Song commented on GitHub (Jul 8, 2024):

i use dify 0.6.12 deploy with docker
set MILVUS_HOST=172.17.0.1 and it works for me.

@Marvin-Song commented on GitHub (Jul 8, 2024): i use dify 0.6.12 deploy with docker set MILVUS_HOST=172.17.0.1 and it works for me.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 👻 feat:rag
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#4307
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?