mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-12 02:19:10 +00:00
David Dallakyanandxiaofanluan
3dfae89e93
General LIKE with interior `%` or `_` was declined by FMINDEX even though PrefixMatch, PostfixMatch, and InnerMatch already use it. This routes VARCHAR Match through `ShouldUseOp`: split the pattern on wildcards, take candidates from the rarest literal fragment, then recheck those rows on sealed VARCHAR with `LikePatternMatcher` via `ProcessDataByOffsets`. FMINDEX load no longer materializes ISA. Match phase 2 rechecks candidate rows on sealed VARCHAR via ProcessDataByOffsets. The Match guard is locate-only again, the same occ times sa_sample_rate bound as Prefix, Inner, and Postfix. Empty-fragment patterns still decline. RegexMatch, equality, IN, TEXT, JSON, ARRAY, and growing segments are unchanged. FMINDEX Match is never served as unrechecked PatternMatch candidates through the ScalarIndex path. Design: `docs/design-docs/design_docs/20260708-fm_index_scalar_index.md` issue: #52683 ## How to test `LOCAL_STORAGE_SIZE=10` is required on this host. Without the VARCHAR recheck (product from `47f5cf5791`, tests kept): 5 FAILED (`ShouldUseOp(Match)` still prices Extract, so `%RARE%` / `%ZEBRA%` stay declined). With `7a3e274a36`: ``` source scripts/setenv.sh export DYLD_LIBRARY_PATH="$PWD/cmake_build/src:$DYLD_LIBRARY_PATH" LOCAL_STORAGE_SIZE=10 cmake_build/unittest/all_tests --gtest_filter='FMIndex.*' [==========] 23 tests from 1 test suite ran. [ PASSED ] 23 tests. ``` Includes `FMIndex.ExecutorPathMatchRechecksVarchar` and `FMIndex.MatchOracleEqualsBruteForce`. Whole C++ module (three Darwin crashers excluded): ``` export LOCAL_STORAGE_SIZE=10 internal/core/output/unittest/all_tests --gtest_filter='-JsonIndexTest.LargeInt64LiteralDoesNotAliasInDoublePathIndex:JsonFlatIndexExprTest.PreservesLargeInt64LiteralPrecision:SortBufferTest.*' [==========] Running 8326 tests from 473 test suites. [ PASSED ] 8167 tests. ``` None of the 138 failures are `FMIndex.*`. Reverse deps: Ngram 10/10, scalar/hybrid/inverted 26/26, json_stats_test 79/79, LikeConjunctExpr 1/1. /cc @xiaofan-luan --------- Signed-off-by: David Dallakyan <[email protected]> Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]>
Compile and install milvus cluster
Environment
OS: Ubuntu 20.04
go：1.21
cmake: >=3.18
gcc: >= 11
Install dependencies
Install compile dependencies
$ sudo apt install -y g++ gcc make libssl-dev zlib1g-dev libboost-regex-dev \
libboost-program-options-dev libboost-system-dev libboost-filesystem-dev \
libboost-serialization-dev python3-dev libboost-python-dev libcurl4-openssl-dev gfortran libtbb-dev
$ export GO111MODULE=on
$ go get github.com/golang/protobuf/[email protected]
OpenBLAS is managed by Conan for C++ core builds. Install the compiler toolchain and Fortran runtime needed by Conan, then run the normal third-party build step.
Compile
Generate the go files from proto file
$ make check-proto-product
Check code specifications
$ make verifiers
Compile milvus
$ make milvus
Install docker-compose
refer: https://docs.docker.com/compose/install/
$ sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
$ docker-compose --version
$ docker compose --version
Start service
Start third-party service:
$ cd [milvus project path]/deployments/docker/cluster
$ docker-compose up -d
$ docker compose up -d
Start milvus cluster:
$ cd [milvus project path]
$ ./scripts/start_cluster.sh
Run unittest
Run all unittest including go and cpp cases:
$ make unittest
You also can run go unittest only:
$ make test-go
Run cpp unittest only:
$ make test-cpp
Run code coverage
Run code coverage including go and cpp:
$ make codecov
You also can run go code coverage only:
$ make codecov-go
Run cpp code coverage only:
$ make codecov-cpp