Hybrid Search Fails with Sparse Vector Type — "Search target vector type is illegal" Exception

[qidafang0413]

Hi Milvus team,

I encountered an issue while using the Java SDK for hybrid search. I followed the example provided in the documentation and translated it into Java code. However, when executing the hybrid search, it throws a ParamException indicating that the vector type is invalid for sparse vectors.

Here's the Python example from the documentation that I tried to replicate:

search_params_dense = {
    "data": [[1.25, 2, 3.5]],
    "anns_field": "vector",
    "param": {
        "metric_type": "IP",
        "params": {"nprobe": 10}, 
    },
    "limit": 100
}

req_dense = ANNSearchRequest(**search_params_dense)

search_params_sparse = {
    "data": ["shoes"],
    "anns_field": "text_sparse",
    "param": {
        "metric_type": "BM25",
        "params": {"drop_ratio_search": 0.2}
    }
}

req_sparse = ANNSearchRequest(**search_params_sparse)

res = client.hybrid_search(
    collection_name="my_collection",
    reqs=[req_dense, req_sparse],
    reranker=RRFRanker(),
    limit=10
)

And here's my Java equivalent:

List<AnnSearchReq> searchRequests = new ArrayList<>();
searchRequests.add(AnnSearchReq.builder()
        .vectorFieldName("dense_vector")
        .vectors(Collections.singletonList(new FloatVec(queryVector)))
        .metricType(IndexParam.MetricType.COSINE)
        .params("{\"nprobe\": 10}")
        .topK(topK)
        .build());
searchRequests.add(AnnSearchReq.builder()
        .vectorFieldName("sparse_vector")
        .vectors(Collections.singletonList(new EmbeddedText(question)))
        .metricType(IndexParam.MetricType.BM25)
        .params("{\"drop_ratio_build\": 0.2}")
        .topK(topK)
        .build());

BaseRanker reranker = new WeightedRanker(Arrays.asList(0.8f, 0.3f));

HybridSearchReq hybridSearchReq = HybridSearchReq.builder()
    .collectionName(collectionName)
    .searchRequests(searchRequests)
    .ranker(reranker)
    .topK(topK)
    .build();

SearchResp searchResp = milvusClient.hybridSearch(hybridSearchReq);

But I get the following exception:

io.milvus.exception.ParamException: Search target vector type is illegal. Only allow List<Float> for FloatVector, ByteBuffer for BinaryVector/Float16Vector/BFloat16Vector, List<SortedMap<Long, Float>> for SparseFloatVector.

It seems the Java SDK does not accept EmbeddedText as a valid input for sparse vector hybrid search, even though the Python SDK allows a string for sparse text fields.

Could you clarify what the correct way is to represent sparse vector input (text-based or otherwise) in the Java SDK? If the EmbeddedText class is not supported for this purpose, some guidance or equivalent usage would be much appreciated.

Thanks in advance!

[yhmo]

You are using metric_type="BM25", it is for full-text-match. The example for full-text-match is here: https://github.com/milvus-io/milvus-sdk-java/blob/master/examples/src/main/java/io/milvus/v2/FullTextSearchExample.java The doc for full-text-match is here: https://milvus.io/docs/full-text-search.md

The following example is a demo for full-text-match in hybridSearch:

package io.milvus.v2;

import com.google.gson.Gson;
import com.google.gson.JsonObject;
import io.milvus.common.clientenum.FunctionType;
import io.milvus.v2.common.DataType;
import io.milvus.v2.client.ConnectConfig;
import io.milvus.v2.client.MilvusClientV2;
import io.milvus.v2.common.ConsistencyLevel;
import io.milvus.v2.common.IndexParam;
import io.milvus.v2.service.collection.request.AddFieldReq;
import io.milvus.v2.service.collection.request.CreateCollectionReq;
import io.milvus.v2.service.collection.request.DropCollectionReq;
import io.milvus.v2.service.utility.request.FlushReq;
import io.milvus.v2.service.vector.request.AnnSearchReq;
import io.milvus.v2.service.vector.request.HybridSearchReq;
import io.milvus.v2.service.vector.request.InsertReq;
import io.milvus.v2.service.vector.request.data.*;
import io.milvus.v2.service.vector.request.ranker.RRFRanker;
import io.milvus.v2.service.vector.response.SearchResp;

import java.util.*;

public class HybridSearchExample {
    private static final MilvusClientV2 client;

    static {
        ConnectConfig config = ConnectConfig.builder()
                .uri("http://localhost:19530")
                .build();
        client = new MilvusClientV2(config);
    }

    private static void createCollection() {
        client.dropCollection(DropCollectionReq.builder()
                .collectionName("test")
                .build());

        // Create collection
        CreateCollectionReq.CollectionSchema collectionSchema = CreateCollectionReq.CollectionSchema.builder()
                .build();
        collectionSchema.addField(AddFieldReq.builder()
                .fieldName("id")
                .dataType(DataType.Int64)
                .isPrimaryKey(Boolean.TRUE)
                .autoID(true)
                .build());
        collectionSchema.addField(AddFieldReq.builder()
                .fieldName("dense_vector")
                .dataType(DataType.FloatVector)
                .dimension(4)
                .build());

        collectionSchema.addField(AddFieldReq.builder()
                .fieldName("text")
                .dataType(DataType.VarChar)
                .maxLength(500)
                .enableAnalyzer(true)
                .build());
        collectionSchema.addField(AddFieldReq.builder()
                .fieldName("sparse_vector")
                .dataType(DataType.SparseFloatVector)
                .build());

        // With this function, milvus will convert the strings of "text" field to sparse vectors of "vector" field
        // by built-in tokenizer and analyzer
        // Read the link for more info: https://milvus.io/docs/full-text-search.md
        collectionSchema.addFunction(CreateCollectionReq.Function.builder()
                .functionType(FunctionType.BM25)
                .name("function_bm25")
                .inputFieldNames(Collections.singletonList("text"))
                .outputFieldNames(Collections.singletonList("sparse_vector"))
                .build());

        List<IndexParam> indexes = new ArrayList<>();
        indexes.add(IndexParam.builder()
                .fieldName("dense_vector")
                .indexType(IndexParam.IndexType.AUTOINDEX)
                .metricType(IndexParam.MetricType.COSINE)
                .build());
        indexes.add(IndexParam.builder()
                .fieldName("sparse_vector")
                .indexType(IndexParam.IndexType.SPARSE_INVERTED_INDEX)
                .metricType(IndexParam.MetricType.BM25)
                .build());

        CreateCollectionReq requestCreate = CreateCollectionReq.builder()
                .collectionName("test")
                .collectionSchema(collectionSchema)
                .indexParams(indexes)
                .build();
        client.createCollection(requestCreate);
        System.out.println("Collection created");
    }

    private static void insertData() {
        // Insert entities by rows
        List<JsonObject> rows = new ArrayList<>();
        Gson gson = new Gson();
        {
            JsonObject row = new JsonObject();
            row.addProperty("text", "hello world, hello database");
            row.add("dense_vector", gson.toJsonTree(Arrays.asList(1.0, 1.1, 1.2, 1.3)));
            rows.add(row);
        }
        {
            JsonObject row = new JsonObject();
            row.addProperty("text", "sky is blue, vector is red");
            row.add("dense_vector", gson.toJsonTree(Arrays.asList(2.0, 2.1, 2.2, 2.3)));
            rows.add(row);
        }
        {
            JsonObject row = new JsonObject();
            row.addProperty("text", "this is milvus, a vector database");
            row.add("dense_vector", gson.toJsonTree(Arrays.asList(3.0, 3.1, 3.2, 3.3)));
            rows.add(row);
        }

        client.insert(InsertReq.builder()
                .collectionName("test")
                .data(rows)
                .build());
        client.flush(FlushReq.builder().collectionNames(Collections.singletonList("test")).build());
    }

    private static void hybridSearch() {
        // Search on multiple vector fields
        List<BaseVector> floatVectors = new ArrayList<>();
        floatVectors.add(new FloatVec(Arrays.asList(1.0f, 2.0f, 3.0f, 4.0f)));

        List<BaseVector> sparseTexts = new ArrayList<>();
        sparseTexts.add(new EmbeddedText("what is milvus?"));

        List<AnnSearchReq> searchRequests = new ArrayList<>();
        searchRequests.add(AnnSearchReq.builder()
                .vectorFieldName("dense_vector")
                .vectors(floatVectors)
                .topK(10)
                .build());

        searchRequests.add(AnnSearchReq.builder()
                .vectorFieldName("sparse_vector")
                .vectors(sparseTexts)
                .topK(10)
                .build());

        HybridSearchReq hybridSearchReq = HybridSearchReq.builder()
                .collectionName("test")
                .searchRequests(searchRequests)
                .ranker(new RRFRanker(2))
                .topK(5)
                .outFields(Collections.singletonList("text"))
                .consistencyLevel(ConsistencyLevel.BOUNDED)
                .build();
        SearchResp searchResp = client.hybridSearch(hybridSearchReq);
        List<List<SearchResp.SearchResult>> searchResults = searchResp.getSearchResults();
        for (int i = 0; i < searchResults.size(); i++) {
            System.out.printf("============= Search result of No.%d vector =============\n", i);
            List<SearchResp.SearchResult> results = searchResults.get(i);
            for (SearchResp.SearchResult result : results) {
                System.out.printf("{score: %f, entity: %s}%n", result.getScore(), result.getEntity());
            }
        }
    }

    public static void main(String[] args) {
        createCollection();
        insertData();
        hybridSearch();
    }
}

The output of this example:

Collection created
============= Search result of No.0 vector =============
{score: 0.533333, entity: {text=this is milvus, a vector database}}
{score: 0.500000, entity: {text=sky is blue, vector is red}}
{score: 0.333333, entity: {text=hello world, hello database}}

[yhmo]

How to enable full-text-match? Just set enableAnalyzer to be true for a text field, and configure a Function for the collection schema:

schema.addField(AddFieldReq.builder()
                .fieldName("text")
                .dataType(DataType.VarChar)
                .maxLength(65535)
                .enableAnalyzer(true) // must enable this if you use Function
                .build());

collectionSchema.addFunction(CreateCollectionReq.Function.builder()
                .functionType(FunctionType.BM25)
                .name("function_bm25")
                .inputFieldNames(Collections.singletonList("text"))
                .outputFieldNames(Collections.singletonList("sparse_vector"))
                .build());

With full-text-match, all the sparse vectors are automatically generated by milvus by a text field, users don't need to provide. Users only insert text content, and milvus generates the sparse vectors. So, in a search request, users don't need to input sparse vector, too. Users only input a text context, which is wrapped by an EmbededText object, and milvus will convert it to a new sparse vector, and use the sparse vector to compute.

If full-text-match is disabled, milvus will require users to provide sparse vectors. An example of sparse vector without full-text-match is here: https://github.com/milvus-io/milvus-sdk-java/blob/master/examples/src/main/java/io/milvus/v2/SparseVectorExample.java

[qidafang0413]

You are using metric_type="BM25", it is for full-text-match. The example for full-text-match is here: https://github.com/milvus-io/milvus-sdk-java/blob/master/examples/src/main/java/io/milvus/v2/FullTextSearchExample.java The doc for full-text-match is here: https://milvus.io/docs/full-text-search.md

The following example is a demo for full-text-match in hybridSearch:

package io.milvus.v2;

import com.google.gson.Gson;
import com.google.gson.JsonObject;
import io.milvus.common.clientenum.FunctionType;
import io.milvus.v2.common.DataType;
import io.milvus.v2.client.ConnectConfig;
import io.milvus.v2.client.MilvusClientV2;
import io.milvus.v2.common.ConsistencyLevel;
import io.milvus.v2.common.IndexParam;
import io.milvus.v2.service.collection.request.AddFieldReq;
import io.milvus.v2.service.collection.request.CreateCollectionReq;
import io.milvus.v2.service.collection.request.DropCollectionReq;
import io.milvus.v2.service.utility.request.FlushReq;
import io.milvus.v2.service.vector.request.AnnSearchReq;
import io.milvus.v2.service.vector.request.HybridSearchReq;
import io.milvus.v2.service.vector.request.InsertReq;
import io.milvus.v2.service.vector.request.data.*;
import io.milvus.v2.service.vector.request.ranker.RRFRanker;
import io.milvus.v2.service.vector.response.SearchResp;

import java.util.*;

public class HybridSearchExample {
    private static final MilvusClientV2 client;

    static {
        ConnectConfig config = ConnectConfig.builder()
                .uri("http://localhost:19530")
                .build();
        client = new MilvusClientV2(config);
    }

    private static void createCollection() {
        client.dropCollection(DropCollectionReq.builder()
                .collectionName("test")
                .build());

        // Create collection
        CreateCollectionReq.CollectionSchema collectionSchema = CreateCollectionReq.CollectionSchema.builder()
                .build();
        collectionSchema.addField(AddFieldReq.builder()
                .fieldName("id")
                .dataType(DataType.Int64)
                .isPrimaryKey(Boolean.TRUE)
                .autoID(true)
                .build());
        collectionSchema.addField(AddFieldReq.builder()
                .fieldName("dense_vector")
                .dataType(DataType.FloatVector)
                .dimension(4)
                .build());

        collectionSchema.addField(AddFieldReq.builder()
                .fieldName("text")
                .dataType(DataType.VarChar)
                .maxLength(500)
                .enableAnalyzer(true)
                .build());
        collectionSchema.addField(AddFieldReq.builder()
                .fieldName("sparse_vector")
                .dataType(DataType.SparseFloatVector)
                .build());

        // With this function, milvus will convert the strings of "text" field to sparse vectors of "vector" field
        // by built-in tokenizer and analyzer
        // Read the link for more info: https://milvus.io/docs/full-text-search.md
        collectionSchema.addFunction(CreateCollectionReq.Function.builder()
                .functionType(FunctionType.BM25)
                .name("function_bm25")
                .inputFieldNames(Collections.singletonList("text"))
                .outputFieldNames(Collections.singletonList("sparse_vector"))
                .build());

        List<IndexParam> indexes = new ArrayList<>();
        indexes.add(IndexParam.builder()
                .fieldName("dense_vector")
                .indexType(IndexParam.IndexType.AUTOINDEX)
                .metricType(IndexParam.MetricType.COSINE)
                .build());
        indexes.add(IndexParam.builder()
                .fieldName("sparse_vector")
                .indexType(IndexParam.IndexType.SPARSE_INVERTED_INDEX)
                .metricType(IndexParam.MetricType.BM25)
                .build());

        CreateCollectionReq requestCreate = CreateCollectionReq.builder()
                .collectionName("test")
                .collectionSchema(collectionSchema)
                .indexParams(indexes)
                .build();
        client.createCollection(requestCreate);
        System.out.println("Collection created");
    }

    private static void insertData() {
        // Insert entities by rows
        List<JsonObject> rows = new ArrayList<>();
        Gson gson = new Gson();
        {
            JsonObject row = new JsonObject();
            row.addProperty("text", "hello world, hello database");
            row.add("dense_vector", gson.toJsonTree(Arrays.asList(1.0, 1.1, 1.2, 1.3)));
            rows.add(row);
        }
        {
            JsonObject row = new JsonObject();
            row.addProperty("text", "sky is blue, vector is red");
            row.add("dense_vector", gson.toJsonTree(Arrays.asList(2.0, 2.1, 2.2, 2.3)));
            rows.add(row);
        }
        {
            JsonObject row = new JsonObject();
            row.addProperty("text", "this is milvus, a vector database");
            row.add("dense_vector", gson.toJsonTree(Arrays.asList(3.0, 3.1, 3.2, 3.3)));
            rows.add(row);
        }

        client.insert(InsertReq.builder()
                .collectionName("test")
                .data(rows)
                .build());
        client.flush(FlushReq.builder().collectionNames(Collections.singletonList("test")).build());
    }

    private static void hybridSearch() {
        // Search on multiple vector fields
        List<BaseVector> floatVectors = new ArrayList<>();
        floatVectors.add(new FloatVec(Arrays.asList(1.0f, 2.0f, 3.0f, 4.0f)));

        List<BaseVector> sparseTexts = new ArrayList<>();
        sparseTexts.add(new EmbeddedText("what is milvus?"));

        List<AnnSearchReq> searchRequests = new ArrayList<>();
        searchRequests.add(AnnSearchReq.builder()
                .vectorFieldName("dense_vector")
                .vectors(floatVectors)
                .topK(10)
                .build());

        searchRequests.add(AnnSearchReq.builder()
                .vectorFieldName("sparse_vector")
                .vectors(sparseTexts)
                .topK(10)
                .build());

        HybridSearchReq hybridSearchReq = HybridSearchReq.builder()
                .collectionName("test")
                .searchRequests(searchRequests)
                .ranker(new RRFRanker(2))
                .topK(5)
                .outFields(Collections.singletonList("text"))
                .consistencyLevel(ConsistencyLevel.BOUNDED)
                .build();
        SearchResp searchResp = client.hybridSearch(hybridSearchReq);
        List<List<SearchResp.SearchResult>> searchResults = searchResp.getSearchResults();
        for (int i = 0; i < searchResults.size(); i++) {
            System.out.printf("============= Search result of No.%d vector =============\n", i);
            List<SearchResp.SearchResult> results = searchResults.get(i);
            for (SearchResp.SearchResult result : results) {
                System.out.printf("{score: %f, entity: %s}%n", result.getScore(), result.getEntity());
            }
        }
    }

    public static void main(String[] args) {
        createCollection();
        insertData();
        hybridSearch();
    }
}

The output of this example:

Collection created
============= Search result of No.0 vector =============
{score: 0.533333, entity: {text=this is milvus, a vector database}}
{score: 0.500000, entity: {text=sky is blue, vector is red}}
{score: 0.333333, entity: {text=hello world, hello database}}

Exception in thread "main" io.milvus.v2.exception.MilvusClientException: io.milvus.exception.ParamException: Search target vector type is illegal. Only allow List for FloatVector, ByteBuffer for BinaryVector/Float16Vector/BFloat16Vector, List<SortedMap<Long, Float>> for SparseFloatVector.
at io.milvus.v2.client.MilvusClientV2.retry(MilvusClientV2.java:239)
at io.milvus.v2.client.MilvusClientV2.hybridSearch(MilvusClientV2.java:539)
at com.demo.milvus.HybridSearchExample.hybridSearch(HybridSearchExample.java:155)

[yhmo]

Which version of your milvus and Java SDK?

[qidafang0413]

Which version of your milvus and Java SDK?

My Milvus version is 2.5.11, and I was using Java SDK version 2.5.2.
After I changed the Java SDK version to 2.5.9, the problem was fixed.
Thanks a lot for your help!

[yhmo]

Right, "HybridSearch supports full text search" is supported from Java SDK v2.5.4.
https://github.com/milvus-io/milvus-sdk-java/releases/tag/v2.5.4

[qidafang0413]

Right, "HybridSearch supports full text search" is supported from Java SDK v2.5.4. https://github.com/milvus-io/milvus-sdk-java/releases/tag/v2.5.4

We’ve done some further testing and noticed that the container crash tends to happen when the input text passed to EmbeddedText is in Chinese. This behavior is not consistent, but it occurs frequently enough to be a concern.

Our suspicion is that this may be related to server hardware limitations (e.g., memory or CPU), but the issue doesn't seem to happen with English input, which makes us wonder if there is something specific about how non-ASCII text is processed internally.

Here's an excerpt from the container logs around the time of the crash:

---

/usr/local/go/src/runtime/chan.go:511 +0x12 fp=0xc001534f88 sp=0xc001534f60 pc=0x1eb5412

github.com/panjf2000/ants/v2.(*goWorker).run.func1()
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:63 +0x6a fp=0xc001534fe0 sp=0xc001534f88 pc=0x3df696a
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc001534fe8 sp=0xc001534fe0 pc=0x1f2b4c1
created by github.com/panjf2000/ants/v2.(*goWorker).run in goroutine 23493
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:48 +0x5c

goroutine 23516 gp=0xc003b99c00 m=nil [select]:
runtime.gopark(0xc002431f48?, 0x2?, 0x6c?, 0x4e?, 0xc002431ef4?)
/usr/local/go/src/runtime/proc.go:435 +0xce fp=0xc003b3fd80 sp=0xc003b3fd60 pc=0x1f2278e
runtime.selectgo(0xc003b3ff48, 0xc002431ef0, 0xc00422c900?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:351 +0x837 fp=0xc003b3feb8 sp=0xc003b3fd80 pc=0x1efdcb7
net/http.(*persistConn).writeLoop(0xc00430d560)
/usr/local/go/src/net/http/transport.go:2590 +0xe7 fp=0xc003b3ffc8 sp=0xc003b3feb8 pc=0x2265e27
net/http.(*Transport).dialConn.gowrap3()
/usr/local/go/src/net/http/transport.go:1945 +0x25 fp=0xc003b3ffe0 sp=0xc003b3ffc8 pc=0x2262bc5
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc003b3ffe8 sp=0xc003b3ffe0 pc=0x1f2b4c1
created by net/http.(*Transport).dialConn in goroutine 23529
/usr/local/go/src/net/http/transport.go:1945 +0x17a5

goroutine 19101 gp=0xc003d22380 m=nil [select]:
runtime.gopark(0xc000099740?, 0x4?, 0x78?, 0x1a?, 0xc0000996b8?)
/usr/local/go/src/runtime/proc.go:435 +0xce fp=0xc000099540 sp=0xc000099520 pc=0x1f2278e
runtime.selectgo(0xc000099740, 0xc0000996b0, 0x65b5fd949ec0002?, 0x0, 0x68dcb58?, 0x1)
/usr/local/go/src/runtime/select.go:351 +0x837 fp=0xc000099678 sp=0xc000099540 pc=0x1efdcb7
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc003c18600)
/go/pkg/mod/google.golang.org/grpc@v1.65.0/internal/transport/http2_server.go:1168 +0x1ec fp=0xc0000997c8 sp=0xc000099678 pc=0x2679a2c
google.golang.org/grpc/internal/transport.NewServerTransport.gowrap1()
/go/pkg/mod/google.golang.org/grpc@v1.65.0/internal/transport/http2_server.go:355 +0x25 fp=0xc0000997e0 sp=0xc0000997c8 pc=0x2672885
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000997e8 sp=0xc0000997e0 pc=0x1f2b4c1
created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 19099
/go/pkg/mod/google.golang.org/grpc@v1.65.0/internal/transport/http2_server.go:355 +0x191d

goroutine 23550 gp=0xc003d22540 m=nil [chan receive]:
runtime.gopark(0xea539ff9fed071?, 0x2?, 0x70?, 0x0?, 0x7c49a4a359a8?)
/usr/local/go/src/runtime/proc.go:435 +0xce fp=0xc0033a4a80 sp=0xc0033a4a60 pc=0x1f2278e
runtime.chanrecv(0xc0041f59d0, 0x0, 0x1)
/usr/local/go/src/runtime/chan.go:664 +0x445 fp=0xc0033a4af8 sp=0xc0033a4a80 pc=0x1eb5865
runtime.chanrecv1(0x5224979?, 0xc0033a4b50?)
/usr/local/go/src/runtime/chan.go:506 +0x12 fp=0xc0033a4b20 sp=0xc0033a4af8 pc=0x1eb53f2
github.com/milvus-io/milvus/pkg/v2/util/conc.(*Future[...]).OK(...)
/workspace/source/pkg/util/conc/future.go:70
github.com/milvus-io/milvus/pkg/v2/util/conc.(*Future[...]).OK()
/workspace/source/pkg/util/conc/future.go:69 +0x26 fp=0xc0033a4b40 sp=0xc0033a4b20 pc=0x5224b06
github.com/milvus-io/milvus/pkg/v2/util/conc.AwaitAll[...]({0xc00455e8f0, 0x2, 0x0})
/workspace/source/pkg/util/conc/future.go:108 +0x59 fp=0xc0033a4b60 sp=0xc0033a4b40 pc=0x4b24019
github.com/milvus-io/milvus/internal/querynodev2/delegator.(*shardDelegator).Search(0xc0028fb500, {0x68dcb90, 0xc004d2a370}, 0xc00451ae00)
/workspace/source/internal/querynodev2/delegator/delegator.go:428 +0x1785 fp=0xc0033a4fe0 sp=0xc0033a4b60 pc=0x51fdb85
github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).searchChannel(0xc0014e5760, {0x68dcb90, 0xc004544fa0}, 0xc00451ae00, {0xc0013e0f60, 0x2b})
/workspace/source/internal/querynodev2/handlers.go:430 +0x9de fp=0xc0033a5388 sp=0xc0033a4fe0 pc=0x52451be
github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Search(0xc0014e5760, {0x68dcb90, 0xc004544fa0}, 0xc00451ad90)
/workspace/source/internal/querynodev2/services.go:807 +0xbce fp=0xc0033a5598 sp=0xc0033a5388 pc=0x52554ee
github.com/milvus-io/milvus/internal/util/wrappers.(*qnServerWrapper).Search(0xc004248960?, {0x68dcb90?, 0xc004544fa0?}, 0x4?, {0x68dcb90?, 0xc004544fa0?, 0xc00446a630?})
/workspace/source/internal/util/wrappers/qn_wrapper.go:93 +0x28 fp=0xc0033a55c8 sp=0xc0033a5598 pc=0x4dc80c8
github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard(0xc004c00420, {0x68dcb90, 0xc004544fa0}, 0x20, {0x695d0a8, 0xc00455e510}, {0xc0013e0f60, 0x2b})
/workspace/source/internal/proxy/task_search.go:867 +0x534 fp=0xc0033a5798 sp=0xc0033a55c8 pc=0x4effb34
github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard-fm({0x68dcb90?, 0xc004544fa0?}, 0xc004544fa0?, {0x695d0a8?, 0xc00455e510?}, {0xc0013e0f60?, 0x2b?})
:1 +0x55 fp=0xc0033a57e8 sp=0xc0033a5798 pc=0x4f5e5d5
github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1()
/workspace/source/internal/proxy/lb_policy.go:210 +0x57d fp=0xc0033a5ae8 sp=0xc0033a57e8 pc=0x4e828dd
github.com/milvus-io/milvus/pkg/v2/util/retry.Do({0x68dcb90, 0xc004544fa0}, 0xc0033a5dd8, {0xc0000addd0, 0x1, 0x1?})
/workspace/source/pkg/util/retry/retry.go:44 +0x14e fp=0xc0033a5cc8 sp=0xc0033a5ae8 pc=0x2e4e74e
github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry(0xc000aa2930, {0x68dcb90, 0xc004544fa0}, {{0x60fd3d0, 0x7}, {0xc003c74d98, 0x4}, 0x65b5fd578a0017a, {0xc0013e0f60, 0x2b}, ...})
/workspace/source/internal/proxy/lb_policy.go:179 +0x1bd fp=0xc0033a5e80 sp=0xc0033a5cc8 pc=0x4e8231d
github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1()
/workspace/source/internal/proxy/lb_policy.go:246 +0x138 fp=0xc0033a5f78 sp=0xc0033a5e80 pc=0x4e83998
golang.org/x/sync/errgroup.(*Group).Go.func1()
/go/pkg/mod/golang.org/x/sync@v0.11.0/errgroup/errgroup.go:78 +0x50 fp=0xc0033a5fe0 sp=0xc0033a5f78 pc=0x35dfbd0
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0033a5fe8 sp=0xc0033a5fe0 pc=0x1f2b4c1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 23548
/go/pkg/mod/golang.org/x/sync@v0.11.0/errgroup/errgroup.go:75 +0x93

rax 0x0
rbx 0x7c4963400640
rcx 0x7c4a0e8779fc
rdx 0x6
rdi 0x8
rsi 0xa3
rbp 0xa3
rsp 0x7c49633e0da0
r8 0x7c49633e0e70
r9 0x27
r10 0x8
r11 0x246
r12 0x6
r13 0x16
r14 0x0
r15 0x7c49633e1098
rip 0x7c4a0e8779fc
rflags 0x246
cs 0x33
fs 0x0
gs 0x0

---

Please let us know if you think this could be related to how input encoding is handled, or if there are any recommended resource thresholds for running Milvus more stably with multilingual inputs.

Thanks again!

[yhmo]

Do you have a full log of the milvus server? You can use the command "docker logs [milvus container id] > 1.log" to output its log.

Do you mean it crashed when you continually call search()/hybridSearch() with Chinese characters in the EmbeddedText like this?

        List<BaseVector> sparseTexts = new ArrayList<>();
        sparseTexts.add(new EmbeddedText("汉字"));

        List<AnnSearchReq> searchRequests = new ArrayList<>();
        searchRequests.add(AnnSearchReq.builder()
                .vectorFieldName("sparse_vector")
                .vectors(sparseTexts)
                .topK(10)
                .build());

        HybridSearchReq hybridSearchReq = HybridSearchReq.builder()
                .collectionName("test")
                .searchRequests(searchRequests)
                .ranker(new RRFRanker(2))
                .topK(5)
                .outFields(Collections.singletonList("text"))
                .consistencyLevel(ConsistencyLevel.BOUNDED)
                .build();

Are there any other special characters in the EmbeddedText?

[yhmo]

Sometimes, a user's disk could be exhausted by log content if the log level is debug.
Double-check the milvus.yaml, ensure the log level is not debug.
Optionally, you can set log.stdout = false and log.file.rootPath = /var/lib/milvus/log to output the log as files.

# Configures the system log output.
log:
  # Milvus log level. Option: debug, info, warn, error, panic, and fatal.
  # It is recommended to use debug level under test and development environments, and info level in production environment.
  level: info
  file:
    # Root path to the log files.
    # The default value is set empty, indicating to output log files to standard output (stdout) and standard error (stderr).
    # If this parameter is set to a valid local path, Milvus writes and stores log files in this path.
    # Set this parameter as the path that you have permission to write.
    rootPath: /var/lib/milvus/log
    maxSize: 300 # The maximum size of a log file, unit: MB.
    maxAge: 10 # The maximum retention time before a log file is automatically cleared, unit: day. The minimum value is 1.
    maxBackups: 20 # The maximum number of log files to back up, unit: day. The minimum value is 1.
  format: text # Milvus log format. Option: text and JSON
  stdout: false # Stdout enable or not

[qidafang0413]

Do you have a full log of the milvus server? You can use the command "docker logs [milvus container id] > 1.log" to output its log.

Do you mean it crashed when you continually call search()/hybridSearch() with Chinese characters in the EmbeddedText like this?

        List<BaseVector> sparseTexts = new ArrayList<>();
        sparseTexts.add(new EmbeddedText("汉字"));

        List<AnnSearchReq> searchRequests = new ArrayList<>();
        searchRequests.add(AnnSearchReq.builder()
                .vectorFieldName("sparse_vector")
                .vectors(sparseTexts)
                .topK(10)
                .build());

        HybridSearchReq hybridSearchReq = HybridSearchReq.builder()
                .collectionName("test")
                .searchRequests(searchRequests)
                .ranker(new RRFRanker(2))
                .topK(5)
                .outFields(Collections.singletonList("text"))
                .consistencyLevel(ConsistencyLevel.BOUNDED)
                .build();

Are there any other special characters in the EmbeddedText?

The original log file was over 1GB with more than 3 million lines, so I’ve extracted only the last 5000 lines for easier review. Also, the input texts only contain basic Chinese characters like "汉字" — no special or unusual characters were used.

2.log

[yhmo]

I didn't get any hints from the log. Do you have the log at the crash point?

These lines only show the goroutines closed by the crashed process. There should be some call-stacks above these goroutines.

github.com/panjf2000/ants/v2.(*goWorker).run.func1()
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:63 +0x6a fp=0xc001534fe0 sp=0xc001534f88 pc=0x3df696a
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc001534fe8 sp=0xc001534fe0 pc=0x1f2b4c1
created by github.com/panjf2000/ants/v2.(*goWorker).run in goroutine 23493
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:48 +0x5c

goroutine 23516 gp=0xc003b99c00 m=nil [select]:
runtime.gopark(0xc002431f48?, 0x2?, 0x6c?, 0x4e?, 0xc002431ef4?)
/usr/local/go/src/runtime/proc.go:435 +0xce fp=0xc003b3fd80 sp=0xc003b3fd60 pc=0x1f2278e
runtime.selectgo(0xc003b3ff48, 0xc002431ef0, 0xc00422c900?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:351 +0x837 fp=0xc003b3feb8 sp=0xc003b3fd80 pc=0x1efdcb7
net/http.(*persistConn).writeLoop(0xc00430d560)
/usr/local/go/src/net/http/transport.go:2590 +0xe7 fp=0xc003b3ffc8 sp=0xc003b3feb8 pc=0x2265e27
net/http.(*Transport).dialConn.gowrap3()
/usr/local/go/src/net/http/transport.go:1945 +0x25 fp=0xc003b3ffe0 sp=0xc003b3ffc8 pc=0x2262bc5
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc003b3ffe8 sp=0xc003b3ffe0 pc=0x1f2b4c1
created by net/http.(*Transport).dialConn in goroutine 23529
/usr/local/go/src/net/http/transport.go:1945 +0x17a5

[xiaofan-luan]

Do you have a full log of the milvus server? You can use the command "docker logs [milvus container id] > 1.log" to output its log.
Do you mean it crashed when you continually call search()/hybridSearch() with Chinese characters in the EmbeddedText like this?

        List<BaseVector> sparseTexts = new ArrayList<>();
        sparseTexts.add(new EmbeddedText("汉字"));

        List<AnnSearchReq> searchRequests = new ArrayList<>();
        searchRequests.add(AnnSearchReq.builder()
                .vectorFieldName("sparse_vector")
                .vectors(sparseTexts)
                .topK(10)
                .build());

        HybridSearchReq hybridSearchReq = HybridSearchReq.builder()
                .collectionName("test")
                .searchRequests(searchRequests)
                .ranker(new RRFRanker(2))
                .topK(5)
                .outFields(Collections.singletonList("text"))
                .consistencyLevel(ConsistencyLevel.BOUNDED)
                .build();

Are there any other special characters in the EmbeddedText?

The original log file was over 1GB with more than 3 million lines, so I’ve extracted only the last 5000 lines for easier review. Also, the input texts only contain basic Chinese characters like "汉字" — no special or unusual characters were used.

2.log

From the log you offered we only see some error message that collection does not exist.

could you check if there are any OOM message or detailed panic information?

[xiaofan-luan]

did you bulk import the data?
we do have a tokenizer leakage issue when bulk import with some tokenzier.

Also you want to search chinese make sure you are using jieba tokenizer