Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1675] [BUG]: using an embedding model serving with VLLM generate issue during usage #1093

New Issue
Closed
opened 2026-02-22 18:23:06 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:23:06 -05:00
Owner
Copy Link
Copy Source

Originally created by @akkulum on GitHub (Jun 13, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1675

How are you running AnythingLLM?

Docker (remote machine)

What happened?

I use VLLM to serve the embedding model SFR-Embedding-Mistral
I define this embedding in the Embedding Preference using the Local AI provider (and get automatically the running Model Name).

I create New workspace and embed a basic txt file.
When Asking a question I see in the instance of VLLM logs that the embedding model response correctly but in AnythingLLM I get the following issue :

LanceDBError: LanceError(IO): Execution error: LanceError(IO): Execution error: LanceError(IO): Execution error: Failed to compute scores: LanceError(Arrow): Invalid argument error: all columns in a record batch must have the same length

image

This is the kind of response that I get from the embedding model :

{
    "id": "cmpl-2b68cee...",
    "object": "list",
    "created": 407104,
    "model": "/SFR-Embedding-Mistral/",
    "data": [
        {
            "index": 0,
            "object": "embedding",
            "embedding": [
                0.0162200927734375,
                0.00872802734375,
                0.00435638427734375,
                0.0008563995361328125,
                -0.0070343017578125,
                -0.019927978515625,
                0.0008945465087890625,
               ....
             ]
         }
    ],
    "usage": {
        "prompt_tokens": 18,
        "total_tokens": 18,
        "completion_tokens": 0
    }
}

At the beginning I thought that it was because of the embedding values that did not have the same number of decimal But I try with other embedding model same result.
I check special parameter for VLLM to harmonize the values but didn't find anything.

Could you please tell me if this is a normal behavior or is there something wrong with anythingLLM ?

Thanks for your support!

Are there known steps to reproduce?

No response

Originally created by @akkulum on GitHub (Jun 13, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1675 ### How are you running AnythingLLM? Docker (remote machine) ### What happened? I use VLLM to serve the embedding model [SFR-Embedding-Mistral](https://huggingface.co/Salesforce/SFR-Embedding-Mistral) I define this embedding in the Embedding Preference using the **Local AI** provider (and get automatically the running Model Name). I create New workspace and embed a basic txt file. When Asking a question I see in the instance of VLLM logs that the embedding model response correctly but in AnythingLLM I get the following issue : > LanceDBError: LanceError(IO): Execution error: LanceError(IO): Execution error: LanceError(IO): Execution error: Failed to compute scores: LanceError(Arrow): Invalid argument error: all columns in a record batch must have the same length <img width="370" alt="image" src="https://github.com/Mintplex-Labs/anything-llm/assets/172613148/72c809c5-f268-43ca-9551-ec27eeed6fb9"> This is the kind of response that I get from the embedding model : ````json { "id": "cmpl-2b68cee...", "object": "list", "created": 407104, "model": "/SFR-Embedding-Mistral/", "data": [ { "index": 0, "object": "embedding", "embedding": [ 0.0162200927734375, 0.00872802734375, 0.00435638427734375, 0.0008563995361328125, -0.0070343017578125, -0.019927978515625, 0.0008945465087890625, .... ] } ], "usage": { "prompt_tokens": 18, "total_tokens": 18, "completion_tokens": 0 } } ```` At the beginning I thought that it was because of the embedding values that did not have the same number of decimal But I try with other embedding model same result. I check special parameter for VLLM to harmonize the values but didn't find anything. Could you please tell me if this is a normal behavior or is there something wrong with anythingLLM ? Thanks for your support! ### Are there known steps to reproduce? _No response_
yindo added the possible bug label 2026-02-22 18:23:06 -05:00
yindo closed this issue 2026-02-22 18:23:06 -05:00
yindo commented 2026-02-22 18:23:06 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 13, 2024):

Looks like VLLM is working as intended but the Workspaces vectordb's very first record was not the dimension of the output of whatever the current embedder is. Did you even use any other embedder - even the native?

To solve this quickly, you can go to workspace settings -> Vector Database -> Reset Vector Database

all columns in a record batch must have the same length

It indicates that whatever you are attempting to insert is not the schema lancedb expects and its schema is set on the very first upserted vector (first document embedded to workspace).

@timothycarambat commented on GitHub (Jun 13, 2024): Looks like VLLM is working as intended but the Workspaces vectordb's _very first record_ was not the dimension of the output of whatever the current embedder is. Did you even use any other embedder - even the native? To solve this quickly, you can go to workspace settings -> Vector Database -> Reset Vector Database > all columns in a record batch must have the same length It indicates that whatever you are attempting to insert is not the schema lancedb expects and its schema is set on the very first upserted vector (first document embedded to workspace).
yindo commented 2026-02-22 18:23:06 -05:00
Author
Owner
Copy Link
Copy Source

@akkulum commented on GitHub (Jun 14, 2024):

Yeah you are right I was using the native AnythingLLM embedder before trying with this one And now it's clear :)
Thanks for your quick response !

I didn't find this Reset Vector Database option but manually delete all elements from vector-cache (inside the mounted volume).
Now it's working 👍
Thanks again

@akkulum commented on GitHub (Jun 14, 2024): Yeah you are right I was using the native AnythingLLM embedder before trying with this one And now it's clear :) Thanks for your quick response ! I didn't find this Reset Vector Database option but manually delete all elements from vector-cache (inside the mounted volume). Now it's working 👍 Thanks again
yindo changed title from [BUG]: using an embedding model serving with VLLM generate issue during usage to [GH-ISSUE #1675] [BUG]: using an embedding model serving with VLLM generate issue during usage 2026-06-05 14:38:56 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1093
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?