Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4920] [BUG]: AnythingLLM creates embeddings of wrong size in qdrant #3091

New Issue
Closed
opened 2026-02-22 18:32:35 -05:00 by yindo · 4 comments
yindo commented 2026-02-22 18:32:35 -05:00
Owner
Copy Link
Copy Source

Originally created by @walcz-de on GitHub (Jan 28, 2026).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4920

Originally assigned to: @timothycarambat on GitHub.

How are you running AnythingLLM?

Docker (remote machine)

What happened?

I have been having this issue for quite a bit but could not find any resources pointing to a solution I haven't been over and over again. The behavior is identical on Docker (remote) and Docker Desktop (Mac).

Environment:
Dockerized
Anythingllm 1.10.0 (but observed with 1.9.0 and 1.9.1)
Localai 3.10
qdrant 1.16

Observations.
Models observed: any model but I will keep it to
jinai-embeddings-Q8_0.gguf
This model supports different dimensions and ist set to use 1024 in localai with llama.cpp backend
Collection is created with 256 dimensions

Testing with different models left the impression that anythingllm always "decides" the right dimension by 4

Measures already taken
Deep analysis of localai and qdrant and removal of any unnecessary config

  • Localai recognizes the correct dimensions of the model
  • qdrant can be used successfully if embeddings do not go through anythingllm

Expected behavior
Anythingllm creates embeddings correctly through localai.

Are there known steps to reproduce?

Yes replicate the setup and upload documents

Originally created by @walcz-de on GitHub (Jan 28, 2026). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4920 Originally assigned to: @timothycarambat on GitHub. ### How are you running AnythingLLM? Docker (remote machine) ### What happened? I have been having this issue for quite a bit but could not find any resources pointing to a solution I haven't been over and over again. The behavior is identical on Docker (remote) and Docker Desktop (Mac). **Environment:** Dockerized Anythingllm 1.10.0 (but observed with 1.9.0 and 1.9.1) Localai 3.10 qdrant 1.16 **Observations.** Models observed: any model but I will keep it to jinai-embeddings-Q8_0.gguf This model supports different dimensions and ist set to use 1024 in localai with llama.cpp backend Collection is created with 256 dimensions Testing with different models left the impression that anythingllm always "decides" the right dimension by 4 **Measures already taken** Deep analysis of localai and qdrant and removal of any unnecessary config - Localai recognizes the correct dimensions of the model - qdrant can be used successfully if embeddings do not go through anythingllm **Expected behavior** Anythingllm creates embeddings correctly through localai. ### Are there known steps to reproduce? Yes replicate the setup and upload documents
yindo added the possible bugneeds info / can't replicateinvestigating labels 2026-02-22 18:32:35 -05:00
yindo closed this issue 2026-02-22 18:32:35 -05:00
yindo commented 2026-02-22 18:32:35 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Feb 5, 2026):

In the UI can you screenshot the embedder preference you have configured (omit any URL/KEY if present).

Image

Is the JINA v4 or another model - the GGUF does not make that clear. v3 & v4 are the ones with adjustable dimensions. We dont pass a dimension output parameter into the output config - we are using the openai client for LocalAI, so no value is auto-configured here either.

For qdrant, on collection creation we specify the dimensions (based on first record to upsert). It should throw an error if dimensions were mismatched.

@timothycarambat commented on GitHub (Feb 5, 2026): In the UI can you screenshot the embedder preference you have configured (omit any URL/KEY if present). <img width="1689" height="1106" alt="Image" src="https://github.com/user-attachments/assets/eaef64eb-5a10-4421-af22-24cfd3847083" /> Is the JINA v4 or another model - the GGUF does not make that clear. v3 & v4 are the ones with adjustable dimensions. We dont pass a dimension output parameter into the output config - we are using the openai client for LocalAI, so no value is auto-configured here either. For qdrant, on collection creation we specify the dimensions (based on first record to upsert). It _should_ throw an error if dimensions were mismatched.
yindo commented 2026-02-22 18:32:35 -05:00
Author
Owner
Copy Link
Copy Source

@walcz-de commented on GitHub (Feb 6, 2026):

Hi Timothy,

Here comes the screenshot.

Image

And here is my YAML config from localai:

name: jina-embeddings-v3-Q8_0
backend: llama-cpp
description: Jina Embeddings v3 (GGUF, Q8_0) - GPU-optimiert für ROCm/AMD
known_usecases:

  • embedding
    options:
  • embeddings
    parameters:
    model: jina-embeddings-v3-Q8_0.gguf
    embedding: true
    normalize: true
    dimensions: 1024
    threads: 8
    n_ctx: 8192
    batch: 2048
    ubatch: 512
    gpu_layers: -1
    mmap: true
    f16_kv: true
    flash_attn: true
    mlc: true
    warm_up: true
    use_jinja: true
    template:
    use_tokenizer_template: true

As you can see, it is a GGUF of version 3. That said, I was able to reproduce the same behavior with any other embedding model I used before: jina-embeddings-v2, bge-m3, etc. I always ended up with /4.

I built other ingestion pipelines, used other clients with the same localai setup, and they produced embeddings of the right dimensions.

Please also find attached a PDF supplementing the screenshot, which documents the steps when creating a workspace in AnythingLLM, embedding a PDF using AnythingLLM, and what is created in Qdrant.
Long story short:
The collection in Qdrant is only created when embedding the first document into the workspace created in AnythingLLM, and then instead of 1024 dims (which is the size I want to use), I get a collection of 256 dims.

Creating the collection manually in Qdrant and then creating the workspace in AnythingLLM, and then trying to embed a document leads to an error because of dimension mismatch, which is the correct behavior given the mismatching vectors.

screens.pdf

@walcz-de commented on GitHub (Feb 6, 2026): Hi Timothy, Here comes the screenshot. <img width="2672" height="1448" alt="Image" src="https://github.com/user-attachments/assets/5f9f1405-947d-44ee-89a1-05ef47daf87a" /> And here is my YAML config from localai: name: jina-embeddings-v3-Q8_0 backend: llama-cpp description: Jina Embeddings v3 (GGUF, Q8_0) - GPU-optimiert für ROCm/AMD known_usecases: - embedding options: - embeddings parameters: model: jina-embeddings-v3-Q8_0.gguf embedding: true normalize: true dimensions: 1024 threads: 8 n_ctx: 8192 batch: 2048 ubatch: 512 gpu_layers: -1 mmap: true f16_kv: true flash_attn: true mlc: true warm_up: true use_jinja: true template: use_tokenizer_template: true As you can see, it is a GGUF of version 3. That said, I was able to reproduce the same behavior with any other embedding model I used before: jina-embeddings-v2, bge-m3, etc. I always ended up with <model dim>/4. I built other ingestion pipelines, used other clients with the same localai setup, and they produced embeddings of the right dimensions. Please also find attached a PDF supplementing the screenshot, which documents the steps when creating a workspace in AnythingLLM, embedding a PDF using AnythingLLM, and what is created in Qdrant. Long story short: The collection in Qdrant is only created when embedding the first document into the workspace created in AnythingLLM, and then instead of 1024 dims (which is the size I want to use), I get a collection of 256 dims. Creating the collection manually in Qdrant and then creating the workspace in AnythingLLM, and then trying to embed a document leads to an error because of dimension mismatch, which is the correct behavior given the mismatching vectors. [screens.pdf](https://github.com/user-attachments/files/25127775/screens.pdf)
yindo commented 2026-02-22 18:32:35 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Feb 6, 2026):

I am going to need to bootup LocalAI to debug this and pull that model - this is so weird!

@timothycarambat commented on GitHub (Feb 6, 2026): I am going to need to bootup LocalAI to debug this and pull that model - this is so weird!
yindo commented 2026-02-22 18:32:35 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Feb 10, 2026):

@walcz-de Just merged in #4980 which will allow you to force specify the dimensions in request since apparently they are not implied by the YAML. Should fix this issue.

Image
@timothycarambat commented on GitHub (Feb 10, 2026): @walcz-de Just merged in #4980 which will allow you to force specify the dimensions in request since apparently they are not implied by the YAML. Should fix this issue. <img width="1431" height="463" alt="Image" src="https://github.com/user-attachments/assets/80758081-0c69-44ee-9dde-cc6ac2477596" />
yindo changed title from [BUG]: AnythingLLM creates embeddings of wrong size in qdrant to [GH-ISSUE #4920] [BUG]: AnythingLLM creates embeddings of wrong size in qdrant 2026-06-05 14:50:16 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label investigating needs info / can't replicate possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#3091
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?