Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4896] [BUG]: embedding with lanced and llama.cpp --embedding via Generic OpenAI provider results in addDocumentToNamespace Could not embed document chunks! This document will not be recorded. #3081

New Issue
Closed
opened 2026-02-22 18:32:31 -05:00 by yindo · 3 comments
yindo commented 2026-02-22 18:32:31 -05:00
Owner
Copy Link
Copy Source

Originally created by @mdengler on GitHub (Jan 22, 2026).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4896

How are you running AnythingLLM?

Docker (local)

What happened?

Hi,

Love anythingllm; I'm having problem with the Embedding provider of "Generic OpenAI provider" ( a local llamma.cpp embedding model -- the model is overkill for embedding right now, but I don't think that's the issue): when I try to "Move to Workspace" and "Save & Embed", it looks like the response is not getting parsed correctly around https://github.com/chaserhkj/anything-llm/blob/master/server/utils/EmbeddingEngines/genericOpenAi/index.js#L66 , perhaps due to some data format of the returned JSON from llama.cpp (see below).

Can you see an obvious error/problem? What's the best way for me to debug this and/or get you more information?

UI: Image

logged error:

$ docker run -it --publish 5558:3001 --cap-add SYS_ADMIN --mount=type=bind,source=${STORAGE_LOCATION},target=/app/server/storage --mount type=bind,source=${STORAGE_LOCATION}/.env,target=/app/
server/.env -e STORAGE_DIR="/app/server/storage" --add-host=172.17.0.1:host-gateway mintplexlabs/anythingllm --enable-debugging
[...]
[backend] info: [VectorDB::LanceDb] Adding new vectorized document into namespace my-workspace
[backend] info: [GenericOpenAiEmbedder] Initialized ggml-org/gpt-oss-120b-GGUF {"baseURL":"http://172.17.0.1:5555","maxConcurrentChunks":1,"embeddingMaxChunkLength":10000}
[backend] info: [RecursiveSplitter] Will split with {"chunkSize":10000,"chunkOverlap":200,"chunkHeader":"<document_metadata>\nsourceDocument: plan\npublished..."}
[backend] info: [VectorDB::LanceDb] Snippets created from document: 1
[backend] info: [GenericOpenAiEmbedder] Delaying new batch request for 1000ms
[backend] info: [VectorDB::LanceDb] addDocumentToNamespace Could not embed document chunks! This document will not be recorded.
[backend] error: Failed to vectorize plan
[backend] info: [TELEMETRY SENT] {"event":"documents_embedded_in_workspace","distinctId":"d21dfcf9-3b8e-4952-86cc-535b7483edb3","properties":{"LLMSelection":"generic-openai","Embedder":"generic-openai","VectorDbSelection":"lancedb","TTSSelection":"native","LLMModel":"ggml-org/gpt-oss-120b-GGUF","runtime":"docker"}}
[backend] info: [Event Logged] - workspace_documents_added

llama.cpp server log:

$ build/bin/llama-server -hf ggml-org/gpt-oss-120b-GGUF --jinja --no-mmap -c 0 --host 0.0.0.0 --port 5555 --embeddings --ubatch-size 11000 --batch-size 11000  --log-verbose

srv  update_slots: all slots are idle                                                                                 
que    start_loop: waiting for new tasks                                                                              
res  remove_waiti: remove task 0 from waiting list. current waiting = 1 (before remove)                               
srv          stop: all tasks already finished, no need to cancel                                                      
srv  log_server_r: request: POST /embeddings 172.17.0.2 200                                                           
srv  log_server_r: request:  {                                                                                        
  "model": "ggml-org/gpt-oss-120b-GGUF",                                                                              
  "input": [                                                                                                          
    "<document_metadata>\nsourceDocument: plan\npublished: 1/22/2026, 11:42:09 PM\n</document_metadata>\n\nMartin's .plan\n\n- ...."                     
  ],                                                                                                                  
  "encoding_format": "base64"                                                                                         
}                                                                                                                     
srv  log_server_r: response: [{"index":0,"embedding":[[18.170156478881836,2.133310317993164,-4.037582874298096,5.61179
8286437988,4.367753982543945,-2.053842782974243,1.1467355489730835,0.9587997794151306,-6.66697883605957,
[...]
0.18568067252635956,2.3528289794921875,-0.560083270072937,0.4906448721885681,-3.766796588897705,-1.5748509168624878]]}]

AnythingLLM .env file:

# Auto-dump ENV from system call on 23:29:31 GMT+0000 (Coordinated Universal Time)                                    LLM_PROVIDER='generic-openai'                                                                                         OPEN_AI_KEY='dummy'                                                                                                   EMBEDDING_MODEL_PREF='ggml-org/gpt-oss-120b-GGUF'                                                                     OLLAMA_BASE_PATH='http://172.17.0.1:5555'                                                                             OLLAMA_MODEL_PREF='ggml-org/gpt-oss-120b-GGUF'                                                                        OLLAMA_PERFORMANCE_MODE='base'                                                                                        OLLAMA_KEEP_ALIVE_TIMEOUT='-1'                                                                                        OLLAMA_AUTH_TOKEN='dummy'                                                                                             GENERIC_OPEN_AI_BASE_PATH='http://172.17.0.1:5555'                                                                    GENERIC_OPEN_AI_MODEL_PREF='ggml-org/gpt-oss-120b-GGUF'                                                               GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT='131072'                                                                            GENERIC_OPEN_AI_API_KEY='dummy'                                                                                       GENERIC_OPEN_AI_MAX_TOKENS='131072'                                                                                   EMBEDDING_ENGINE='generic-openai'                                                                                     EMBEDDING_BASE_PATH='http://172.17.0.1:5555'                                                                          EMBEDDING_MODEL_MAX_CHUNK_LENGTH='10000'                                                                              GENERIC_OPEN_AI_EMBEDDING_API_KEY='dummy'                                                                             GENERIC_OPEN_AI_EMBEDDING_MAX_CONCURRENT_CHUNKS='1'                                                                   GENERIC_OPEN_AI_EMBEDDING_API_DELAY_MS=1000                                                                           VECTOR_DB='lancedb'                                                                                                   WHISPER_PROVIDER='local'                                                                                              WHISPER_MODEL_PREF='Xenova/whisper-large'                                                                             STORAGE_DIR='/app/server/storage'                                                                                     SIG_KEY='a0ce12ba4c101775964b279323de6a1176520fd2e4c00d5aeedb93c16c052f80'                                            SIG_SALT='c6d71e554bc0a1f2e8b03b43c5e6520e7b927cc11297ddfd7e7be7ce11f3caeb'

Are there known steps to reproduce?

No response

Originally created by @mdengler on GitHub (Jan 22, 2026). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4896 ### How are you running AnythingLLM? Docker (local) ### What happened? Hi, Love anythingllm; I'm having problem with the Embedding provider of "Generic OpenAI provider" ( a local llamma.cpp embedding model -- the model is overkill for embedding right now, but I don't think that's the issue): when I try to "Move to Workspace" and "Save & Embed", it looks like the response is not getting parsed correctly around https://github.com/chaserhkj/anything-llm/blob/master/server/utils/EmbeddingEngines/genericOpenAi/index.js#L66 , perhaps due to some data format of the returned JSON from llama.cpp (see below). Can you see an obvious error/problem? What's the best way for me to debug this and/or get you more information? UI: <img width="587" height="475" alt="Image" src="https://github.com/user-attachments/assets/d310b2bf-bc9e-4cbd-9192-228bd9a85dd2" /> logged error: ``` $ docker run -it --publish 5558:3001 --cap-add SYS_ADMIN --mount=type=bind,source=${STORAGE_LOCATION},target=/app/server/storage --mount type=bind,source=${STORAGE_LOCATION}/.env,target=/app/ server/.env -e STORAGE_DIR="/app/server/storage" --add-host=172.17.0.1:host-gateway mintplexlabs/anythingllm --enable-debugging [...] [backend] info: [VectorDB::LanceDb] Adding new vectorized document into namespace my-workspace [backend] info: [GenericOpenAiEmbedder] Initialized ggml-org/gpt-oss-120b-GGUF {"baseURL":"http://172.17.0.1:5555","maxConcurrentChunks":1,"embeddingMaxChunkLength":10000} [backend] info: [RecursiveSplitter] Will split with {"chunkSize":10000,"chunkOverlap":200,"chunkHeader":"<document_metadata>\nsourceDocument: plan\npublished..."} [backend] info: [VectorDB::LanceDb] Snippets created from document: 1 [backend] info: [GenericOpenAiEmbedder] Delaying new batch request for 1000ms [backend] info: [VectorDB::LanceDb] addDocumentToNamespace Could not embed document chunks! This document will not be recorded. [backend] error: Failed to vectorize plan [backend] info: [TELEMETRY SENT] {"event":"documents_embedded_in_workspace","distinctId":"d21dfcf9-3b8e-4952-86cc-535b7483edb3","properties":{"LLMSelection":"generic-openai","Embedder":"generic-openai","VectorDbSelection":"lancedb","TTSSelection":"native","LLMModel":"ggml-org/gpt-oss-120b-GGUF","runtime":"docker"}} [backend] info: [Event Logged] - workspace_documents_added ``` llama.cpp server log: ``` $ build/bin/llama-server -hf ggml-org/gpt-oss-120b-GGUF --jinja --no-mmap -c 0 --host 0.0.0.0 --port 5555 --embeddings --ubatch-size 11000 --batch-size 11000 --log-verbose srv update_slots: all slots are idle que start_loop: waiting for new tasks res remove_waiti: remove task 0 from waiting list. current waiting = 1 (before remove) srv stop: all tasks already finished, no need to cancel srv log_server_r: request: POST /embeddings 172.17.0.2 200 srv log_server_r: request: { "model": "ggml-org/gpt-oss-120b-GGUF", "input": [ "<document_metadata>\nsourceDocument: plan\npublished: 1/22/2026, 11:42:09 PM\n</document_metadata>\n\nMartin's .plan\n\n- ...." ], "encoding_format": "base64" } srv log_server_r: response: [{"index":0,"embedding":[[18.170156478881836,2.133310317993164,-4.037582874298096,5.61179 8286437988,4.367753982543945,-2.053842782974243,1.1467355489730835,0.9587997794151306,-6.66697883605957, [...] 0.18568067252635956,2.3528289794921875,-0.560083270072937,0.4906448721885681,-3.766796588897705,-1.5748509168624878]]}] ``` AnythingLLM `.env` file: ``` # Auto-dump ENV from system call on 23:29:31 GMT+0000 (Coordinated Universal Time) LLM_PROVIDER='generic-openai' OPEN_AI_KEY='dummy' EMBEDDING_MODEL_PREF='ggml-org/gpt-oss-120b-GGUF' OLLAMA_BASE_PATH='http://172.17.0.1:5555' OLLAMA_MODEL_PREF='ggml-org/gpt-oss-120b-GGUF' OLLAMA_PERFORMANCE_MODE='base' OLLAMA_KEEP_ALIVE_TIMEOUT='-1' OLLAMA_AUTH_TOKEN='dummy' GENERIC_OPEN_AI_BASE_PATH='http://172.17.0.1:5555' GENERIC_OPEN_AI_MODEL_PREF='ggml-org/gpt-oss-120b-GGUF' GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT='131072' GENERIC_OPEN_AI_API_KEY='dummy' GENERIC_OPEN_AI_MAX_TOKENS='131072' EMBEDDING_ENGINE='generic-openai' EMBEDDING_BASE_PATH='http://172.17.0.1:5555' EMBEDDING_MODEL_MAX_CHUNK_LENGTH='10000' GENERIC_OPEN_AI_EMBEDDING_API_KEY='dummy' GENERIC_OPEN_AI_EMBEDDING_MAX_CONCURRENT_CHUNKS='1' GENERIC_OPEN_AI_EMBEDDING_API_DELAY_MS=1000 VECTOR_DB='lancedb' WHISPER_PROVIDER='local' WHISPER_MODEL_PREF='Xenova/whisper-large' STORAGE_DIR='/app/server/storage' SIG_KEY='a0ce12ba4c101775964b279323de6a1176520fd2e4c00d5aeedb93c16c052f80' SIG_SALT='c6d71e554bc0a1f2e8b03b43c5e6520e7b927cc11297ddfd7e7be7ce11f3caeb' ``` ### Are there known steps to reproduce? _No response_
yindo added the possible bug label 2026-02-22 18:32:31 -05:00
yindo closed this issue 2026-02-22 18:32:31 -05:00
yindo commented 2026-02-22 18:32:31 -05:00
Author
Owner
Copy Link
Copy Source

@mdengler commented on GitHub (Jan 22, 2026):

Apologies -- the EMBEDDING_BASE_PATH='http://172.17.0.1:5555' was the problem; the right value is EMBEDDING_BASE_PATH='http://172.17.0.1:5555/v1'

@mdengler commented on GitHub (Jan 22, 2026): Apologies -- the `EMBEDDING_BASE_PATH='http://172.17.0.1:5555'` was the problem; the right value is `EMBEDDING_BASE_PATH='http://172.17.0.1:5555/v1'`
yindo commented 2026-02-22 18:32:32 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 23, 2026):

Just a question because i wasnt aware of this, but ggml-org/gpt-oss-120b-GGUF as the embedding model seems odd since gpt-oss is not an embedding model? It is a text generation model, so I am not really sure if your embeddings are accurate or not...

Embedding model examples are:
BAAI/bge-m3 (Great multilingual support)
nomic-ai/nomic-embed-text-v1.5 (Good balance of speed/quality)

Just wondering if that is working (also how is that even working 😅 )

@timothycarambat commented on GitHub (Jan 23, 2026): Just a question because i wasnt aware of this, but `ggml-org/gpt-oss-120b-GGUF` as the embedding model seems odd since gpt-oss is not an embedding model? It is a text generation model, so I am not really sure if your embeddings are accurate or not... Embedding model examples are: BAAI/bge-m3 (Great multilingual support) nomic-ai/nomic-embed-text-v1.5 (Good balance of speed/quality) Just wondering if that is working (also how is that even working 😅 )
yindo commented 2026-02-22 18:32:32 -05:00
Author
Owner
Copy Link
Copy Source

@mdengler commented on GitHub (Feb 1, 2026):

Just a question because i wasnt aware of this, but ggml-org/gpt-oss-120b-GGUF as the embedding model seems odd since gpt-oss is not an embedding model? It is a text generation model, so I am not really sure if your embeddings are accurate or not...

At first I was really naive :) -- having learned a little more, it is indeed unusual, but it's working "fine" (the embeddings are reasonable -- I was mainly working on the chunking for a RAG test, anyway, so getting better embeddings was not too critical). It's definitely engineering-wise questionable, because it's complete overkill and not a customized embedding model. I was running it anyway.

Embedding model examples are: BAAI/bge-m3 (Great multilingual support) nomic-ai/nomic-embed-text-v1.5 (Good balance of speed/quality)

Thanks for the pointers!

Just wondering if that is working (also how is that even working 😅 )

Hah :). gpt-oss is a transformer so llama.cpp can extract the embeddings "OK" (tbh I haven't sanity-checked the vectors themselves, but I can see that the vector-search results are sane).

@mdengler commented on GitHub (Feb 1, 2026): > Just a question because i wasnt aware of this, but `ggml-org/gpt-oss-120b-GGUF` as the embedding model seems odd since gpt-oss is not an embedding model? It is a text generation model, so I am not really sure if your embeddings are accurate or not... At first I was really naive :) -- having learned a little more, it is indeed unusual, but it's working "fine" (the embeddings are reasonable -- I was mainly working on the chunking for a RAG test, anyway, so getting better embeddings was not too critical). It's definitely engineering-wise questionable, because it's complete overkill and not a customized embedding model. I was running it anyway. > Embedding model examples are: BAAI/bge-m3 (Great multilingual support) nomic-ai/nomic-embed-text-v1.5 (Good balance of speed/quality) Thanks for the pointers! > Just wondering if that is working (also how is that even working 😅 ) Hah :). gpt-oss is a transformer so llama.cpp can extract the embeddings "OK" (tbh I haven't sanity-checked the vectors themselves, but I can see that the vector-search results are sane).
yindo changed title from [BUG]: embedding with lanced and llama.cpp --embedding via Generic OpenAI provider results in addDocumentToNamespace Could not embed document chunks! This document will not be recorded. to [GH-ISSUE #4896] [BUG]: embedding with lanced and llama.cpp --embedding via Generic OpenAI provider results in addDocumentToNamespace Could not embed document chunks! This document will not be recorded. 2026-06-05 14:50:11 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#3081
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?