Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2330] [BUG]: Issue with Embedding Text (only 1 chunks) Using nomic-embed-text:latest (or Naive) #1520

New Issue
Closed
opened 2026-02-22 18:25:14 -05:00 by yindo · 1 comment
yindo commented 2026-02-22 18:25:14 -05:00
Owner
Copy Link
Copy Source

Originally created by @Kias-llm on GitHub (Sep 19, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2330

How are you running AnythingLLM?

Docker (local)

What happened?

anythingllm | [backend] info: [DocumentManager] Found 1 pinned sources - prepending to content with ~14426 tokens of content.
ollama | [GIN] 2024/09/20 - 01:14:44 | 200 | 23.585µs | 192.168.176.1 | HEAD "/"
anythingllm | [backend] info: [OllamaEmbedder] Embedding 1 chunks of text with nomic-embed-text:latest.
ollama | [GIN] 2024/09/20 - 01:14:44 | 200 | 107.954705ms | 192.168.176.1 | POST "/api/embeddings"
anythingllm | [backend] info: LanceDB: A source was filtered from context as it's parent document is pinned.
anythingllm | [backend] info: LanceDB: A source was filtered from context as it's parent document is pinned.
anythingllm | [backend] info: LanceDB: A source was filtered from context as it's parent document is pinned.
anythingllm | [backend] info: LanceDB: A source was filtered from context as it's parent document is pinned.
anythingllm | [backend] info: [fillSourceWindow] Need to backfill 4 chunks to fill in the source window for RAG!
anythingllm | [backend] info: Cannonball results 14439 -> 930 tokens.
ollama | [GIN] 2024/09/20 - 01:15:00 | 200 | 14.900320052s | 192.168.176.1 | POST "/api/chat"

There is an issue with the embedding process when using nomic-embed-text:latest. According to the logs, 6-7 chunks of text are being created, but only 1 chunk is successfully embedded. This may lead to potential data loss, as not all chunks are being processed as expected. Additionally, there is no mention in the logs of any configuration settings that might explain or resolve the issue.

P.S.: Thank you for creating such a great application!

Are there known steps to reproduce?

No response

Originally created by @Kias-llm on GitHub (Sep 19, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2330 ### How are you running AnythingLLM? Docker (local) ### What happened? anythingllm | [backend] info: [DocumentManager] Found 1 pinned sources - prepending to content with ~14426 tokens of content. ollama | [GIN] 2024/09/20 - 01:14:44 | 200 | 23.585µs | 192.168.176.1 | HEAD "/" anythingllm | [backend] info: [OllamaEmbedder] **Embedding 1 chunks of text with nomic-embed-text:latest**. ollama | [GIN] 2024/09/20 - 01:14:44 | 200 | 107.954705ms | 192.168.176.1 | POST "/api/embeddings" anythingllm | [backend] info: LanceDB: A source was filtered from context as it's parent document is pinned. anythingllm | [backend] info: LanceDB: A source was filtered from context as it's parent document is pinned. anythingllm | [backend] info: LanceDB: A source was filtered from context as it's parent document is pinned. anythingllm | [backend] info: LanceDB: A source was filtered from context as it's parent document is pinned. anythingllm | [backend] info: [fillSourceWindow] Need to backfill 4 chunks to fill in the source window for RAG! anythingllm | [backend] info: Cannonball results 14439 -> 930 tokens. ollama | [GIN] 2024/09/20 - 01:15:00 | 200 | 14.900320052s | 192.168.176.1 | POST "/api/chat" ------------------------------------------------------------------------------------------------- There is an issue with the embedding process when using nomic-embed-text:latest. According to the logs, 6-7 chunks of text are being created, but only 1 chunk is successfully embedded. This may lead to potential data loss, as not all chunks are being processed as expected. Additionally, there is no mention in the logs of any configuration settings that might explain or resolve the issue. P.S.: Thank you for creating such a great application! ### Are there known steps to reproduce? _No response_
yindo added the possible bug label 2026-02-22 18:25:14 -05:00
yindo closed this issue 2026-02-22 18:25:14 -05:00
yindo commented 2026-02-22 18:25:14 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Sep 20, 2024):

According to the logs, 6-7 chunks of text are being created, but only 1 chunk is successfully embedded.

I am not seeing this in the log attached. This looks to be a log that is produced when sending a chat, which 1 chunk would make sense (since it depends on the length of the prompt) and is used for similarity search.

Which then seems to return some results which are then filtered out because your pinned document must contain them and is highjacking the entire context window since you want the whole document in the chat - but your Ollama window is not large enough so we have to truncate.

From the logs above, nothing glaringly wrong with the embedder, but I would unpin your document since its being truncated due to your limited context window using Ollama

@timothycarambat commented on GitHub (Sep 20, 2024): > According to the logs, 6-7 chunks of text are being created, but only 1 chunk is successfully embedded. I am not seeing this in the log attached. This looks to be a log that is produced when sending a chat, which 1 chunk would make sense (since it depends on the length of the prompt) and is used for similarity search. Which then seems to return some results which are _then_ filtered out because your pinned document must contain them and is highjacking the entire context window since you want the whole document in the chat - but your Ollama window is not large enough so we have to truncate. From the logs above, nothing glaringly wrong with the embedder, but I would unpin your document since its being truncated due to your limited context window using Ollama
yindo changed title from [BUG]: Issue with Embedding Text (only 1 chunks) Using nomic-embed-text:latest (or Naive) to [GH-ISSUE #2330] [BUG]: Issue with Embedding Text (only 1 chunks) Using nomic-embed-text:latest (or Naive) 2026-06-05 14:41:12 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1520
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?