Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-18 19:34:33 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #5403] [FEAT]: Embedder Config: add chunking strategy and embedder query/passage prefix #5071

New Issue
Open
opened 2026-06-05 14:51:50 -04:00 by yindo · 1 comment
yindo commented 2026-06-05 14:51:50 -04:00
Owner
Copy Link
Copy Source

Originally created by @chrisjunlee on GitHub (Apr 10, 2026).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5403

What would you like to see?

For the Embedder, there's two missing configurable parameters:

  1. Chunking strategy: fixed (current default), recursive (model figures out dynamically), structural, etc. You don't have to include the options since they're evolving. Just an optional text field to pass to embedder
  2. Query prefix: models like Qwen3-Embedding-8B require a query and instruct prefix. This could also be a text field in the Embedder settings
Originally created by @chrisjunlee on GitHub (Apr 10, 2026). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5403 ### What would you like to see? For the Embedder, there's two missing configurable parameters: 1. Chunking strategy: fixed (current default), recursive (model figures out dynamically), structural, etc. You don't have to include the options since they're evolving. Just an optional text field to pass to embedder 2. Query prefix: models like Qwen3-Embedding-8B require a [query and instruct prefix](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B/discussions/29). This could also be a text field in the Embedder settings
yindo added the enhancementfeature request labels 2026-06-05 14:51:50 -04:00
yindo commented 2026-06-05 14:51:51 -04:00
Author
Owner
Copy Link
Copy Source

@jhsmith409 commented on GitHub (May 7, 2026):

Confirming this is a real, reproducible accuracy regression for any asymmetric embedding model (Qwen3-Embedding family, BGE-M3 in retrieval mode, E5-instruct, multilingual-e5-*, etc.). Caught it on a deployment serving Qwen3-Embedding-0.6B via an OpenAI-compatible gateway through EMBEDDING_ENGINE=generic-openai.

Source-level evidence

server/utils/EmbeddingEngines/genericOpenAi/index.js passes input straight through with no query/passage distinction:

async embedChunks(textChunks = []) {
  // ...
  this.openai.embeddings
    .create({
      model: this.model,
      input: chunk,
    })
  // ...
}

async embedTextInput(textInput) {
  const result = await this.embedChunks(
    Array.isArray(textInput) ? textInput : [textInput]
  );
  return result?.[0] || [];
}

No env var or constructor option exists to inject a prefix.

Wire-level evidence

Spied on the OpenAI client inside the running container to capture the actual request body:

--- query path (LLMConnector.embedTextInput) ---
WIRE_REQUEST_BODY: {"model":"qwen3-embedding-0.6b","input":["what is the snow load on a barn roof"]}

--- doc path (EmbedderEngine.embedChunks) ---
WIRE_REQUEST_BODY: {"model":"qwen3-embedding-0.6b","input":["Section 7.3.2 specifies a ground snow load of 30 psf for the building."]}

Per the Qwen3-Embedding model card, queries should be wrapped:

Instruct: Given a web search query, retrieve relevant passages that answer the query
Query:{query}

…with passages left raw. Skipping the wrap costs ~1–5% on the asymmetric MTEB tasks the model was trained for.

Why a fix is straightforward (the prefix half of this issue)

The architecture is already prefix-friendly: every vector-DB provider in server/utils/vectorDbProviders/* cleanly separates the two paths — embedTextInput is called only for queries, embedChunks only for ingest. So the change can be local to each embedder class with zero call-site churn. Grep confirms (lance, chroma, qdrant, pinecone, milvus, weaviate, astra, pgvector all follow the pattern):

LLMConnector.embedTextInput(input)        // queries
EmbedderEngine.embedChunks(textChunks)    // ingest

Proposed minimal scope

Two new env vars on the GenericOpenAI embedder (with empty defaults, fully backward-compatible):

  • EMBEDDING_QUERY_PREFIX — prepended to inputs in embedTextInput
  • EMBEDDING_PASSAGE_PREFIX — prepended to inputs in embedChunks (often left empty for Qwen3)

For Qwen3-Embedding the user would set:

EMBEDDING_QUERY_PREFIX="Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:"
EMBEDDING_PASSAGE_PREFIX=

Happy to put up a PR for the prefix half of this issue if maintainers are open to that scoping (chunking-strategy half can stay as a follow-up). Could also extend the same env knobs to the sibling OpenAI-compatible embedders (openai, lmstudio, localai, voyageai, ollama) in the same PR or a follow-up — preference?

<!-- gh-comment-id:4396775933 --> @jhsmith409 commented on GitHub (May 7, 2026): Confirming this is a real, reproducible accuracy regression for any asymmetric embedding model (Qwen3-Embedding family, BGE-M3 in retrieval mode, E5-instruct, multilingual-e5-*, etc.). Caught it on a deployment serving Qwen3-Embedding-0.6B via an OpenAI-compatible gateway through `EMBEDDING_ENGINE=generic-openai`. ### Source-level evidence `server/utils/EmbeddingEngines/genericOpenAi/index.js` passes input straight through with no query/passage distinction: ```js async embedChunks(textChunks = []) { // ... this.openai.embeddings .create({ model: this.model, input: chunk, }) // ... } async embedTextInput(textInput) { const result = await this.embedChunks( Array.isArray(textInput) ? textInput : [textInput] ); return result?.[0] || []; } ``` No env var or constructor option exists to inject a prefix. ### Wire-level evidence Spied on the `OpenAI` client inside the running container to capture the actual request body: ``` --- query path (LLMConnector.embedTextInput) --- WIRE_REQUEST_BODY: {"model":"qwen3-embedding-0.6b","input":["what is the snow load on a barn roof"]} --- doc path (EmbedderEngine.embedChunks) --- WIRE_REQUEST_BODY: {"model":"qwen3-embedding-0.6b","input":["Section 7.3.2 specifies a ground snow load of 30 psf for the building."]} ``` Per the [Qwen3-Embedding model card](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B), queries should be wrapped: ``` Instruct: Given a web search query, retrieve relevant passages that answer the query Query:{query} ``` …with passages left raw. Skipping the wrap costs ~1–5% on the asymmetric MTEB tasks the model was trained for. ### Why a fix is straightforward (the prefix half of this issue) The architecture is already prefix-friendly: every vector-DB provider in `server/utils/vectorDbProviders/*` cleanly separates the two paths — `embedTextInput` is called **only** for queries, `embedChunks` **only** for ingest. So the change can be local to each embedder class with zero call-site churn. Grep confirms (lance, chroma, qdrant, pinecone, milvus, weaviate, astra, pgvector all follow the pattern): ``` LLMConnector.embedTextInput(input) // queries EmbedderEngine.embedChunks(textChunks) // ingest ``` ### Proposed minimal scope Two new env vars on the GenericOpenAI embedder (with empty defaults, fully backward-compatible): - `EMBEDDING_QUERY_PREFIX` — prepended to inputs in `embedTextInput` - `EMBEDDING_PASSAGE_PREFIX` — prepended to inputs in `embedChunks` (often left empty for Qwen3) For Qwen3-Embedding the user would set: ``` EMBEDDING_QUERY_PREFIX="Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:" EMBEDDING_PASSAGE_PREFIX= ``` Happy to put up a PR for the prefix half of this issue if maintainers are open to that scoping (chunking-strategy half can stay as a follow-up). Could also extend the same env knobs to the sibling OpenAI-compatible embedders (openai, lmstudio, localai, voyageai, ollama) in the same PR or a follow-up — preference?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#5071
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?