langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 03:55:25 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Multimodal KB: Image query does not rank identical image highest, but image URL query returns score 1.0 #21848

New Issue
Open
opened 2026-02-21 20:14:34 -05:00 by yindo · 11 comments
yindo commented 2026-02-21 20:14:34 -05:00
Owner
Copy Link
Copy Source

Originally created by @unmo on GitHub (Jan 22, 2026).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.11.4

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. Upload a DOCX file containing multiple images (each image is chunked as a standalone image chunk).
  2. Multimodal Knowledge Base settings
  • Index method: High Quality
  • Embedding model: Vertex AI multimodalembedding@001
  • Retrieval: Vector search
  • Rerank: OFF
  1. Search Test.
  2. Upload an image that exists in the KB (e.g., orange image) and run the search.
  3. Run another search using the image URL string as text query, e.g.: ![image](http://<host>/files/<id>/file-preview)

✔️ Expected Behavior

When uploading an image that exists in the knowledge base, the same image chunk should be ranked #1 with the highest score.

So knowledge ui have tag [multimodal]

❌ Actual Behavior

  • When uploading an image that exists in the knowledge base, a different image chunk is ranked higher.
    • Example scores:
      • same image chunk: 0.04
      • different image chunk: 0.03
  • When searching using the image URL string (as text), the result returns very high scores (e.g., 1.0 / 0.9+).
Originally created by @unmo on GitHub (Jan 22, 2026). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.11.4 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. Upload a DOCX file containing multiple images (each image is chunked as a standalone image chunk). 2. Multimodal Knowledge Base settings * Index method: High Quality * Embedding model: Vertex AI multimodalembedding@001 * Retrieval: Vector search * Rerank: OFF 3. Search Test. 4. Upload an image that exists in the KB (e.g., orange image) and run the search. 5. Run another search using the image URL string as text query, e.g.: `![image](http://<host>/files/<id>/file-preview) ` ### ✔️ Expected Behavior When uploading an image that exists in the knowledge base, the same image chunk should be ranked #1 with the highest score. So knowledge ui have tag [multimodal] ### ❌ Actual Behavior * When uploading an image that exists in the knowledge base, a different image chunk is ranked higher. * Example scores: * same image chunk: 0.04 * different image chunk: 0.03 * When searching using the image URL string (as text), the result returns very high scores (e.g., 1.0 / 0.9+).
yindo added the 👻 feat:rag label 2026-02-21 20:14:34 -05:00
yindo commented 2026-02-21 20:14:34 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 22, 2026):

This is a known issue in Dify's multimodal knowledge base search with Vertex AI multimodalembedding@001: image file queries and image URL (text) queries follow different embedding paths, which leads to inconsistent ranking.

When you upload an image, Dify encodes the file and generates an embedding using the multimodal model. When you search using an image URL as text, Dify embeds the URL string as plain text. If the chunk in your KB contains the same signed URL, the text embedding will match almost perfectly, resulting in artificially high scores (e.g., 1.0), even though this does not reflect true image similarity. This is why image URL queries as text can "cheat" the vector search and always rank highest, while actual image queries rely on the multimodal embedding and may not rank the identical image chunk highest due to subtle differences in how embeddings are generated and stored for images vs. text queries. The two pipelines use different cache keys and embedding logic, which further contributes to the discrepancy [source 1] [source 2].

There is no configuration workaround in v1.11.4 to harmonize these behaviors. Disabling rerank means ranking depends solely on vector similarity, which is affected by this pipeline split. The best mitigation is to avoid using image URLs as text queries for evaluation, and rely on image file queries for real similarity search. For maintainers, a long-term fix would involve harmonizing how image and text queries are embedded and compared, or filtering out signed URLs from chunk text before embedding.

You can track related changes and fixes in PRs like #29115 and #30210, and monitor for future updates addressing this ranking inconsistency.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Jan 22, 2026): <!-- Answer --> This is a known issue in Dify's multimodal knowledge base search with Vertex AI multimodalembedding@001: image file queries and image URL (text) queries follow different embedding paths, which leads to inconsistent ranking. When you upload an image, Dify encodes the file and generates an embedding using the multimodal model. When you search using an image URL as text, Dify embeds the URL string as plain text. If the chunk in your KB contains the same signed URL, the text embedding will match almost perfectly, resulting in artificially high scores (e.g., 1.0), even though this does not reflect true image similarity. This is why image URL queries as text can "cheat" the vector search and always rank highest, while actual image queries rely on the multimodal embedding and may not rank the identical image chunk highest due to subtle differences in how embeddings are generated and stored for images vs. text queries. The two pipelines use different cache keys and embedding logic, which further contributes to the discrepancy [[source 1]](https://github.com/langgenius/dify/blob/61f8647f374ed38003e5b34d213487c34b113d61/api/core/rag/embedding/cached_embedding.py#L28-L230) [[source 2]](https://github.com/langgenius/dify/blob/61f8647f374ed38003e5b34d213487c34b113d61/api/core/rag/datasource/vdb/vector_factory.py#L38-L331). There is no configuration workaround in v1.11.4 to harmonize these behaviors. Disabling rerank means ranking depends solely on vector similarity, which is affected by this pipeline split. The best mitigation is to avoid using image URLs as text queries for evaluation, and rely on image file queries for real similarity search. For maintainers, a long-term fix would involve harmonizing how image and text queries are embedded and compared, or filtering out signed URLs from chunk text before embedding. You can track related changes and fixes in PRs like [#29115](https://github.com/langgenius/dify/pull/29115) and [#30210](https://github.com/langgenius/dify/pull/30210), and monitor for future updates addressing this ranking inconsistency. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/8e0817bc-ca16-4077-9e81-a132bbae5946?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/8e0817bc-ca16-4077-9e81-a132bbae5946?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/8e0817bc-ca16-4077-9e81-a132bbae5946?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/8e0817bc-ca16-4077-9e81-a132bbae5946?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/8e0817bc-ca16-4077-9e81-a132bbae5946?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/8e0817bc-ca16-4077-9e81-a132bbae5946?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/8e0817bc-ca16-4077-9e81-a132bbae5946?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=This%20is%20a%20known%20issue%20in%20Dify%27s%20multimodal%20knowledge%20base%20search%20with%20Vertex%20AI%20multimodalembedding%40001%3A%20image%20file%20queries%20and%20image%20URL%20%28text%29%20queries%20follow%20different%20embedding%20paths%2C%20which%20leads%20to%20inconsistent%20ranking.%0A%0AWhen%20you%20upload%20an%20image%2C%20Dify%20encodes%20the%20file%20and%20generates%20an%20embedding%20using%20the%20multimodal%20model.%20When%20you%20search%20using%20an%20image%20URL%20as%20text%2C%20Dify%20embeds%20the%20URL%20string%20as%20plain%20text.%20If%20the%20chunk%20in%20your%20KB%20contains%20the%20same%20signed%20URL%2C%20the%20text%20embedding%20will%20match%20almost%20perfectly%2C%20resulting%20in%20artificially%20high%20scores%20%28e.g.%2C%201.0%29%2C%20even%20though%20this%20does%20not%20reflect%20true%20image%20similarity.%20This%20is%20why%20image%20URL%20queries%20as%20text%20can%20%22cheat%22%20the%20vector%20search%20and%20always%20rank%20highest%2C%20while%20actual%20image%20queries%20rely%20on%20the%20multimodal%20embedding%20and%20may%20not%20rank%20the%20identical%20image%20chunk%20highest%20due%20to%20subtle%20differences%20in%20how%20embeddings%20are%20generated%20and%20stored%20for%20images%20vs.%20text%20queries.%20The%20two%20pipelines%20use%20different%20cache%20keys%20and%20embedding%20logic%2C%20which%20further%20contributes%20to%20the%20discrepancy%20%5B%5Bsource%201%5D%5D%28https%3A//github.com/langgenius/dify/blob/61f8647f374ed38003e5b34d213487c34b113d61/api/core/rag/embedding/cached_embedding.py%23L28-L230%29%20%5B%5Bsource%202%5D%5D%28https%3A//github.com/langgenius/dify/blob/61f8647f374ed38003e5b34d213487c34b113d61/api/core/rag/datasource/vdb/vector_factory.py%23L38-L331%29.%0A%0AThere%20is%20no%20configuration%20workaround%20in%20v1.11.4%20to%20harmonize%20these%20behaviors.%20Disabling%20rerank%20means%20ranking%20depends%20solely%20on%20vector%20similarity%2C%20which%20is%20affected%20by%20this%20pipeline%20split.%20The%20best%20mitigation%20is%20to%20avoid%20using%20image%20URLs%20as%20text%20queries%20for%20evaluation%2C%20and%20rely%20on%20image%20file%20queries%20for%20real%20similarity%20search.%20For%20maintainers%2C%20a%20long-term%20fix%20would%20involve%20harmonizing%20how%20image%20and%20text%20queries%20are%20embedded%20and%20compared%2C%20or%20filtering%20out%20signed%20URLs%20from%20chunk%20text%20before%20embedding.%0A%0AYou%20can%20track%20related%20changes%20and%20fixes%20in%20PRs%20like%20%5B%2329115%5D%28https%3A//github.com/langgenius/dify/pull/29115%29%20and%20%5B%2330210%5D%28https%3A//github.com/langgenius/dify/pull/30210%29%2C%20and%20monitor%20for%20future%20updates%20addressing%20this%20ranking%20inconsistency.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/31420)
yindo commented 2026-02-21 20:14:35 -05:00
Author
Owner
Copy Link
Copy Source

@unmo commented on GitHub (Jan 22, 2026):

@dosu
Thanks for the explanation.

In our tests, the similarity score for an identical image query is consistently very low (~0.03–0.05), which seems too low even for non-exact matching.

This happens not only with Vertex AI multimodalembedding@001, but also with other embedding models we tested. The same behavior is reproducible: uploading an image that exists in the KB does not produce a high similarity score for the corresponding chunk.

So the issue is not limited to URL/text queries. The main concern is that image file queries do not produce high similarity scores even for identical images stored in the KB.

@unmo commented on GitHub (Jan 22, 2026): @dosu Thanks for the explanation. In our tests, the similarity score for an identical image query is consistently very low (~0.03–0.05), which seems too low even for non-exact matching. This happens not only with Vertex AI multimodalembedding@001, but also with other embedding models we tested. The same behavior is reproducible: uploading an image that exists in the KB does not produce a high similarity score for the corresponding chunk. So the issue is not limited to URL/text queries. The main concern is that image file queries do not produce high similarity scores even for identical images stored in the KB.
yindo commented 2026-02-21 20:14:35 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 22, 2026):

This is a systemic issue in Dify's multimodal pipeline: the same image bytes are used for both indexing and query, but the only difference is the EmbeddingInputType flag (DOCUMENT for indexing, QUERY for search) sent to the embedding provider/model. There’s no resizing, format conversion, or other preprocessing in Dify itself that would explain the low similarity scores you’re seeing for identical images—this applies across embedding models, not just Vertex AI. The most likely cause is that the embedding provider/model generates different embeddings for DOCUMENT and QUERY input types, resulting in low similarity even for identical images reference.

To confirm, you can enable debug logging in Dify and compare the raw embeddings for the same image at both indexing and query time, or test the provider/model directly outside Dify with both input types. If the embeddings differ, this is a provider/model behavior, not a Dify bug. If high image-to-image similarity is critical, you may want to avoid relying on this feature until the underlying issue is resolved, or check with your embedding provider for clarification on DOCUMENT vs. QUERY handling.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Jan 22, 2026): <!-- Answer --> This is a systemic issue in Dify's multimodal pipeline: the same image bytes are used for both indexing and query, but the only difference is the EmbeddingInputType flag (DOCUMENT for indexing, QUERY for search) sent to the embedding provider/model. There’s no resizing, format conversion, or other preprocessing in Dify itself that would explain the low similarity scores you’re seeing for identical images—this applies across embedding models, not just Vertex AI. The most likely cause is that the embedding provider/model generates different embeddings for DOCUMENT and QUERY input types, resulting in low similarity even for identical images [reference](https://github.com/langgenius/dify/blob/61f8647f374ed38003e5b34d213487c34b113d61/api/core/rag/embedding/cached_embedding.py#L23-L274). To confirm, you can enable debug logging in Dify and compare the raw embeddings for the same image at both indexing and query time, or test the provider/model directly outside Dify with both input types. If the embeddings differ, this is a provider/model behavior, not a Dify bug. If high image-to-image similarity is critical, you may want to avoid relying on this feature until the underlying issue is resolved, or check with your embedding provider for clarification on DOCUMENT vs. QUERY handling. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/0d62ec1f-0355-44e2-a126-81a3b4ae7aa4?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/0d62ec1f-0355-44e2-a126-81a3b4ae7aa4?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/0d62ec1f-0355-44e2-a126-81a3b4ae7aa4?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/0d62ec1f-0355-44e2-a126-81a3b4ae7aa4?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/0d62ec1f-0355-44e2-a126-81a3b4ae7aa4?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/0d62ec1f-0355-44e2-a126-81a3b4ae7aa4?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/0d62ec1f-0355-44e2-a126-81a3b4ae7aa4?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=This%20is%20a%20systemic%20issue%20in%20Dify%27s%20multimodal%20pipeline%3A%20the%20same%20image%20bytes%20are%20used%20for%20both%20indexing%20and%20query%2C%20but%20the%20only%20difference%20is%20the%20EmbeddingInputType%20flag%20%28DOCUMENT%20for%20indexing%2C%20QUERY%20for%20search%29%20sent%20to%20the%20embedding%20provider/model.%20There%E2%80%99s%20no%20resizing%2C%20format%20conversion%2C%20or%20other%20preprocessing%20in%20Dify%20itself%20that%20would%20explain%20the%20low%20similarity%20scores%20you%E2%80%99re%20seeing%20for%20identical%20images%E2%80%94this%20applies%20across%20embedding%20models%2C%20not%20just%20Vertex%20AI.%20The%20most%20likely%20cause%20is%20that%20the%20embedding%20provider/model%20generates%20different%20embeddings%20for%20DOCUMENT%20and%20QUERY%20input%20types%2C%20resulting%20in%20low%20similarity%20even%20for%20identical%20images%C2%A0%5Breference%5D%28https%3A//github.com/langgenius/dify/blob/61f8647f374ed38003e5b34d213487c34b113d61/api/core/rag/embedding/cached_embedding.py%23L23-L274%29.%0A%0ATo%20confirm%2C%20you%20can%20enable%20debug%20logging%20in%20Dify%20and%20compare%20the%20raw%20embeddings%20for%20the%20same%20image%20at%20both%20indexing%20and%20query%20time%2C%20or%20test%20the%20provider/model%20directly%20outside%20Dify%20with%20both%20input%20types.%20If%20the%20embeddings%20differ%2C%20this%20is%20a%20provider/model%20behavior%2C%20not%20a%20Dify%20bug.%20If%20high%20image-to-image%20similarity%20is%20critical%2C%20you%20may%20want%20to%20avoid%20relying%20on%20this%20feature%20until%20the%20underlying%20issue%20is%20resolved%2C%20or%20check%20with%20your%20embedding%20provider%20for%20clarification%20on%20DOCUMENT%20vs.%20QUERY%20handling.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/31420)
yindo commented 2026-02-21 20:14:35 -05:00
Author
Owner
Copy Link
Copy Source

@unmo commented on GitHub (Jan 22, 2026):

@dosu
Thanks. I checked the current Vertex provider implementation in Dify.

In the multimodal image embedding path (vertexai.vision_models.MultiModalEmbeddingModel.get_embeddings(image=...)), EmbeddingInputType (DOCUMENT/QUERY) is not mapped to any provider parameter (no task_type or input_type is passed for image embeddings). The image embedding call looks identical for indexing and query.

Also, the official Vertex multimodal embeddings API docs for multimodalembedding@001 do not expose a DOCUMENT/QUERY option in the request examples.

https://github.com/langgenius/dify-official-plugins/blob/main/models/vertex_ai/models/text_embedding/text_embedding.py#L274

So the low similarity scores (~0.03–0.05) for identical image queries may not be explained by DOCUMENT vs QUERY handling on the provider side (at least for the image embedding call).

Could someone confirm if this behavior is expected in v1.11.4?

@unmo commented on GitHub (Jan 22, 2026): @dosu Thanks. I checked the current Vertex provider implementation in Dify. In the multimodal image embedding path (`vertexai.vision_models.MultiModalEmbeddingModel.get_embeddings(image=...)`), `EmbeddingInputType` (DOCUMENT/QUERY) is not mapped to any provider parameter (no task_type or input_type is passed for image embeddings). The image embedding call looks identical for indexing and query. Also, the official Vertex multimodal embeddings API docs for `multimodalembedding@001` do not expose a DOCUMENT/QUERY option in the request examples. https://github.com/langgenius/dify-official-plugins/blob/main/models/vertex_ai/models/text_embedding/text_embedding.py#L274 So the low similarity scores (~0.03–0.05) for identical image queries may not be explained by DOCUMENT vs QUERY handling on the provider side (at least for the image embedding call). Could someone confirm if this behavior is expected in v1.11.4?
yindo commented 2026-02-21 20:14:35 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 22, 2026):

Based on the code and plugin implementation, the low similarity scores for identical image queries are not caused by any Dify-side transformation, configuration, or DOCUMENT/QUERY handling—Dify sends the same image bytes and uses the same embedding call for both indexing and query, and the Vertex AI API does not distinguish input types for images. This behavior is observed across multiple embedding models and is a known issue in Dify's multimodal retrieval pipeline, not a configuration error or expected/intended outcome. There is no explicit confirmation in the docs or release notes that this is "expected," but it is a recognized limitation/bug in v1.11.4 and related versions, and not something you can resolve through settings or environment changes. Monitoring future releases for a fix is recommended.
Reference: plugin code
Reference: core pipeline
Related issue
Recent PRs

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Jan 22, 2026): <!-- Answer --> Based on the code and plugin implementation, the low similarity scores for identical image queries are not caused by any Dify-side transformation, configuration, or DOCUMENT/QUERY handling—Dify sends the same image bytes and uses the same embedding call for both indexing and query, and the Vertex AI API does not distinguish input types for images. This behavior is observed across multiple embedding models and is a known issue in Dify's multimodal retrieval pipeline, not a configuration error or expected/intended outcome. There is no explicit confirmation in the docs or release notes that this is "expected," but it is a recognized limitation/bug in v1.11.4 and related versions, and not something you can resolve through settings or environment changes. Monitoring future releases for a fix is recommended. [Reference: plugin code](https://github.com/langgenius/dify-official-plugins/blob/main/models/vertex_ai/models/text_embedding/text_embedding.py#L274) [Reference: core pipeline](https://github.com/langgenius/dify/blob/c8ae6e39d2eb3000afbb83b96463615d7adb4a3f/api/core/rag/embedding/cached_embedding.py#L107-L274) [Related issue](https://github.com/langgenius/dify/issues/31420) [Recent PRs](https://github.com/langgenius/dify/pull/30210) <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/39474b82-4f80-4cb7-bb64-9031de3ea6e4?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/39474b82-4f80-4cb7-bb64-9031de3ea6e4?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/39474b82-4f80-4cb7-bb64-9031de3ea6e4?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/39474b82-4f80-4cb7-bb64-9031de3ea6e4?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/39474b82-4f80-4cb7-bb64-9031de3ea6e4?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/39474b82-4f80-4cb7-bb64-9031de3ea6e4?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/39474b82-4f80-4cb7-bb64-9031de3ea6e4?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=Based%20on%20the%20code%20and%20plugin%20implementation%2C%20the%20low%20similarity%20scores%20for%20identical%20image%20queries%20are%20not%20caused%20by%20any%20Dify-side%20transformation%2C%20configuration%2C%20or%20DOCUMENT/QUERY%20handling%E2%80%94Dify%20sends%20the%20same%20image%20bytes%20and%20uses%20the%20same%20embedding%20call%20for%20both%20indexing%20and%20query%2C%20and%20the%20Vertex%20AI%20API%20does%20not%20distinguish%20input%20types%20for%20images.%20This%20behavior%20is%20observed%20across%20multiple%20embedding%20models%20and%20is%20a%20known%20issue%20in%20Dify%27s%20multimodal%20retrieval%20pipeline%2C%20not%20a%20configuration%20error%20or%20expected/intended%20outcome.%20There%20is%20no%20explicit%20confirmation%20in%20the%20docs%20or%20release%20notes%20that%20this%20is%20%22expected%2C%22%20but%20it%20is%20a%20recognized%20limitation/bug%20in%20v1.11.4%20and%20related%20versions%2C%20and%20not%20something%20you%20can%20resolve%20through%20settings%20or%20environment%20changes.%20Monitoring%20future%20releases%20for%20a%20fix%20is%20recommended.%20%20%0A%5BReference%3A%20plugin%20code%5D%28https%3A//github.com/langgenius/dify-official-plugins/blob/main/models/vertex_ai/models/text_embedding/text_embedding.py%23L274%29%20%20%0A%5BReference%3A%20core%20pipeline%5D%28https%3A//github.com/langgenius/dify/blob/c8ae6e39d2eb3000afbb83b96463615d7adb4a3f/api/core/rag/embedding/cached_embedding.py%23L107-L274%29%20%20%0A%5BRelated%20issue%5D%28https%3A//github.com/langgenius/dify/issues/31420%29%20%20%0A%5BRecent%20PRs%5D%28https%3A//github.com/langgenius/dify/pull/30210%29)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/31420)
yindo commented 2026-02-21 20:14:35 -05:00
Author
Owner
Copy Link
Copy Source

@unmo commented on GitHub (Jan 22, 2026):

Could someone (a maintainer or anyone familiar with Dify multimodal retrieval) please help explain this behavior?

When we query the KB using an image that should be identical to an image already indexed in the multimodal knowledge base, the similarity score stays extremely low (~0.03–0.05).
This is far lower than expected for an identical image query.

We reproduced the same behavior across multiple embedding models.

If anyone knows the root cause (which stage/component makes the score drop, or if this is a known limitation/bug), please share details.
If there are related issues or PRs, links would be greatly appreciated.

@unmo commented on GitHub (Jan 22, 2026): Could someone (a maintainer or anyone familiar with Dify multimodal retrieval) please help explain this behavior? When we query the KB using an image that should be identical to an image already indexed in the multimodal knowledge base, the similarity score stays extremely low (**~0.03–0.05**). This is far lower than expected for an identical image query. We reproduced the same behavior across multiple embedding models. If anyone knows the root cause (which stage/component makes the score drop, or if this is a known limitation/bug), please share details. If there are related issues or PRs, links would be greatly appreciated.
yindo commented 2026-02-21 20:14:35 -05:00
Author
Owner
Copy Link
Copy Source

@unmo commented on GitHub (Jan 23, 2026):

@dosu

I compared Dify hit-testing scores with cosine similarity computed locally using Vertex AI embeddings.

Image query case:
Dify score = 0.0461731553
Local cosine(query image embedding, chunk image-URL text embedding) = 0.0461730764

Text query case:
Dify score = 0.2682865262
Local cosine(query text embedding, chunk image-URL text embedding) = 0.2682864400

Since both values match almost exactly, it suggests that the Dify score may be computed against the embedding of the image URL/markdown string stored in the chunk, even when the query is an image (i.e., image/text ↔ URL-text), rather than image ↔ image embedding similarity.

Does anyone know how to verify whether image chunks are indexed and searched using image embeddings, not URL/text embeddings?

@unmo commented on GitHub (Jan 23, 2026): @dosu I compared Dify hit-testing scores with cosine similarity computed locally using Vertex AI embeddings. Image query case: Dify score = 0.0461731553 Local cosine(query image embedding, chunk image-URL text embedding) = 0.0461730764 Text query case: Dify score = 0.2682865262 Local cosine(query text embedding, chunk image-URL text embedding) = 0.2682864400 Since both values match almost exactly, it suggests that the Dify score may be computed against the embedding of the image URL/markdown string stored in the chunk, even when the query is an image (i.e., image/text ↔ URL-text), rather than image ↔ image embedding similarity. Does anyone know how to verify whether image chunks are indexed and searched using image embeddings, not URL/text embeddings?
yindo commented 2026-02-21 20:14:35 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 23, 2026):

You can verify how Dify indexes and searches image chunks by checking the chunk metadata and the vector database. In Dify's high-quality multimodal pipeline, images from DOCX files are saved as files and referenced in the chunk text as markdown image links (e.g., ![image](...)). However, for actual image retrieval, Dify creates separate "attachment" chunks (with doc_type=IMAGE) and embeds the raw image bytes—not the markdown or URL text—using the embedding model. These image embeddings are stored in the vector DB and are used for image queries, while text chunks (including those with image markdown) are embedded as text and used for text queries. The two paths are distinct and kept in sync via metadata like doc_id and doc_type source.

If your Dify search results and scores match the cosine similarity between your query image embedding and the chunk's text (URL/markdown) embedding, it means your search is hitting text chunks, not image chunks. To confirm image chunks are indexed and used, inspect your vector DB (e.g., Qdrant) for entries with doc_type=IMAGE and check that their embeddings were created from image bytes. You can also enable debug logs to trace embedding calls and see which chunks are being compared during retrieval source.

If your retrieval UI or API is only showing text chunks (not image attachments), or if your search is performed as a text query (even with an image URL as text), you’ll only see text-to-text similarity, not image-to-image. For true image-to-image retrieval, ensure your queries and results are using the image attachment chunks and their embeddings.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Jan 23, 2026): <!-- Answer --> You can verify how Dify indexes and searches image chunks by checking the chunk metadata and the vector database. In Dify's high-quality multimodal pipeline, images from DOCX files are saved as files and referenced in the chunk text as markdown image links (e.g., `![image](...)`). However, for actual image retrieval, Dify creates separate "attachment" chunks (with `doc_type=IMAGE`) and embeds the raw image bytes—not the markdown or URL text—using the embedding model. These image embeddings are stored in the vector DB and are used for image queries, while text chunks (including those with image markdown) are embedded as text and used for text queries. The two paths are distinct and kept in sync via metadata like `doc_id` and `doc_type` [source](https://github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/core/rag/datasource/vdb/vector_factory.py#L197-L289). If your Dify search results and scores match the cosine similarity between your query image embedding and the chunk's text (URL/markdown) embedding, it means your search is hitting text chunks, not image chunks. To confirm image chunks are indexed and used, inspect your vector DB (e.g., Qdrant) for entries with `doc_type=IMAGE` and check that their embeddings were created from image bytes. You can also enable debug logs to trace embedding calls and see which chunks are being compared during retrieval [source](https://github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/core/rag/embedding/cached_embedding.py#L28-L274). If your retrieval UI or API is only showing text chunks (not image attachments), or if your search is performed as a text query (even with an image URL as text), you’ll only see text-to-text similarity, not image-to-image. For true image-to-image retrieval, ensure your queries and results are using the image attachment chunks and their embeddings. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/e7c9cf0b-f965-47f9-953b-6b380201afbf?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/e7c9cf0b-f965-47f9-953b-6b380201afbf?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/e7c9cf0b-f965-47f9-953b-6b380201afbf?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/e7c9cf0b-f965-47f9-953b-6b380201afbf?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/e7c9cf0b-f965-47f9-953b-6b380201afbf?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/e7c9cf0b-f965-47f9-953b-6b380201afbf?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/e7c9cf0b-f965-47f9-953b-6b380201afbf?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=You%20can%20verify%20how%20Dify%20indexes%20and%20searches%20image%20chunks%20by%20checking%20the%20chunk%20metadata%20and%20the%20vector%20database.%20In%20Dify%27s%20high-quality%20multimodal%20pipeline%2C%20images%20from%20DOCX%20files%20are%20saved%20as%20files%20and%20referenced%20in%20the%20chunk%20text%20as%20markdown%20image%20links%20%28e.g.%2C%20%60%21%5Bimage%5D%28...%29%60%29.%20However%2C%20for%20actual%20image%20retrieval%2C%20Dify%20creates%20separate%20%22attachment%22%20chunks%20%28with%20%60doc_type%3DIMAGE%60%29%20and%20embeds%20the%20raw%20image%20bytes%E2%80%94not%20the%20markdown%20or%20URL%20text%E2%80%94using%20the%20embedding%20model.%20These%20image%20embeddings%20are%20stored%20in%20the%20vector%20DB%20and%20are%20used%20for%20image%20queries%2C%20while%20text%20chunks%20%28including%20those%20with%20image%20markdown%29%20are%20embedded%20as%20text%20and%20used%20for%20text%20queries.%20The%20two%20paths%20are%20distinct%20and%20kept%20in%20sync%20via%20metadata%20like%20%60doc_id%60%20and%20%60doc_type%60%20%5Bsource%5D%28https%3A//github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/core/rag/datasource/vdb/vector_factory.py%23L197-L289%29.%0A%0AIf%20your%20Dify%20search%20results%20and%20scores%20match%20the%20cosine%20similarity%20between%20your%20query%20image%20embedding%20and%20the%20chunk%27s%20text%20%28URL/markdown%29%20embedding%2C%20it%20means%20your%20search%20is%20hitting%20text%20chunks%2C%20not%20image%20chunks.%20To%20confirm%20image%20chunks%20are%20indexed%20and%20used%2C%20inspect%20your%20vector%20DB%20%28e.g.%2C%20Qdrant%29%20for%20entries%20with%20%60doc_type%3DIMAGE%60%20and%20check%20that%20their%20embeddings%20were%20created%20from%20image%20bytes.%20You%20can%20also%20enable%20debug%20logs%20to%20trace%20embedding%20calls%20and%20see%20which%20chunks%20are%20being%20compared%20during%20retrieval%20%5Bsource%5D%28https%3A//github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/core/rag/embedding/cached_embedding.py%23L28-L274%29.%0A%0AIf%20your%20retrieval%20UI%20or%20API%20is%20only%20showing%20text%20chunks%20%28not%20image%20attachments%29%2C%20or%20if%20your%20search%20is%20performed%20as%20a%20text%20query%20%28even%20with%20an%20image%20URL%20as%20text%29%2C%20you%E2%80%99ll%20only%20see%20text-to-text%20similarity%2C%20not%20image-to-image.%20For%20true%20image-to-image%20retrieval%2C%20ensure%20your%20queries%20and%20results%20are%20using%20the%20image%20attachment%20chunks%20and%20their%20embeddings.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/31420)
yindo commented 2026-02-21 20:14:35 -05:00
Author
Owner
Copy Link
Copy Source

@unmo commented on GitHub (Jan 23, 2026):

@Dosu

When searching the KB using an uploaded image (that should exist in the indexed DOCX), the response items returned by Search Test look like “text segments”, and the document.doc_type / document.doc_metadata fields are always null.

For example, the image query returns records like:

{
    "query": {
        "content": ""
    },
    "records": [
        {
            "segment": {
                "id": "1c2fd00e-4ee5-4e99-ada3-7483dd46b419",
                "position": 4,
                "document_id": "91aba59a-f5fe-4136-bd0a-d7c49741bd51",
                "content": "\u3053\u306e\u753b\u50cf\u306f\u864e\u306e\u753b\u50cf\u3067\u3059\u3002\u864e\u306f\u5168\u9762\u306b\u5927\u304d\u304f\u6620\u3063\u3066\u304a\u308a\u3001\u80cc\u666f\u306f\u771f\u3063\u6697\u3067\u3059\u3002\u864e\u306e\u8868\u60c5\u306f\u7121\u8868\u60c5\u3067\u3042\u308a\u3001\u3068\u3066\u3082\u5a01\u5727\u611f\u304c\u3042\u308b\u753b\u50cf\u3068\u306a\u3063\u3066\u3044\u307e\u3059\u3002",
                "sign_content": <same content>,
                "answer": null,
                "word_count": 66,
                "tokens": 67,
                "keywords": null,
                "index_node_id": "32ff77dd-ff45-41fe-bc0c-9d1700077f85",
                "index_node_hash": "2878ba5f1f894e0cf1edf233d88f27987761499257190db6c612a4c0a4c1ad72",
                "hit_count": 0,
                "enabled": true,
                "disabled_at": null,
                "disabled_by": null,
                "status": "completed",
                "created_by": "ee77b759-da1a-4063-8127-7c2e61b9e57b",
                "created_at": 1769160997,
                "indexing_at": 1769160996,
                "completed_at": 1769161000,
                "error": null,
                "stopped_at": null,
                "document": {
                    "id": "91aba59a-f5fe-4136-bd0a-d7c49741bd51",
                    "data_source_type": "upload_file",
                    "name": "\u30de\u30eb\u30c1\u30e2\u30fc\u30c0\u30ebRAG\u30c6\u30b9\u30c8.docx",
                    "doc_type": null,
                    "doc_metadata": null
                }
            },
            "child_chunks": null,
            "score": 0.07502943277359009,
            "tsne_position": null,
            "files": null
        },
        {
            "segment": {
                "id": "befbbc59-747e-4030-b1cf-2161a1df5460",
                "position": 7,
                "document_id": "91aba59a-f5fe-4136-bd0a-d7c49741bd51",
                "content": "![image](\/files\/13ce443d-1161-4d23-88df-c217939671bc\/file-preview)",
                "sign_content": "<content>",
                "answer": null,
                "word_count": 66,
                "tokens": 28,
                "keywords": null,
                "index_node_id": "728ae045-73ce-4e78-bc6a-bfe82735c45f",
                "index_node_hash": "2bdcb6050bcf9aa7b274a099988947ef6a1aec414d263fab90e6c2fc5ce67434",
                "hit_count": 0,
                "enabled": true,
                "disabled_at": null,
                "disabled_by": null,
                "status": "completed",
                "created_by": "ee77b759-da1a-4063-8127-7c2e61b9e57b",
                "created_at": 1769160997,
                "indexing_at": 1769160996,
                "completed_at": 1769161015,
                "error": null,
                "stopped_at": null,
                "document": {
                    "id": "91aba59a-f5fe-4136-bd0a-d7c49741bd51",
                    "data_source_type": "upload_file",
                    "name": "\u30de\u30eb\u30c1\u30e2\u30fc\u30c0\u30ebRAG\u30c6\u30b9\u30c8.docx",
                    "doc_type": null,
                    "doc_metadata": null
                }
            },
            "child_chunks": null,
            "score": 0.053455650806427,
            "tsne_position": null,
            "files": null
        },...

Questions

  1. Is it expected that segment.document.doc_type and segment.document.doc_metadata are always null in Search Test results, even for multimodal KBs?
  2. When a user queries with an image, how can we verify that the retrieval is actually using image embeddings (image attachment chunks) rather than text segments that contain image markdown/URLs?
  3. Are image chunks extracted from DOCX/PDF indexed as a separate “image chunk type” (e.g., doc_type=IMAGE or similar) in High Quality multimodal KB?
  • If yes, does the Search Test API return those image chunks, or does it only return text segments?

Any clarification or pointers to the relevant code path would be greatly appreciated.

@unmo commented on GitHub (Jan 23, 2026): @Dosu When searching the KB using an uploaded image (that should exist in the indexed DOCX), the response items returned by Search Test look like “text segments”, and the document.doc_type / document.doc_metadata fields are always null. For example, the image query returns records like: ```json { "query": { "content": "" }, "records": [ { "segment": { "id": "1c2fd00e-4ee5-4e99-ada3-7483dd46b419", "position": 4, "document_id": "91aba59a-f5fe-4136-bd0a-d7c49741bd51", "content": "\u3053\u306e\u753b\u50cf\u306f\u864e\u306e\u753b\u50cf\u3067\u3059\u3002\u864e\u306f\u5168\u9762\u306b\u5927\u304d\u304f\u6620\u3063\u3066\u304a\u308a\u3001\u80cc\u666f\u306f\u771f\u3063\u6697\u3067\u3059\u3002\u864e\u306e\u8868\u60c5\u306f\u7121\u8868\u60c5\u3067\u3042\u308a\u3001\u3068\u3066\u3082\u5a01\u5727\u611f\u304c\u3042\u308b\u753b\u50cf\u3068\u306a\u3063\u3066\u3044\u307e\u3059\u3002", "sign_content": <same content>, "answer": null, "word_count": 66, "tokens": 67, "keywords": null, "index_node_id": "32ff77dd-ff45-41fe-bc0c-9d1700077f85", "index_node_hash": "2878ba5f1f894e0cf1edf233d88f27987761499257190db6c612a4c0a4c1ad72", "hit_count": 0, "enabled": true, "disabled_at": null, "disabled_by": null, "status": "completed", "created_by": "ee77b759-da1a-4063-8127-7c2e61b9e57b", "created_at": 1769160997, "indexing_at": 1769160996, "completed_at": 1769161000, "error": null, "stopped_at": null, "document": { "id": "91aba59a-f5fe-4136-bd0a-d7c49741bd51", "data_source_type": "upload_file", "name": "\u30de\u30eb\u30c1\u30e2\u30fc\u30c0\u30ebRAG\u30c6\u30b9\u30c8.docx", "doc_type": null, "doc_metadata": null } }, "child_chunks": null, "score": 0.07502943277359009, "tsne_position": null, "files": null }, { "segment": { "id": "befbbc59-747e-4030-b1cf-2161a1df5460", "position": 7, "document_id": "91aba59a-f5fe-4136-bd0a-d7c49741bd51", "content": "![image](\/files\/13ce443d-1161-4d23-88df-c217939671bc\/file-preview)", "sign_content": "<content>", "answer": null, "word_count": 66, "tokens": 28, "keywords": null, "index_node_id": "728ae045-73ce-4e78-bc6a-bfe82735c45f", "index_node_hash": "2bdcb6050bcf9aa7b274a099988947ef6a1aec414d263fab90e6c2fc5ce67434", "hit_count": 0, "enabled": true, "disabled_at": null, "disabled_by": null, "status": "completed", "created_by": "ee77b759-da1a-4063-8127-7c2e61b9e57b", "created_at": 1769160997, "indexing_at": 1769160996, "completed_at": 1769161015, "error": null, "stopped_at": null, "document": { "id": "91aba59a-f5fe-4136-bd0a-d7c49741bd51", "data_source_type": "upload_file", "name": "\u30de\u30eb\u30c1\u30e2\u30fc\u30c0\u30ebRAG\u30c6\u30b9\u30c8.docx", "doc_type": null, "doc_metadata": null } }, "child_chunks": null, "score": 0.053455650806427, "tsne_position": null, "files": null },... ``` Questions 1. Is it expected that segment.document.doc_type and segment.document.doc_metadata are always null in Search Test results, even for multimodal KBs? 2. When a user queries with an image, how can we verify that the retrieval is actually using image embeddings (image attachment chunks) rather than text segments that contain image markdown/URLs? 3. Are image chunks extracted from DOCX/PDF indexed as a separate “image chunk type” (e.g., doc_type=IMAGE or similar) in High Quality multimodal KB? * If yes, does the Search Test API return those image chunks, or does it only return text segments? Any clarification or pointers to the relevant code path would be greatly appreciated.
yindo commented 2026-02-21 20:14:35 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 23, 2026):

  1. Yes, it's expected in Dify v1.11.4 for segment.document.doc_type and segment.document.doc_metadata to often be null in Search Test results, even for multimodal KBs. These fields are nullable at the database/model level and may not be set or populated during indexing or retrieval, especially for segments representing text or markdown image links rather than direct image attachments. This is a known limitation in current releases [source].

  2. To verify if retrieval on an image query is actually using image embeddings (image attachment chunks), check if your query is passed as an attachment ID (not as text or URL) and inspect the vector database for entries with doc_type=IMAGE. You can also enable debug logs to trace embedding calls and see if the retrieval pipeline is using embed_multimodal_query with image bytes, not text. If your search results and scores match the cosine similarity between your query image embedding and a chunk's text (URL/markdown) embedding, you're likely hitting text chunks, not image chunks [source].

  3. Yes, image chunks extracted from DOCX/PDF are indexed as a separate type (doc_type=IMAGE) in high-quality multimodal KBs. However, the Search Test API typically returns text segments (including those with markdown image links) by default, not the raw image chunks. Image chunks are stored and linked via SegmentAttachmentBinding, but unless the retrieval UI/API is configured to show image attachments, you'll only see text segments in the response. Full support for reliable retrieval and rendering of image-containing segments is only available in versions after PR #27793 (December 2025); in v1.11.4, this behavior is limited [source].

Relevant code paths:

Let me know if you want practical steps for inspecting your vector DB or enabling debug logs to trace image chunk retrieval.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Jan 23, 2026): <!-- Answer --> 1. Yes, it's expected in Dify v1.11.4 for `segment.document.doc_type` and `segment.document.doc_metadata` to often be null in Search Test results, even for multimodal KBs. These fields are nullable at the database/model level and may not be set or populated during indexing or retrieval, especially for segments representing text or markdown image links rather than direct image attachments. This is a known limitation in current releases [[source](https://github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/models/dataset.py#L418-L909)]. 2. To verify if retrieval on an image query is actually using image embeddings (image attachment chunks), check if your query is passed as an attachment ID (not as text or URL) and inspect the vector database for entries with `doc_type=IMAGE`. You can also enable debug logs to trace embedding calls and see if the retrieval pipeline is using `embed_multimodal_query` with image bytes, not text. If your search results and scores match the cosine similarity between your query image embedding and a chunk's text (URL/markdown) embedding, you're likely hitting text chunks, not image chunks [[source](https://github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/core/rag/datasource/vdb/vector_factory.py#L197-L290)]. 3. Yes, image chunks extracted from DOCX/PDF are indexed as a separate type (`doc_type=IMAGE`) in high-quality multimodal KBs. However, the Search Test API typically returns text segments (including those with markdown image links) by default, not the raw image chunks. Image chunks are stored and linked via `SegmentAttachmentBinding`, but unless the retrieval UI/API is configured to show image attachments, you'll only see text segments in the response. Full support for reliable retrieval and rendering of image-containing segments is only available in versions after PR #27793 (December 2025); in v1.11.4, this behavior is limited [[source](https://github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/core/rag/retrieval/dataset_retrieval.py#L234-L706)]. Relevant code paths: - Document and segment metadata: [`models/dataset.py`](https://github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/models/dataset.py#L418-L909) - Vector indexing and multimodal embedding: [`core/rag/datasource/vdb/vector_factory.py`](https://github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/core/rag/datasource/vdb/vector_factory.py#L197-L290) - Retrieval and Search Test API: [`core/rag/retrieval/dataset_retrieval.py`](https://github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/core/rag/retrieval/dataset_retrieval.py#L234-L706) Let me know if you want practical steps for inspecting your vector DB or enabling debug logs to trace image chunk retrieval. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/f1e0cff7-a9a6-498c-82b1-364bb7de416a?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/f1e0cff7-a9a6-498c-82b1-364bb7de416a?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/f1e0cff7-a9a6-498c-82b1-364bb7de416a?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/f1e0cff7-a9a6-498c-82b1-364bb7de416a?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/f1e0cff7-a9a6-498c-82b1-364bb7de416a?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/f1e0cff7-a9a6-498c-82b1-364bb7de416a?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/f1e0cff7-a9a6-498c-82b1-364bb7de416a?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=1.%20Yes%2C%20it%27s%20expected%20in%20Dify%20v1.11.4%20for%20%60segment.document.doc_type%60%20and%20%60segment.document.doc_metadata%60%20to%20often%20be%20null%20in%20Search%20Test%20results%2C%20even%20for%20multimodal%20KBs.%20These%20fields%20are%20nullable%20at%20the%20database/model%20level%20and%20may%20not%20be%20set%20or%20populated%20during%20indexing%20or%20retrieval%2C%20especially%20for%20segments%20representing%20text%20or%20markdown%20image%20links%20rather%20than%20direct%20image%20attachments.%20This%20is%20a%20known%20limitation%20in%20current%20releases%20%5B%5Bsource%5D%28https%3A//github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/models/dataset.py%23L418-L909%29%5D.%0A%0A2.%20To%20verify%20if%20retrieval%20on%20an%20image%20query%20is%20actually%20using%20image%20embeddings%20%28image%20attachment%20chunks%29%2C%20check%20if%20your%20query%20is%20passed%20as%20an%20attachment%20ID%20%28not%20as%20text%20or%20URL%29%20and%20inspect%20the%20vector%20database%20for%20entries%20with%20%60doc_type%3DIMAGE%60.%20You%20can%20also%20enable%20debug%20logs%20to%20trace%20embedding%20calls%20and%20see%20if%20the%20retrieval%20pipeline%20is%20using%20%60embed_multimodal_query%60%20with%20image%20bytes%2C%20not%20text.%20If%20your%20search%20results%20and%20scores%20match%20the%20cosine%20similarity%20between%20your%20query%20image%20embedding%20and%20a%20chunk%27s%20text%20%28URL/markdown%29%20embedding%2C%20you%27re%20likely%20hitting%20text%20chunks%2C%20not%20image%20chunks%20%5B%5Bsource%5D%28https%3A//github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/core/rag/datasource/vdb/vector_factory.py%23L197-L290%29%5D.%0A%0A3.%20Yes%2C%20image%20chunks%20extracted%20from%20DOCX/PDF%20are%20indexed%20as%20a%20separate%20type%20%28%60doc_type%3DIMAGE%60%29%20in%20high-quality%20multimodal%20KBs.%20However%2C%20the%20Search%20Test%20API%20typically%20returns%20text%20segments%20%28including%20those%20with%20markdown%20image%20links%29%20by%20default%2C%20not%20the%20raw%20image%20chunks.%20Image%20chunks%20are%20stored%20and%20linked%20via%20%60SegmentAttachmentBinding%60%2C%20but%20unless%20the%20retrieval%20UI/API%20is%20configured%20to%20show%20image%20attachments%2C%20you%27ll%20only%20see%20text%20segments%20in%20the%20response.%20Full%20support%20for%20reliable%20retrieval%20and%20rendering%20of%20image-containing%20segments%20is%20only%20available%20in%20versions%20after%20PR%20%2327793%20%28December%202025%29%3B%20in%20v1.11.4%2C%20this%20behavior%20is%20limited%20%5B%5Bsource%5D%28https%3A//github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/core/rag/retrieval/dataset_retrieval.py%23L234-L706%29%5D.%0A%0ARelevant%20code%20paths%3A%0A-%20Document%20and%20segment%20metadata%3A%20%5B%60models/dataset.py%60%5D%28https%3A//github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/models/dataset.py%23L418-L909%29%0A-%20Vector%20indexing%20and%20multimodal%20embedding%3A%20%5B%60core/rag/datasource/vdb/vector_factory.py%60%5D%28https%3A//github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/core/rag/datasource/vdb/vector_factory.py%23L197-L290%29%0A-%20Retrieval%20and%20Search%20Test%20API%3A%20%5B%60core/rag/retrieval/dataset_retrieval.py%60%5D%28https%3A//github.com/langgenius/dify/blob/6342d196e874d7a4485fb49bf41141d4d44af0b9/api/core/rag/retrieval/dataset_retrieval.py%23L234-L706%29%0A%0ALet%20me%20know%20if%20you%20want%20practical%20steps%20for%20inspecting%20your%20vector%20DB%20or%20enabling%20debug%20logs%20to%20trace%20image%20chunk%20retrieval.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/31420)
yindo commented 2026-02-21 20:14:35 -05:00
Author
Owner
Copy Link
Copy Source

@unmo commented on GitHub (Jan 23, 2026):

@crazywoola
Hi, could you please advise on this behavior?

We are testing Dify v1.11.4 (self-hosted) with a Multimodal Knowledge Base
(High Quality / Vector search / Rerank OFF).
When we query by uploading an image that should be identical to an image indexed from a DOCX,
the expected image chunk does not rank at the top, and the similarity score stays very low
(~0.03–0.08).

Most importantly, we found a strong clue:

  • The Dify score for an image query matches almost exactly the cosine similarity computed locally as:
    cosine(query_image_embedding, chunk_image_url_markdown_text_embedding)

This suggests the retrieval might be comparing image embedding ↔ image-URL/markdown text embedding,
instead of image ↔ image embedding similarity.
(I checked the code but couldn’t clearly identify where/why this path is taken.)

Questions:

  1. In v1.11.4, is it expected that Search Test returns only text segments even for image queries
    (i.e., not returning IMAGE attachment chunks / image embeddings)?
  2. For DOCX extracted images in High Quality multimodal KB, are IMAGE chunks stored separately
    (e.g., doc_type=IMAGE or similar) in the vector DB?
    If yes, what is the recommended way to verify they exist and are actually used during retrieval?

Thanks in advance.

@unmo commented on GitHub (Jan 23, 2026): @crazywoola Hi, could you please advise on this behavior? We are testing Dify v1.11.4 (self-hosted) with a Multimodal Knowledge Base (High Quality / Vector search / Rerank OFF). When we query by uploading an image that should be identical to an image indexed from a DOCX, the expected image chunk does not rank at the top, and the similarity score stays very low (~0.03–0.08). Most importantly, we found a strong clue: - The Dify score for an image query matches almost exactly the cosine similarity computed locally as: cosine(query_image_embedding, chunk_image_url_markdown_text_embedding) This suggests the retrieval might be comparing image embedding ↔ image-URL/markdown text embedding, instead of image ↔ image embedding similarity. (I checked the code but couldn’t clearly identify where/why this path is taken.) Questions: 1) In v1.11.4, is it expected that Search Test returns only text segments even for image queries (i.e., not returning IMAGE attachment chunks / image embeddings)? 2) For DOCX extracted images in High Quality multimodal KB, are IMAGE chunks stored separately (e.g., doc_type=IMAGE or similar) in the vector DB? If yes, what is the recommended way to verify they exist and are actually used during retrieval? Thanks in advance.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 👻 feat:rag
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#21848
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?