langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-20 19:50:06 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Knowledge pipeline: suspected image embeddings not indexed (only URLs) — retrieval fails for text present only in images #21821

New Issue
Open
opened 2026-02-21 20:14:28 -05:00 by yindo · 2 comments
yindo commented 2026-02-21 20:14:28 -05:00
Owner
Copy Link
Copy Source

Originally created by @nicho2 on GitHub (Jan 21, 2026).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.11.4

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Summary
I built a multimodal Knowledge pipeline using MinerU to extract text + images from PDFs, a parent/child chunker, and a Jina multimodal embedding model (jina-clip-v2). The indexed documents show the text content and Markdown image links (URLs). However, queries that rely on text present only inside images fail to retrieve the correct chunks/pages. This strongly suggests that only image URLs are being embedded/indexed (as text) rather than the actual image content being embedded.

In addition, in Chat responses the retriever sometimes seems to “read/describe” images, but the descriptions are often unrelated to the user query, which makes the behavior confusing and hard to diagnose.

  1. Create a Knowledge pipeline:

    • Extractor: MinerU (PDF → Markdown/text + image references)
    • Chunker: Parent/Child
    • Embedding model: jina-clip-v2
    • Index into the KB

  2. Upload a PDF containing images with text inside the images (e.g., UI screenshot, diagram with labels).

  3. Confirm in KB preview / chunk viewer that:

    • Text is present
    • Image references are present as Markdown like ![](https://.../files/...)

  4. Run a query where the answer exists only inside the image (not in surrounding text).

  5. Observe retrieval results.

Image Image Image Image

✔️ Expected Behavior

  • A query for image-only text should retrieve the chunk/page containing the relevant image, because the image should have been embedded as part of the multimodal pipeline (or at least OCR/text extraction should make it searchable if configured).
  • Retriever results should include the correct chunk(s) even if the target string is not present in the plain text.

❌ Actual Behavior

  • Retrieval fails for image-only text queries.
  • Retriever tends to return chunks whose text matches the query and which merely contain an image URL. It looks like the system retrieves based on textual content only and treats the image as a link, not as embeddable content.
  • Example symptom: when querying for content present only inside an image, results contain other chunks where a textual match exists, often including an image URL, but not the correct image/page.

Additionally:

  • From the Chat UI, the model sometimes outputs responses that look like a generic description of a retrieved UI screenshot, e.g. “It seems you show an interface … options like ‘Restart’, ‘Factory reset’, ‘Calibration’…”, but this content is unrelated to the user question. This gives the impression that some image content is being passed to the LLM, but retrieval relevance is incorrect or the image evidence is not properly indexed.
Originally created by @nicho2 on GitHub (Jan 21, 2026). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.11.4 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce **Summary** I built a multimodal Knowledge pipeline using MinerU to extract text + images from PDFs, a parent/child chunker, and a Jina multimodal embedding model (`jina-clip-v2`). The indexed documents show the text content and Markdown image links (URLs). However, queries that rely on **text present only inside images** fail to retrieve the correct chunks/pages. This strongly suggests that **only image URLs are being embedded/indexed (as text)** rather than the actual image content being embedded. In addition, in Chat responses the retriever sometimes seems to “read/describe” images, but the descriptions are **often unrelated to the user query**, which makes the behavior confusing and hard to diagnose. 1. Create a Knowledge pipeline: * Extractor: **MinerU** (PDF → Markdown/text + image references) * Chunker: **Parent/Child** * Embedding model: **`jina-clip-v2`** * Index into the KB 2. Upload a PDF containing images with **text inside the images** (e.g., UI screenshot, diagram with labels). 3. Confirm in KB preview / chunk viewer that: * Text is present * Image references are present as Markdown like `![](https://.../files/...)` 4. Run a query where the answer exists **only inside the image** (not in surrounding text). 5. Observe retrieval results. <img width="1389" height="755" alt="Image" src="https://github.com/user-attachments/assets/be304170-4aea-4872-8d88-cba5ab9acf50" /> <img width="1465" height="496" alt="Image" src="https://github.com/user-attachments/assets/e4be1297-49a9-4529-b71e-ce218fb94284" /> <img width="737" height="431" alt="Image" src="https://github.com/user-attachments/assets/8e074f65-2b07-4a56-8655-d2c22576a7bf" /> <img width="828" height="719" alt="Image" src="https://github.com/user-attachments/assets/b52596ae-068b-41b7-89b0-012480151372" /> ### ✔️ Expected Behavior * A query for **image-only text** should retrieve the chunk/page containing the relevant image, because the image should have been embedded as part of the multimodal pipeline (or at least OCR/text extraction should make it searchable if configured). * Retriever results should include the correct chunk(s) even if the target string is not present in the plain text. ### ❌ Actual Behavior * Retrieval fails for image-only text queries. * Retriever tends to return chunks whose *text* matches the query and which merely contain an image URL. It looks like the system retrieves based on textual content only and treats the image as a link, not as embeddable content. * Example symptom: when querying for content present only inside an image, results contain other chunks where a textual match exists, often including an image URL, but not the correct image/page. Additionally: * From the Chat UI, the model sometimes outputs responses that look like a **generic description of a retrieved UI screenshot**, e.g. “It seems you show an interface … options like ‘Restart’, ‘Factory reset’, ‘Calibration’…”, but **this content is unrelated to the user question**. This gives the impression that some image content is being passed to the LLM, but retrieval relevance is incorrect or the image evidence is not properly indexed.
yindo added the 🐞 bug label 2026-02-21 20:14:28 -05:00
yindo commented 2026-02-21 20:14:28 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 21, 2026):

The root cause is that images are not actually embedded or indexed for retrieval in your current pipeline—only Markdown image links (URLs) are included in the text, so queries for text present only inside images will always fail unless the pipeline is properly configured for multimodal embedding and the model supports image input. By default, Dify's PDF/image extractors do not perform OCR on images, nor do they embed image pixel data unless explicitly set up to do so. The embedding pipeline only processes text content (including Markdown links), not the image content itself source.

To enable image embedding and retrieval with models like jina-clip-v2, you must:

  1. Ensure your embedding model advertises the VISION feature in its schema. This sets the is_multimodal flag for your dataset and enables image input for recall and retrieval. For most setups, this means editing the xinference plugin/provider config for your embedding model (inside your Docker container) so the features list includes VISION, then restarting the relevant services and updating or recreating your knowledge base with the modified model source.
  2. Make sure images are passed as attachments, not just Markdown links. If your workflow uses MinerU, you may need to add a Template node to extract the images array and a Parameter Extractor node to parse it, so images are available for downstream multimodal embedding source.
  3. If you need to retrieve text inside images, you must add an OCR step before ingestion, as Dify does not perform OCR on images by default source.

If your embedding model expects public URLs instead of base64-encoded images, or if the multimodal embedding endpoint is not implemented, embedding will fail. Also, ensure the INTERNAL_FILES_URL environment variable is set in Docker so containers can access image files source.

For more details and troubleshooting, see this issue and workflow handling tips.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Jan 21, 2026): <!-- Answer --> The root cause is that images are not actually embedded or indexed for retrieval in your current pipeline—only Markdown image links (URLs) are included in the text, so queries for text present only inside images will always fail unless the pipeline is properly configured for multimodal embedding and the model supports image input. By default, Dify's PDF/image extractors do not perform OCR on images, nor do they embed image pixel data unless explicitly set up to do so. The embedding pipeline only processes text content (including Markdown links), not the image content itself [source](https://github.com/langgenius/dify/blob/211c57f7b62fe3578c77fcb00aebc5760e8e5466/api/core/rag/extractor/pdf_extractor.py#L83-L176). To enable image embedding and retrieval with models like `jina-clip-v2`, you must: 1. Ensure your embedding model advertises the `VISION` feature in its schema. This sets the `is_multimodal` flag for your dataset and enables image input for recall and retrieval. For most setups, this means editing the xinference plugin/provider config for your embedding model (inside your Docker container) so the `features` list includes `VISION`, then restarting the relevant services and updating or recreating your knowledge base with the modified model [source](https://github.com/langgenius/dify/issues/30401#issuecomment-3701336264). 2. Make sure images are passed as attachments, not just Markdown links. If your workflow uses MinerU, you may need to add a Template node to extract the images array and a Parameter Extractor node to parse it, so images are available for downstream multimodal embedding [source](https://github.com/langgenius/dify/issues/25671#issuecomment-3289483145). 3. If you need to retrieve text inside images, you must add an OCR step before ingestion, as Dify does not perform OCR on images by default [source](https://github.com/langgenius/dify/blob/211c57f7b62fe3578c77fcb00aebc5760e8e5466/api/core/rag/extractor/pdf_extractor.py#L83-L176). If your embedding model expects public URLs instead of base64-encoded images, or if the multimodal embedding endpoint is not implemented, embedding will fail. Also, ensure the `INTERNAL_FILES_URL` environment variable is set in Docker so containers can access image files [source](https://github.com/langgenius/dify/issues/30401#issuecomment-3701336264). For more details and troubleshooting, see [this issue](https://github.com/langgenius/dify/issues/30401#issuecomment-3701336264) and [workflow handling tips](https://github.com/langgenius/dify/issues/25671#issuecomment-3289483145). <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/2deea91a-fa13-4f37-979c-f590f88b8b55?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/2deea91a-fa13-4f37-979c-f590f88b8b55?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/2deea91a-fa13-4f37-979c-f590f88b8b55?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/2deea91a-fa13-4f37-979c-f590f88b8b55?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/2deea91a-fa13-4f37-979c-f590f88b8b55?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/2deea91a-fa13-4f37-979c-f590f88b8b55?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/2deea91a-fa13-4f37-979c-f590f88b8b55?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=The%20root%20cause%20is%20that%20images%20are%20not%20actually%20embedded%20or%20indexed%20for%20retrieval%20in%20your%20current%20pipeline%E2%80%94only%20Markdown%20image%20links%20%28URLs%29%20are%20included%20in%20the%20text%2C%20so%20queries%20for%20text%20present%20only%20inside%20images%20will%20always%20fail%20unless%20the%20pipeline%20is%20properly%20configured%20for%20multimodal%20embedding%20and%20the%20model%20supports%20image%20input.%20By%20default%2C%20Dify%27s%20PDF/image%20extractors%20do%20not%20perform%20OCR%20on%20images%2C%20nor%20do%20they%20embed%20image%20pixel%20data%20unless%20explicitly%20set%20up%20to%20do%20so.%20The%20embedding%20pipeline%20only%20processes%20text%20content%20%28including%20Markdown%20links%29%2C%20not%20the%20image%20content%20itself%20%5Bsource%5D%28https%3A//github.com/langgenius/dify/blob/211c57f7b62fe3578c77fcb00aebc5760e8e5466/api/core/rag/extractor/pdf_extractor.py%23L83-L176%29.%0A%0ATo%20enable%20image%20embedding%20and%20retrieval%20with%20models%20like%20%60jina-clip-v2%60%2C%20you%20must%3A%0A%0A1.%20Ensure%20your%20embedding%20model%20advertises%20the%20%60VISION%60%20feature%20in%20its%20schema.%20This%20sets%20the%20%60is_multimodal%60%20flag%20for%20your%20dataset%20and%20enables%20image%20input%20for%20recall%20and%20retrieval.%20For%20most%20setups%2C%20this%20means%20editing%20the%20xinference%20plugin/provider%20config%20for%20your%20embedding%20model%20%28inside%20your%20Docker%20container%29%20so%20the%20%60features%60%20list%20includes%20%60VISION%60%2C%20then%20restarting%20the%20relevant%20services%20and%20updating%20or%20recreating%20your%20knowledge%20base%20with%20the%20modified%20model%20%5Bsource%5D%28https%3A//github.com/langgenius/dify/issues/30401%23issuecomment-3701336264%29.%0A2.%20Make%20sure%20images%20are%20passed%20as%20attachments%2C%20not%20just%20Markdown%20links.%20If%20your%20workflow%20uses%20MinerU%2C%20you%20may%20need%20to%20add%20a%20Template%20node%20to%20extract%20the%20images%20array%20and%20a%20Parameter%20Extractor%20node%20to%20parse%20it%2C%20so%20images%20are%20available%20for%20downstream%20multimodal%20embedding%20%5Bsource%5D%28https%3A//github.com/langgenius/dify/issues/25671%23issuecomment-3289483145%29.%0A3.%20If%20you%20need%20to%20retrieve%20text%20inside%20images%2C%20you%20must%20add%20an%20OCR%20step%20before%20ingestion%2C%20as%20Dify%20does%20not%20perform%20OCR%20on%20images%20by%20default%20%5Bsource%5D%28https%3A//github.com/langgenius/dify/blob/211c57f7b62fe3578c77fcb00aebc5760e8e5466/api/core/rag/extractor/pdf_extractor.py%23L83-L176%29.%0A%0AIf%20your%20embedding%20model%20expects%20public%20URLs%20instead%20of%20base64-encoded%20images%2C%20or%20if%20the%20multimodal%20embedding%20endpoint%20is%20not%20implemented%2C%20embedding%20will%20fail.%20Also%2C%20ensure%20the%20%60INTERNAL_FILES_URL%60%20environment%20variable%20is%20set%20in%20Docker%20so%20containers%20can%20access%20image%20files%20%5Bsource%5D%28https%3A//github.com/langgenius/dify/issues/30401%23issuecomment-3701336264%29.%0A%0AFor%20more%20details%20and%20troubleshooting%2C%20see%20%5Bthis%20issue%5D%28https%3A//github.com/langgenius/dify/issues/30401%23issuecomment-3701336264%29%20and%20%5Bworkflow%20handling%20tips%5D%28https%3A//github.com/langgenius/dify/issues/25671%23issuecomment-3289483145%29.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/31364)
yindo commented 2026-02-21 20:14:28 -05:00
Author
Owner
Copy Link
Copy Source

@Nastu-Koi commented on GitHub (Jan 30, 2026):

@nicho2 Did you find any solutions to this issue? I encountered the same problem. The results of knowledge retrieval only contains the image_url but not the real image file. I tried these two combinations: 1. text embedding(qwen3 embedding) and text rerank model (bge-rerank-v2-m3). 2. vision embedding & rerank model (jina-embedding-v4 & jina-reranker-m0). The original file is a .docx with texts and images.

@Nastu-Koi commented on GitHub (Jan 30, 2026): @nicho2 Did you find any solutions to this issue? I encountered the same problem. The results of knowledge retrieval only contains the image_url but not the real image file. I tried these two combinations: 1. text embedding(qwen3 embedding) and text rerank model (bge-rerank-v2-m3). 2. vision embedding & rerank model (jina-embedding-v4 & jina-reranker-m0). The original file is a .docx with texts and images.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#21821
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?