langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 03:55:25 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

The text data imported from the knowledge base was not found in the vector database. #862

New Issue
Closed
opened 2026-02-21 17:28:45 -05:00 by yindo · 8 comments
yindo commented 2026-02-21 17:28:45 -05:00
Owner
Copy Link
Copy Source

Originally created by @andylzming on GitHub (Jan 4, 2024).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

Dify version

0.4.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

May I ask whether both the annotated data and the imported text data from the knowledge base are stored in the vector database through vectorization? In actuality, the annotated data can be found in the vector data, but the text data from the knowledge base is not found in the vector database.

How can the vectorized data similarity matching be implemented when the imported text data from the knowledge base is stored in a PostgreSQL database?

  • knowledge data

1

  • annotated data

3

  • vector database

2

  • PostgreSQL

4

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @andylzming on GitHub (Jan 4, 2024). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to file this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). ### Dify version 0.4.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce May I ask whether both the annotated data and the imported text data from the knowledge base are stored in the vector database through vectorization? In actuality, the annotated data can be found in the vector data, but the text data from the knowledge base is not found in the vector database. How can the vectorized data similarity matching be implemented when the imported text data from the knowledge base is stored in a PostgreSQL database? - knowledge data ![1](https://github.com/langgenius/dify/assets/21255047/b2fb016b-a6f4-43e3-b657-7c2cec2fe837) - annotated data ![3](https://github.com/langgenius/dify/assets/21255047/df12d054-561b-451f-99d8-795053159a79) - vector database ![2](https://github.com/langgenius/dify/assets/21255047/5079e0c4-6c50-4133-acf2-c6faddb1814d) - PostgreSQL ![4](https://github.com/langgenius/dify/assets/21255047/72e65dfb-3fa3-45f4-a94b-710748fea0d3) ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug label 2026-02-21 17:28:45 -05:00
yindo closed this issue 2026-02-21 17:28:45 -05:00
yindo commented 2026-02-21 17:28:45 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Jan 4, 2024):

pls attach the data of 'dataset_collection_bindings' table in postgresql

@JohnJyong commented on GitHub (Jan 4, 2024): pls attach the data of 'dataset_collection_bindings' table in postgresql
yindo commented 2026-02-21 17:28:46 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Jan 4, 2024):

pls add one annotation and check the logs in docker_worker_1 container at the same time

@JohnJyong commented on GitHub (Jan 4, 2024): pls add one annotation and check the logs in docker_worker_1 container at the same time
yindo commented 2026-02-21 17:28:46 -05:00
Author
Owner
Copy Link
Copy Source

@andylzming commented on GitHub (Jan 14, 2024):

pls attach the data of 'dataset_collection_bindings' table in postgresql

Why is knowledge base text data not stored in a vector database, but in PostgreSQL, what is the consideration?

@andylzming commented on GitHub (Jan 14, 2024): > pls attach the data of 'dataset_collection_bindings' table in postgresql Why is knowledge base text data not stored in a vector database, but in PostgreSQL, what is the consideration?
yindo commented 2026-02-21 17:28:46 -05:00
Author
Owner
Copy Link
Copy Source

@andylzming commented on GitHub (Jan 14, 2024):

pls add one annotation and check the logs in docker_worker_1 container at the same time

  • The log information is as follows:

[2024-01-15 15:28:02,107: INFO/MainProcess] Task tasks.document_indexing_task.document_indexing_task[b9c0f2ad-7a95-44f3-ae0b-870737ad0da8] received [2024-01-15 15:28:02,110: INFO/MainProcess] Start process document: b8e2cc75-393b-4b14-b2de-ee971ac31de9 [2024-01-15 15:28:02,812: INFO/MainProcess] Processed dataset: bd66a1d2-d871-42c4-8fe7-4275be32a591 latency: 0.7015920411795378 [2024-01-15 15:28:02,830: INFO/MainProcess] Task tasks.document_indexing_task.document_indexing_task[b9c0f2ad-7a95-44f3-ae0b-870737ad0da8] succeeded in 0.7203535791486502s: None

@andylzming commented on GitHub (Jan 14, 2024): > pls add one annotation and check the logs in docker_worker_1 container at the same time - **The log information is as follows:** `[2024-01-15 15:28:02,107: INFO/MainProcess] Task tasks.document_indexing_task.document_indexing_task[b9c0f2ad-7a95-44f3-ae0b-870737ad0da8] received [2024-01-15 15:28:02,110: INFO/MainProcess] Start process document: b8e2cc75-393b-4b14-b2de-ee971ac31de9 [2024-01-15 15:28:02,812: INFO/MainProcess] Processed dataset: bd66a1d2-d871-42c4-8fe7-4275be32a591 latency: 0.7015920411795378 [2024-01-15 15:28:02,830: INFO/MainProcess] Task tasks.document_indexing_task.document_indexing_task[b9c0f2ad-7a95-44f3-ae0b-870737ad0da8] succeeded in 0.7203535791486502s: None`
yindo commented 2026-02-21 17:28:46 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Jan 16, 2024):

We will save both vector database and postgresql. The data in postgresql serves business logic, and the data in vector database serves retrieval.
Your logs show that your service is running normally ， I'm not sure what your problem is now @andylzming

@JohnJyong commented on GitHub (Jan 16, 2024): We will save both vector database and postgresql. The data in postgresql serves business logic, and the data in vector database serves retrieval. Your logs show that your service is running normally ， I'm not sure what your problem is now @andylzming
yindo commented 2026-02-21 17:28:46 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Jan 16, 2024):

If you have any other questions, please continue to consult， thanks

@JohnJyong commented on GitHub (Jan 16, 2024): If you have any other questions, please continue to consult， thanks
yindo commented 2026-02-21 17:28:46 -05:00
Author
Owner
Copy Link
Copy Source

@andylzming commented on GitHub (Jan 16, 2024):

We will save both vector database and postgresql. The data in postgresql serves business logic, and the data in vector database serves retrieval. Your logs show that your service is running normally ， I'm not sure what your problem is now @andylzming

The knowledge base text data stored in the vector database is incomplete, for instance: only vector data of PDF and doc files are seen, while data of Markdown files are not observed.

@andylzming commented on GitHub (Jan 16, 2024): > We will save both vector database and postgresql. The data in postgresql serves business logic, and the data in vector database serves retrieval. Your logs show that your service is running normally ， I'm not sure what your problem is now @andylzming The knowledge base text data stored in the vector database is incomplete, for instance: only vector data of PDF and doc files are seen, while data of Markdown files are not observed.
yindo commented 2026-02-21 17:28:46 -05:00
Author
Owner
Copy Link
Copy Source

@andylzming commented on GitHub (Jan 16, 2024):

We will save both vector database and postgresql. The data in postgresql serves business logic, and the data in vector database serves retrieval. Your logs show that your service is running normally ， I'm not sure what your problem is now @andylzming

The knowledge base text data stored in the vector database is incomplete, for instance: only vector data of PDF and doc files are seen, while data of Markdown files are not observed.

The issue has been identified. For the knowledge base text data to be vectorized and stored in the vector database, the indexing mode in the knowledge base settings must be set to "High Quality," and the retrieval settings must be set to "Vector Retrieval."

@andylzming commented on GitHub (Jan 16, 2024): > > We will save both vector database and postgresql. The data in postgresql serves business logic, and the data in vector database serves retrieval. Your logs show that your service is running normally ， I'm not sure what your problem is now @andylzming > > The knowledge base text data stored in the vector database is incomplete, for instance: only vector data of PDF and doc files are seen, while data of Markdown files are not observed. The issue has been identified. For the knowledge base text data to be vectorized and stored in the vector database, the indexing mode in the knowledge base settings must be set to "High Quality," and the retrieval settings must be set to "Vector Retrieval."
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#862
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?