langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-23 04:55:28 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

The table 'dataset_keyword_tables' is too large, resulting in insufficient server storage #5606

New Issue
Closed
opened 2026-02-21 18:11:51 -05:00 by yindo · 7 comments
yindo commented 2026-02-21 18:11:51 -05:00
Owner
Copy Link
Copy Source

Originally created by @starweavingdream on GitHub (Sep 13, 2024).

Originally assigned to: @takatost on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.8.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Hello, I am using a PG database and there is a table named 'dataset_keyword_tables' in the database with a large amount of data; Through testing, uploading a file of about 1M, the size increment of this table can reach 300M, which is strange. Now my table has 300 GB, and I want to know what specific use this table has? Can I clean it up? Otherwise, as more and more knowledge base documents are uploaded, this table will become larger and larger.

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @starweavingdream on GitHub (Sep 13, 2024). Originally assigned to: @takatost on GitHub. ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.8.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Hello, I am using a PG database and there is a table named 'dataset_keyword_tables' in the database with a large amount of data; Through testing, uploading a file of about 1M, the size increment of this table can reach 300M, which is strange. Now my table has 300 GB, and I want to know what specific use this table has? Can I clean it up? Otherwise, as more and more knowledge base documents are uploaded, this table will become larger and larger. ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug label 2026-02-21 18:11:51 -05:00
yindo closed this issue 2026-02-21 18:11:51 -05:00
yindo commented 2026-02-21 18:11:51 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 13, 2024):

Hey @starweavingdream! I'm here to help you with any bugs, questions, or contributions you have for the project. Let me know how I can assist you!

The dataset_keyword_tables table in the PG database is used to store keyword tables associated with datasets. Each entry in this table includes a unique identifier, the dataset ID it is associated with, the keyword table itself (stored as text), and the data source type.

To clean up the dataset_keyword_tables table and prevent it from becoming excessively large, you can remove entries that are no longer needed. This can be done by identifying and deleting rows that are associated with datasets that are no longer in use or have been deleted.

Here is an example of how you might perform such a cleanup operation:

from extensions.ext_database import db
from models.dataset import Dataset, DatasetKeywordTable

def cleanup_dataset_keyword_tables():
    # Find all dataset IDs that are still in use
    active_dataset_ids = {dataset.id for dataset in Dataset.query.all()}

    # Find all entries in the dataset_keyword_tables
    all_keyword_tables = DatasetKeywordTable.query.all()

    # Delete entries that are not associated with active datasets
    for keyword_table in all_keyword_tables:
        if keyword_table.dataset_id not in active_dataset_ids:
            db.session.delete(keyword_table)

    # Commit the changes to the database
    db.session.commit()

# Call the cleanup function
cleanup_dataset_keyword_tables()

Additionally, Dify has an automated cleanup mechanism in the form of a scheduled task named clean_unused_datasets_task that manages the size of the 'dataset_keyword_tables' table. This task removes unused datasets and their associated indexes if they haven't been updated within a specified number of days [1].

For more details on the structure and indexing of the table, you can refer to the initial database setup in the Dify platform [2][3].

If you need to limit the size of the 'dataset_keyword_tables' table, you can use the max_keywords_per_chunk setting in the KeywordTableConfig class, which controls the number of keywords extracted per chunk of text [4].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Sep 13, 2024): <!-- Greeting --> Hey @starweavingdream! I'm here to help you with any bugs, questions, or contributions you have for the project. Let me know how I can assist you! <!-- Answer --> The `dataset_keyword_tables` table in the PG database is used to store keyword tables associated with datasets. Each entry in this table includes a unique identifier, the dataset ID it is associated with, the keyword table itself (stored as text), and the data source type. To clean up the `dataset_keyword_tables` table and prevent it from becoming excessively large, you can remove entries that are no longer needed. This can be done by identifying and deleting rows that are associated with datasets that are no longer in use or have been deleted. Here is an example of how you might perform such a cleanup operation: ```python from extensions.ext_database import db from models.dataset import Dataset, DatasetKeywordTable def cleanup_dataset_keyword_tables(): # Find all dataset IDs that are still in use active_dataset_ids = {dataset.id for dataset in Dataset.query.all()} # Find all entries in the dataset_keyword_tables all_keyword_tables = DatasetKeywordTable.query.all() # Delete entries that are not associated with active datasets for keyword_table in all_keyword_tables: if keyword_table.dataset_id not in active_dataset_ids: db.session.delete(keyword_table) # Commit the changes to the database db.session.commit() # Call the cleanup function cleanup_dataset_keyword_tables() ``` Additionally, Dify has an automated cleanup mechanism in the form of a scheduled task named `clean_unused_datasets_task` that manages the size of the 'dataset_keyword_tables' table. This task removes unused datasets and their associated indexes if they haven't been updated within a specified number of days <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/schedule/clean_unused_datasets_task.py)</sup>. For more details on the structure and indexing of the table, you can refer to the initial database setup in the Dify platform <sup>[[2]](https://github.com/langgenius/dify/blob/main/api/models/dataset.py)</sup><sup>[[3]](https://github.com/langgenius/dify/blob/main/api/migrations/versions/64b051264f32_init.py)</sup>. If you need to limit the size of the 'dataset_keyword_tables' table, you can use the `max_keywords_per_chunk` setting in the `KeywordTableConfig` class, which controls the number of keywords extracted per chunk of text <sup>[[4]](https://github.com/langgenius/dify/blob/main/api/core/rag/datasource/keyword/jieba/jieba.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:11:51 -05:00
Author
Owner
Copy Link
Copy Source

@Scallions commented on GitHub (Oct 10, 2024):

Same problem. Did you find any solution? I found that this table is only used by keyword search. When indexing document, this table will slow down the process.

@Scallions commented on GitHub (Oct 10, 2024): Same problem. Did you find any solution? I found that this table is only used by keyword search. When indexing document, this table will slow down the process.
yindo commented 2026-02-21 18:11:51 -05:00
Author
Owner
Copy Link
Copy Source

@starweavingdream commented on GitHub (Oct 15, 2024):

@Scallions I tried the solution given by the robot, and there were no particularly obvious problems after I cleared the data. If you want to try my method, I suggest you back up your data first.

@starweavingdream commented on GitHub (Oct 15, 2024): @Scallions I tried the solution given by the robot, and there were no particularly obvious problems after I cleared the data. If you want to try my method, I suggest you back up your data first.
yindo commented 2026-02-21 18:11:51 -05:00
Author
Owner
Copy Link
Copy Source

@glacierck commented on GitHub (Oct 22, 2024):

@starweavingdream I need you! I'm going nuts!
image

@glacierck commented on GitHub (Oct 22, 2024): @starweavingdream I need you! I'm going nuts! ![image](https://github.com/user-attachments/assets/c4b32d43-9da0-425c-b0cc-bc999cac2b6e)
yindo commented 2026-02-21 18:11:52 -05:00
Author
Owner
Copy Link
Copy Source

@luckylhb90 commented on GitHub (Nov 1, 2024):

same problems. ....
so difficult.....
The system has a lot to change.....

@luckylhb90 commented on GitHub (Nov 1, 2024): same problems. .... so difficult..... The system has a lot to change.....
yindo commented 2026-02-21 18:11:52 -05:00
Author
Owner
Copy Link
Copy Source

@glacierck commented on GitHub (Jan 22, 2025):

This is my solution, disable the data of this table！

https://github.com/langgenius/dify/issues/8513#issuecomment-2606650048

@glacierck commented on GitHub (Jan 22, 2025): This is my solution, disable the data of this table！ https://github.com/langgenius/dify/issues/8513#issuecomment-2606650048
yindo commented 2026-02-21 18:11:52 -05:00
Author
Owner
Copy Link
Copy Source

@gemchen commented on GitHub (Mar 7, 2025):

@Scallions I tried the solution given by the robot, and there were no particularly obvious problems after I cleared the data. If you want to try my method, I suggest you back up your data first.

Did it work? 好使吗？

@gemchen commented on GitHub (Mar 7, 2025): > [@Scallions](https://github.com/Scallions) I tried the solution given by the robot, and there were no particularly obvious problems after I cleared the data. If you want to try my method, I suggest you back up your data first. Did it work? 好使吗？
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#5606
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?