langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #18835] feat: add administrative commands to free up storage space by removing unused files #29011

New Issue
Closed
opened 2026-02-21 20:44:42 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:44:42 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/18835

State: closed
Merged: Yes

Summary

This PR adds following new administrative commands:

  • flask clear-orphaned-file-records:
    • Delete records from the upload_files table and the tool_files table that are not referenced by any conversation logs or node execution logs.
  • flask remove-orphaned-files-on-storage
    • Remove files in storage that do not have corresponding records in either upload_files or tool_files.

The administrator of the Community Edition can increase the available storage space by using these two commands after deleting old logs with the existing clear-free-plan-tenant-expired-logs commands.

Related:

Detailed Description

In the current implementation, when a user uploads a file or a file is generated by a tool, the file is stored in the storage, and a record is added to either the upload_files or tool_files table for each file.

These files and records can become unreferenced through operations like the following.

  • After attaching images or files to the chat, close the chat before sending or delete the files.
  • After uploading documents to the knowledge base, close the browser or delete the files before performing the embedding process.
  • Delete conversation logs that used images or files by existing clear-free-plan-tenant-expired-logs command.
  • etc.

In these situations, unreferenced records and files can be deleted safely, but currently, there is no ways to do so, resulting in an increase in storage usage.

This PR provides a way to clean both the DB and storage in two steps.

  • Step 1: Delete records from the upload_files table and the tool_files table that are not referenced by any conversation logs or node execution logs.
  • Step 2: Remove files in storage that do not have corresponding records in either upload_files or tool_files.

Step 1: flask clear-orphaned-file-records

  • Enumerate all the files in following tables (A).
    • upload_files
    • tool_files
  • Enumerate all the strings in GUID format from the following tables and columns (B).
    • message_files.upload_file_id
    • documents.data_source_info
    • document_segments.content
    • messages.inputs
    • messages.message
    • messages.answer
    • workflow_node_executions.inputs
    • workflow_node_executions.process_data
    • workflow_node_executions.outputs
    • conversations.introduction
    • conversations.system_instruction
  • Compare A and B and enumerate those in A whose IDs are not included in B (C).
  • All records for C will be deleted from upload_files and tool_files.

Step 2: flask remove-orphaned-files-on-storage

  • Enumerate all the files in following table (A).
    • upload_files
    • tool_files
  • Enumerate the files in following directory of the storage (B).
    • image_files
    • tools
    • upload_files
  • Compare A and B and enumerate those in B that are not included in A (C).
  • All files in C will be deleted.

For this purpose, I have added a scan method to Storage interface. Since this PR only implements it in OpenDAL, Step 2 will work only in OpenDAL. If we implement the scan method for other storage types, this command will be able to support additional storage as well.

Screenshots

flask clear-orphaned-file-records:
image

flask remove-orphaned-files-on-storage
image

Checklist

Important

Please review the checklist below before submitting your pull request.

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
**Original Pull Request:** https://github.com/langgenius/dify/pull/18835 **State:** closed **Merged:** Yes --- # Summary This PR adds following new administrative commands: - `flask clear-orphaned-file-records`: - Delete records from the `upload_files` table and the `tool_files` table that are not referenced by any conversation logs or node execution logs. - `flask remove-orphaned-files-on-storage` - Remove files in storage that do not have corresponding records in either `upload_files` or `tool_files`. The administrator of the Community Edition can increase the available storage space by using these two commands after deleting old logs with the existing `clear-free-plan-tenant-expired-logs` commands. Related: - Close https://github.com/langgenius/dify/issues/18400 - Close https://github.com/langgenius/dify/discussions/16376 - Close https://github.com/langgenius/dify/discussions/3863 - Close https://github.com/langgenius/dify/discussions/11546 - Close https://github.com/langgenius/dify/issues/18814 # Detailed Description In the current implementation, when a user uploads a file or a file is generated by a tool, the file is stored in the storage, and a record is added to either the `upload_files` or `tool_files` table for each file. These files and records can become unreferenced through operations like the following. - After attaching images or files to the chat, close the chat before sending or delete the files. - After uploading documents to the knowledge base, close the browser or delete the files before performing the embedding process. - Delete conversation logs that used images or files by existing `clear-free-plan-tenant-expired-logs` command. - etc. In these situations, unreferenced records and files can be deleted safely, but currently, there is no ways to do so, resulting in an increase in storage usage. This PR provides a way to clean both the DB and storage in two steps. - **Step 1**: Delete records from the `upload_files` table and the `tool_files` table that are not referenced by any conversation logs or node execution logs. - **Step 2**: Remove files in storage that do not have corresponding records in either `upload_files` or `tool_files`. ## Step 1: `flask clear-orphaned-file-records` - Enumerate all the files in following tables (A). - upload_files - tool_files - Enumerate all the strings in GUID format from the following tables and columns (B). - message_files.upload_file_id - documents.data_source_info - document_segments.content - messages.inputs - messages.message - messages.answer - workflow_node_executions.inputs - workflow_node_executions.process_data - workflow_node_executions.outputs - conversations.introduction - conversations.system_instruction - Compare A and B and enumerate those in A whose IDs are not included in B (C). - All records for C will be deleted from upload_files and tool_files. ## Step 2: `flask remove-orphaned-files-on-storage` - Enumerate all the files in following table (A). - upload_files - tool_files - Enumerate the files in following directory of the storage (B). - image_files - tools - upload_files - Compare A and B and enumerate those in B that are not included in A (C). - All files in C will be deleted. For this purpose, I have added a `scan` method to `Storage` interface. Since this PR only implements it in OpenDAL, Step 2 will work only in OpenDAL. If we implement the `scan` method for other storage types, this command will be able to support additional storage as well. # Screenshots `flask clear-orphaned-file-records`: ![image](https://github.com/user-attachments/assets/6cce3b0d-44ab-44f5-b467-d11172b65a77) `flask remove-orphaned-files-on-storage` ![image](https://github.com/user-attachments/assets/53cc8483-4955-475d-830e-0554fe64a1f4) # Checklist > [!IMPORTANT] > Please review the checklist below before submitting your pull request. - [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) - [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!) - [ ] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change. - [ ] I've updated the documentation accordingly. - [x] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods
yindo added the pull-request label 2026-02-21 20:44:42 -05:00
yindo closed this issue 2026-02-21 20:44:42 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#29011
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?