langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #29247] fix: make remove-orphaned-files-on-storage management command work and safer #32342

New Issue
Closed
opened 2026-02-21 20:51:13 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:51:13 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/29247

State: closed
Merged: Yes

Summary

This PR addresses the following two issues about administrative command remove-orphaned-files-on-storage:

  • When creating a list of files on the storage, recursive search was no longer being performed, resulting in no files being detected at all.
  • When running the command with issues connecting to the database, unintended files could be deleted.

The changes by this PR are:

  • ✅ Change storage.scan() to use Operator.scan() instead of Operator.list().

    • Initially, Operator.scan() was used, but PR #25874 changed it to Operator.list(). As a result, recursive search of files on storage was no longer performed.
    • That PR described the change as replace deprecated scan method to list method, but this was not appropriate because:
      • storage.scan() was intended for recursive search, so list() is not a suitable replacement.
      • The latest Python binding for OpenDAL published on PyPI is 0.46.0, and the deprecation of scan is from 0.47.0 onwards.
      • The list method in 0.46.0 does not have a recursive option, so there is no alternative to scan for a recursive search.
    • Therefore, I modified storage.scan() to use Operator.scan() again.

  • ✅ The command will now exit if fetching the list of files from the database table fails.

    • Previously, return was not called in except, so the process continued with an empty file list.
    • As a result, all files on storage could be incorrectly determined as unnecessary and deleted.

Regarding the side effects of reverting from list to scan, actually scan works better as follows:

  • Both PR #22551 and PR #25828 use storage.scan(), and based on its commits and timeline, it is clear that they were implemented assuming Operator.scan() was available.

Closes #29246

Screenshots

Before

$ docker exec -it docker-api-1 flask clear-orphaned-file-records
...
Starting orphaned file records cleanup.
- Listing message_files records where message_id doesn't exist in messages table
No orphaned message_files records found. There is nothing to delete.
- Listing file records in table upload_files
- Listing file records in table tool_files
Found 1 files in tables.
- Listing file ids in column upload_file_id in table message_files
- Listing file-id-like strings in column data_source_info in table documents
- Listing file-id-like strings in column content in table document_segments
- Listing file-id-like strings in column answer in table messages
- Listing file-id-like strings in column inputs in table workflow_node_executions
- Listing file-id-like strings in column process_data in table workflow_node_executions
- Listing file-id-like strings in column outputs in table workflow_node_executions
- Listing file-id-like strings in column introduction in table conversations
- Listing file-id-like strings in column system_instruction in table conversations
- Listing file-id-like strings in column avatar in table accounts
- Listing file-id-like strings in column icon in table apps
- Listing file-id-like strings in column icon in table sites
- Listing file-id-like JSON string in column inputs in table messages
- Listing file-id-like JSON string in column message in table messages
Found 0 file ids in tables.
Found 1 orphaned file records.
- orphaned file id: 4f3d8cf5-5f24-4271-b8a8-7a0da8f1ee96
Do you want to proceed to delete all 1 orphaned file records? [y/N]: y
- Deleting orphaned file records in table upload_files
- Deleting orphaned file records in table tool_files
Removed 1 orphaned file records.   ✅✅✅ Record deleted.

$ docker exec -it docker-api-1 flask remove-orphaned-files-on-storage
...
Starting orphaned files cleanup.
- Listing files from table upload_files
- Listing files from table tool_files
Found 0 files in tables.
- Scanning files on storage path image_files
  -> Skipping path image_files as it does not exist.
- Scanning files on storage path tools
  -> Skipping path tools as it does not exist.
- Scanning files on storage path upload_files
Found 0 files on storage.                            👈👈👈 An orphaned file should be found.
No orphaned files found. There is nothing to remove.

After

$ docker exec -it docker-api-1 flask clear-orphaned-file-records     
...
Starting orphaned file records cleanup.
- Listing message_files records where message_id doesn't exist in messages table
No orphaned message_files records found. There is nothing to delete.
- Listing file records in table upload_files
- Listing file records in table tool_files
Found 1 files in tables.
- Listing file ids in column upload_file_id in table message_files
- Listing file-id-like strings in column data_source_info in table documents
- Listing file-id-like strings in column content in table document_segments
- Listing file-id-like strings in column answer in table messages
- Listing file-id-like strings in column inputs in table workflow_node_executions
- Listing file-id-like strings in column process_data in table workflow_node_executions
- Listing file-id-like strings in column outputs in table workflow_node_executions
- Listing file-id-like strings in column introduction in table conversations
- Listing file-id-like strings in column system_instruction in table conversations
- Listing file-id-like strings in column avatar in table accounts
- Listing file-id-like strings in column icon in table apps
- Listing file-id-like strings in column icon in table sites
- Listing file-id-like JSON string in column inputs in table messages
- Listing file-id-like JSON string in column message in table messages
Found 0 file ids in tables.
Found 1 orphaned file records.
- orphaned file id: 4f3d8cf5-5f24-4271-b8a8-7a0da8f1ee96
Do you want to proceed to delete all 1 orphaned file records? [y/N]: y
- Deleting orphaned file records in table upload_files
- Deleting orphaned file records in table tool_files
Removed 1 orphaned file records.   ✅✅✅ Record deleted.

$ docker exec -it docker-api-1 flask remove-orphaned-files-on-storage
...
Starting orphaned files cleanup.
- Listing files from table upload_files
- Listing files from table tool_files
Found 0 files in tables.
- Scanning files on storage path image_files
  -> Skipping path image_files as it does not exist.
- Scanning files on storage path tools
  -> Skipping path tools as it does not exist.
- Scanning files on storage path upload_files
Found 1 files on storage.   ✅✅✅ Detected.
Found 1 orphaned files.
- orphaned file: upload_files/1536a781-2bb7-4085-8585-463c3151e1fe/26b9abb3-834c-41f3-ad06-febf490c881c.md
Do you want to proceed to remove all 1 orphaned files? [y/N]: y
- Removing orphaned file: upload_files/1536a781-2bb7-4085-8585-463c3151e1fe/26b9abb3-834c-41f3-ad06-febf490c881c.md
Removed 1 orphaned files without errors.

## Checklist

- [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs)
- [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
- [ ] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
- [ ] I've updated the documentation accordingly.
- [ ] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods
**Original Pull Request:** https://github.com/langgenius/dify/pull/29247 **State:** closed **Merged:** Yes --- ## Summary This PR addresses the following two issues about administrative command `remove-orphaned-files-on-storage`: - When creating a list of files on the storage, recursive search was no longer being performed, resulting in no files being detected at all. - When running the command with issues connecting to the database, unintended files could be deleted. The changes by this PR are: - ✅ Change `storage.scan()` to use `Operator.scan()` instead of `Operator.list()`. - Initially, `Operator.scan()` was used, but PR #25874 changed it to `Operator.list()`. As a result, recursive search of files on storage was no longer performed. - That PR described the change as `replace deprecated scan method to list method`, but this was not appropriate because: - `storage.scan()` was intended for recursive search, so `list()` is not a suitable replacement. - The latest Python binding for OpenDAL published on PyPI is `0.46.0`, and the deprecation of `scan` is from `0.47.0` onwards. - The `list` method in `0.46.0` does not have a `recursive` option, so there is no alternative to `scan` for a recursive search. - Therefore, I modified `storage.scan()` to use `Operator.scan()` again. - ✅ The command will now exit if fetching the list of files from the database table fails. - Previously, `return` was not called in `except`, so the process continued with an empty file list. - As a result, all files on storage could be incorrectly determined as unnecessary and deleted. Regarding the side effects of reverting from `list` to `scan`, actually `scan` works better as follows: - Both PR #22551 and PR #25828 use `storage.scan()`, and based on its commits and timeline, it is clear that they were implemented assuming `Operator.scan()` was available. Closes #29246 ## Screenshots ### Before ```bash $ docker exec -it docker-api-1 flask clear-orphaned-file-records ... Starting orphaned file records cleanup. - Listing message_files records where message_id doesn't exist in messages table No orphaned message_files records found. There is nothing to delete. - Listing file records in table upload_files - Listing file records in table tool_files Found 1 files in tables. - Listing file ids in column upload_file_id in table message_files - Listing file-id-like strings in column data_source_info in table documents - Listing file-id-like strings in column content in table document_segments - Listing file-id-like strings in column answer in table messages - Listing file-id-like strings in column inputs in table workflow_node_executions - Listing file-id-like strings in column process_data in table workflow_node_executions - Listing file-id-like strings in column outputs in table workflow_node_executions - Listing file-id-like strings in column introduction in table conversations - Listing file-id-like strings in column system_instruction in table conversations - Listing file-id-like strings in column avatar in table accounts - Listing file-id-like strings in column icon in table apps - Listing file-id-like strings in column icon in table sites - Listing file-id-like JSON string in column inputs in table messages - Listing file-id-like JSON string in column message in table messages Found 0 file ids in tables. Found 1 orphaned file records. - orphaned file id: 4f3d8cf5-5f24-4271-b8a8-7a0da8f1ee96 Do you want to proceed to delete all 1 orphaned file records? [y/N]: y - Deleting orphaned file records in table upload_files - Deleting orphaned file records in table tool_files Removed 1 orphaned file records. ✅✅✅ Record deleted. $ docker exec -it docker-api-1 flask remove-orphaned-files-on-storage ... Starting orphaned files cleanup. - Listing files from table upload_files - Listing files from table tool_files Found 0 files in tables. - Scanning files on storage path image_files -> Skipping path image_files as it does not exist. - Scanning files on storage path tools -> Skipping path tools as it does not exist. - Scanning files on storage path upload_files Found 0 files on storage. 👈👈👈 An orphaned file should be found. No orphaned files found. There is nothing to remove. ``` ### After ```bash $ docker exec -it docker-api-1 flask clear-orphaned-file-records ... Starting orphaned file records cleanup. - Listing message_files records where message_id doesn't exist in messages table No orphaned message_files records found. There is nothing to delete. - Listing file records in table upload_files - Listing file records in table tool_files Found 1 files in tables. - Listing file ids in column upload_file_id in table message_files - Listing file-id-like strings in column data_source_info in table documents - Listing file-id-like strings in column content in table document_segments - Listing file-id-like strings in column answer in table messages - Listing file-id-like strings in column inputs in table workflow_node_executions - Listing file-id-like strings in column process_data in table workflow_node_executions - Listing file-id-like strings in column outputs in table workflow_node_executions - Listing file-id-like strings in column introduction in table conversations - Listing file-id-like strings in column system_instruction in table conversations - Listing file-id-like strings in column avatar in table accounts - Listing file-id-like strings in column icon in table apps - Listing file-id-like strings in column icon in table sites - Listing file-id-like JSON string in column inputs in table messages - Listing file-id-like JSON string in column message in table messages Found 0 file ids in tables. Found 1 orphaned file records. - orphaned file id: 4f3d8cf5-5f24-4271-b8a8-7a0da8f1ee96 Do you want to proceed to delete all 1 orphaned file records? [y/N]: y - Deleting orphaned file records in table upload_files - Deleting orphaned file records in table tool_files Removed 1 orphaned file records. ✅✅✅ Record deleted. $ docker exec -it docker-api-1 flask remove-orphaned-files-on-storage ... Starting orphaned files cleanup. - Listing files from table upload_files - Listing files from table tool_files Found 0 files in tables. - Scanning files on storage path image_files -> Skipping path image_files as it does not exist. - Scanning files on storage path tools -> Skipping path tools as it does not exist. - Scanning files on storage path upload_files Found 1 files on storage. ✅✅✅ Detected. Found 1 orphaned files. - orphaned file: upload_files/1536a781-2bb7-4085-8585-463c3151e1fe/26b9abb3-834c-41f3-ad06-febf490c881c.md Do you want to proceed to remove all 1 orphaned files? [y/N]: y - Removing orphaned file: upload_files/1536a781-2bb7-4085-8585-463c3151e1fe/26b9abb3-834c-41f3-ad06-febf490c881c.md Removed 1 orphaned files without errors. ``` ``` ## Checklist - [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) - [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!) - [ ] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change. - [ ] I've updated the documentation accordingly. - [ ] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods
yindo added the pull-request label 2026-02-21 20:51:13 -05:00
yindo closed this issue 2026-02-21 20:51:13 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#32342
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?