langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 04:25:29 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Add scheduler for documents upload #15322

New Issue
Closed
opened 2026-02-21 19:21:12 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 19:21:12 -05:00
Owner
Copy Link
Copy Source

Originally created by @kelvintsim on GitHub (Jul 13, 2025).

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Add scheduler for documents upload to prevent database spike when office hours or high usage period.

2. Additional context or comments

No response

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @kelvintsim on GitHub (Jul 13, 2025). ### Self Checks - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? Tell me about your story. Add scheduler for documents upload to prevent database spike when office hours or high usage period. ### 2. Additional context or comments _No response_ ### 3. Can you help us with this feature? - [ ] I am interested in contributing to this feature.
yindo added the 💪 enhancement label 2026-02-21 19:21:12 -05:00
yindo closed this issue 2026-02-21 19:21:12 -05:00
yindo commented 2026-02-21 19:21:12 -05:00
Author
Owner
Copy Link
Copy Source

@jubinsoni commented on GitHub (Jul 16, 2025):

@kelvintsim
can we assume that we will receive following info from user regarding

  1. Directory from where the document will be uploaded (all the docs within this directory will be uploaded)
  2. dataset-id(KB)
  3. time at which this daily task will run
@jubinsoni commented on GitHub (Jul 16, 2025): @kelvintsim can we assume that we will receive following info from user regarding 1) Directory from where the document will be uploaded (all the docs within this directory will be uploaded) 2) dataset-id(KB) 3) time at which this daily task will run
yindo commented 2026-02-21 19:21:12 -05:00
Author
Owner
Copy Link
Copy Source

@jubinsoni commented on GitHub (Aug 20, 2025):

@crazywoola
to add this feature
https://github.com/langgenius/dify/pull/24218

pls review

@jubinsoni commented on GitHub (Aug 20, 2025): @crazywoola to add this feature https://github.com/langgenius/dify/pull/24218 pls review
yindo commented 2026-02-21 19:21:12 -05:00
Author
Owner
Copy Link
Copy Source

@jubinsoni commented on GitHub (Aug 20, 2025):

@crazywoola

following is the flow

  1. upload file via /schedule-by-file api
  2. _check_rate_limit function(considering the peak hours and off-peak hours) checks weather we have to enqueue the message or directly process it
  3. if message is enqueued in step 2 then we process this message in process_queue where I ensured that I'm using zpopmin so that race condition is avoided by doing an atomic operation
  4. then _process_upload is called which does uploading of doc and making it available in KB(Knowledge Base)

all the configuration regarding rate limiting is added in configuration files

@jubinsoni commented on GitHub (Aug 20, 2025): @crazywoola following is the flow 1) upload file via /schedule-by-file api 2) _check_rate_limit function(considering the peak hours and off-peak hours) checks weather we have to enqueue the message or directly process it 3) if message is enqueued in step 2 then we process this message in process_queue where I ensured that I'm using zpopmin so that race condition is avoided by doing an atomic operation 4) then _process_upload is called which does uploading of doc and making it available in KB(Knowledge Base) all the configuration regarding rate limiting is added in configuration files
yindo commented 2026-02-21 19:21:12 -05:00
Author
Owner
Copy Link
Copy Source

@jubinsoni commented on GitHub (Aug 25, 2025):

@crazywoola fixed all the build issues and retested

Image
@jubinsoni commented on GitHub (Aug 25, 2025): @crazywoola fixed all the build issues and retested <img width="1052" height="652" alt="Image" src="https://github.com/user-attachments/assets/cd54cc72-a0f9-4ff9-9684-5060b1a8e7ac" />
yindo commented 2026-02-21 19:21:12 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 25, 2025):

Hi, @kelvintsim. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You requested a scheduler feature for document uploads to reduce database load during peak hours.
  • Contributor jubinsoni developed a rate-limited upload API with queue processing to avoid race conditions.
  • A pull request was submitted, reviewed by maintainer crazywoola, and build issues were resolved.
  • The feature now includes configurable rate limiting to manage upload timing effectively.
  • The issue was resolved with the successful implementation and confirmation of the solution.

Next Steps:

  • Please confirm if this feature is still relevant or working as expected in the latest version of Dify by commenting on this issue.
  • If no further updates are provided, I will automatically close this issue in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Sep 25, 2025): Hi, @kelvintsim. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You requested a scheduler feature for document uploads to reduce database load during peak hours. - Contributor jubinsoni developed a rate-limited upload API with queue processing to avoid race conditions. - A pull request was submitted, reviewed by maintainer crazywoola, and build issues were resolved. - The feature now includes configurable rate limiting to manage upload timing effectively. - The issue was resolved with the successful implementation and confirmation of the solution. **Next Steps:** - Please confirm if this feature is still relevant or working as expected in the latest version of Dify by commenting on this issue. - If no further updates are provided, I will automatically close this issue in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 💪 enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#15322
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?