langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Infinite loop risk in clean_unused_datasets_task scheduled task #16396

New Issue
Closed
opened 2026-02-21 19:25:57 -05:00 by yindo · 9 comments
yindo commented 2026-02-21 19:25:57 -05:00
Owner
Copy Link
Copy Source

Originally created by @9527MrLi on GitHub (Aug 24, 2025).

Originally assigned to: @kenwoodjw on GitHub.

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

main

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

A critical logic flaw has been identified in api/schedule/clean_unused_datasets_task.py that may cause the scheduled task to enter an infinite loop.

Root Cause

  1. Fixed pagination query: The task always queries the first page (page=1, per_page=50) within a while True loop
  2. Missing cursor mechanism: No tracking of processed datasets, causing the same datasets to be returned in each iteration
  3. Incomplete state updates: When datasets don't meet cleanup criteria or cleanup fails, dataset states remain unchanged

Trigger Conditions

The infinite loop occurs when queried datasets have any of the following conditions:

  • Datasets have recent query records (dataset_query is not empty)
  • Tenant plan type doesn't match the plan_filter condition
  • Exceptions during cleanup are caught but dataset state isn't updated

Code Location

api/schedule/clean_unused_datasets_task.py:89

datasets = db.paginate(stmt, page=1, per_page=50) # Always queries first page

Processing loop

for dataset in datasets:
# Various skip conditions, but dataset state unchanged
if not dataset_query or len(dataset_query) == 0:
# Processing logic...

Impact

  • Task runs indefinitely, consuming system resources
  • Blocks other tasks in the Celery queue
  • May lead to database connection pool exhaustion
Image

✔️ Expected Behavior

  1. Implement cursor pagination: Use offset or track the last processed dataset ID
  2. Add safety mechanisms: Set loop iteration limits or task timeouts
  3. Improve state management: Ensure processed datasets are marked to avoid reprocessing
  4. Enhance error handling: Log failed datasets to avoid repeated attempts

❌ Actual Behavior

No response

Originally created by @9527MrLi on GitHub (Aug 24, 2025). Originally assigned to: @kenwoodjw on GitHub. ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version main ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce A critical logic flaw has been identified in api/schedule/clean_unused_datasets_task.py that may cause the scheduled task to enter an infinite loop. Root Cause 1. Fixed pagination query: The task always queries the first page (page=1, per_page=50) within a while True loop 2. Missing cursor mechanism: No tracking of processed datasets, causing the same datasets to be returned in each iteration 3. Incomplete state updates: When datasets don't meet cleanup criteria or cleanup fails, dataset states remain unchanged Trigger Conditions The infinite loop occurs when queried datasets have any of the following conditions: - Datasets have recent query records (dataset_query is not empty) - Tenant plan type doesn't match the plan_filter condition - Exceptions during cleanup are caught but dataset state isn't updated Code Location # api/schedule/clean_unused_datasets_task.py:89 datasets = db.paginate(stmt, page=1, per_page=50) # Always queries first page # Processing loop for dataset in datasets: # Various skip conditions, but dataset state unchanged if not dataset_query or len(dataset_query) == 0: # Processing logic... Impact - Task runs indefinitely, consuming system resources - Blocks other tasks in the Celery queue - May lead to database connection pool exhaustion <img width="1332" height="641" alt="Image" src="https://github.com/user-attachments/assets/87a44b59-0e20-42a7-8e5f-bc7c53bdbdb0" /> ### ✔️ Expected Behavior 1. Implement cursor pagination: Use offset or track the last processed dataset ID 2. Add safety mechanisms: Set loop iteration limits or task timeouts 3. Improve state management: Ensure processed datasets are marked to avoid reprocessing 4. Enhance error handling: Log failed datasets to avoid repeated attempts ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug label 2026-02-21 19:25:57 -05:00
yindo closed this issue 2026-02-21 19:25:57 -05:00
yindo commented 2026-02-21 19:25:57 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 24, 2025):

Hi @9527MrLi! I'm Dosu and I’m helping the dify team.

You're correct—Dify v1.7.2's clean_unused_datasets_task.py has a logic flaw that can cause an infinite loop: it always queries the first page (page=1, per_page=50) in a while True loop, without advancing a cursor or offset, and doesn't reliably update dataset states for skipped or failed cleanups. This means the same datasets can be returned and retried endlessly, especially if they don't meet cleanup criteria or if errors occur during processing, leading to resource exhaustion and blocking other Celery tasks source.

A recent PR, #23828, merged on 2025-08-13, directly addresses this by introducing cursor-based pagination (tracking the last processed dataset ID), explicit state marking for processed datasets, and defensive controls like iteration limits and improved error handling. These changes prevent the task from revisiting the same records and ensure robust, efficient cleanup.

If you're running v1.7.2, upgrading to the latest version is the best way to resolve this. If you need to patch manually, update the task to use cursor/offset-based pagination, mark datasets as processed/skipped/failed after each attempt, and add loop limits or timeouts. This will prevent infinite loops and system overload.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Aug 24, 2025): <!-- Greeting --> Hi @9527MrLi! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the dify team. <!-- Answer --> You're correct—Dify v1.7.2's `clean_unused_datasets_task.py` has a logic flaw that can cause an infinite loop: it always queries the first page (`page=1, per_page=50`) in a `while True` loop, without advancing a cursor or offset, and doesn't reliably update dataset states for skipped or failed cleanups. This means the same datasets can be returned and retried endlessly, especially if they don't meet cleanup criteria or if errors occur during processing, leading to resource exhaustion and blocking other Celery tasks [source](https://github.com/langgenius/dify/blob/f7416eb81003b15adb6c8d05de6daaed567ef2ea/api/schedule/clean_unused_datasets_task.py). A recent PR, [#23828](https://github.com/langgenius/dify/pull/23828), merged on 2025-08-13, directly addresses this by introducing cursor-based pagination (tracking the last processed dataset ID), explicit state marking for processed datasets, and defensive controls like iteration limits and improved error handling. These changes prevent the task from revisiting the same records and ensure robust, efficient cleanup. If you're running v1.7.2, upgrading to the latest version is the best way to resolve this. If you need to patch manually, update the task to use cursor/offset-based pagination, mark datasets as processed/skipped/failed after each attempt, and add loop limits or timeouts. This will prevent infinite loops and system overload. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/cc2b71aa-5fb2-43d0-95a8-40af5c47233a?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/cc2b71aa-5fb2-43d0-95a8-40af5c47233a?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/cc2b71aa-5fb2-43d0-95a8-40af5c47233a?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/cc2b71aa-5fb2-43d0-95a8-40af5c47233a?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/cc2b71aa-5fb2-43d0-95a8-40af5c47233a?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/cc2b71aa-5fb2-43d0-95a8-40af5c47233a?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/cc2b71aa-5fb2-43d0-95a8-40af5c47233a?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/24433)
yindo commented 2026-02-21 19:25:57 -05:00
Author
Owner
Copy Link
Copy Source

@Blackoutta commented on GitHub (Aug 24, 2025):

@dosu which lines in #23828 introduced "cursor-based pagination (tracking the last processed dataset ID), explicit state marking for processed datasets, and defensive controls like iteration limits and improved error handling"?

@Blackoutta commented on GitHub (Aug 24, 2025): @dosu which lines in #23828 introduced "cursor-based pagination (tracking the last processed dataset ID), explicit state marking for processed datasets, and defensive controls like iteration limits and improved error handling"?
yindo commented 2026-02-21 19:25:57 -05:00
Author
Owner
Copy Link
Copy Source

@9527MrLi commented on GitHub (Aug 24, 2025):

@dosu I checked lines 145-146, where marking the enabled status as False can prevent querying the same data repeatedly in the next iteration. However, this approach requires all code to be executed correctly. If the
program encounters an exception that gets caught by the except block, the enabled status won't be updated

@9527MrLi commented on GitHub (Aug 24, 2025): @dosu I checked lines 145-146, where marking the enabled status as False can prevent querying the same data repeatedly in the next iteration. However, this approach requires all code to be executed correctly. If the program encounters an exception that gets caught by the except block, the enabled status won't be updated
yindo commented 2026-02-21 19:25:58 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 24, 2025):

You're absolutely right—if an exception occurs before the enabled status is updated (for example, during index cleanup or log creation), the except block only logs the error and does not mark the dataset or its documents as processed, failed, or disabled. There's no finally block or alternative mechanism to ensure state is updated in error cases, so problematic datasets can be retried indefinitely, leading to potential infinite loops and resource exhaustion source.

To make this more robust, consider moving the state update logic into a finally block or introducing a 'failed' state for datasets that encounter exceptions. This way, even if an error occurs, the dataset won't be selected again in future runs. This change would prevent repeated retries and improve system stability.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Aug 24, 2025): <!-- Answer --> You're absolutely right—if an exception occurs before the enabled status is updated (for example, during index cleanup or log creation), the except block only logs the error and does not mark the dataset or its documents as processed, failed, or disabled. There's no finally block or alternative mechanism to ensure state is updated in error cases, so problematic datasets can be retried indefinitely, leading to potential infinite loops and resource exhaustion [source](https://github.com/langgenius/dify/blob/b78ce1689a31f2fd086a70123bd37c093ec8edd3/api/schedule/clean_unused_datasets_task.py). To make this more robust, consider moving the state update logic into a finally block or introducing a 'failed' state for datasets that encounter exceptions. This way, even if an error occurs, the dataset won't be selected again in future runs. This change would prevent repeated retries and improve system stability. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/24433)
yindo commented 2026-02-21 19:25:58 -05:00
Author
Owner
Copy Link
Copy Source

@Blackoutta commented on GitHub (Aug 24, 2025):

You're absolutely right—if an exception occurs before the enabled status is updated (for example, during index cleanup or log creation), the except block only logs the error and does not mark the dataset or its documents as processed, failed, or disabled. There's no finally block or alternative mechanism to ensure state is updated in error cases, so problematic datasets can be retried indefinitely, leading to potential infinite loops and resource exhaustion source.

To make this more robust, consider moving the state update logic into a finally block or introducing a 'failed' state for datasets that encounter exceptions. This way, even if an error occurs, the dataset won't be selected again in future runs. This change would prevent repeated retries and improve system stability.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@kenwoodjw hi there, could you take a look at this issue, it's pretty critical as it caused an infinite loop on our dify-api-worker, stopping all other tasks being executed.

@Blackoutta commented on GitHub (Aug 24, 2025): > You're absolutely right—if an exception occurs before the enabled status is updated (for example, during index cleanup or log creation), the except block only logs the error and does not mark the dataset or its documents as processed, failed, or disabled. There's no finally block or alternative mechanism to ensure state is updated in error cases, so problematic datasets can be retried indefinitely, leading to potential infinite loops and resource exhaustion [source](https://github.com/langgenius/dify/blob/b78ce1689a31f2fd086a70123bd37c093ec8edd3/api/schedule/clean_unused_datasets_task.py). > > To make this more robust, consider moving the state update logic into a finally block or introducing a 'failed' state for datasets that encounter exceptions. This way, even if an error occurs, the dataset won't be selected again in future runs. This change would prevent repeated retries and improve system stability. > > _To reply, just mention [@dosu](https://go.dosu.dev/dosubot)._ > > How did I do? [Good](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/2fb0e5b0-5609-48b1-a09e-4ecaf7170afc?feedback_type=other)  [![Chat with Dosu](https://camo.githubusercontent.com/d9ffebb979729c9ef0fc4e3090dbae5b17cabbfec8afe378ada6b81913d6ad8a/68747470733a2f2f646f73752e6465762f646f73752d636861742d62616467652e737667)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github) [![Join Discord](https://camo.githubusercontent.com/d562c2df5e4567ea41d37f1d91a4da0839d8341037045b103b2da268998de17f/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6a6f696e2d3538363546323f6c6f676f3d646973636f7264266c6f676f436f6c6f723d7768697465266c6162656c3d)](https://go.dosu.dev/discord-bot) [![Share on X](https://camo.githubusercontent.com/4649f705e2396d6f328dbc8e800a0509c9c17b3b20253b36706c393df124b93d/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f582d73686172652d626c61636b)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/24433) @kenwoodjw hi there, could you take a look at this issue, it's pretty critical as it caused an infinite loop on our dify-api-worker, stopping all other tasks being executed.
yindo commented 2026-02-21 19:25:58 -05:00
Author
Owner
Copy Link
Copy Source

@kenwoodjw commented on GitHub (Aug 24, 2025):

@Blackoutta This PR https://github.com/langgenius/dify/issues/24433 not in v1.7.2， @9527MrLi Could you double confirm you use 1.7.2 version or main branch？

@kenwoodjw commented on GitHub (Aug 24, 2025): @Blackoutta This PR https://github.com/langgenius/dify/issues/24433 not in v1.7.2， @9527MrLi Could you double confirm you use 1.7.2 version or main branch？
yindo commented 2026-02-21 19:25:58 -05:00
Author
Owner
Copy Link
Copy Source

@9527MrLi commented on GitHub (Aug 24, 2025):

@kenwoodjw i'm sorry, I selected the wrong Dify version. I found this error on the main branch.

@9527MrLi commented on GitHub (Aug 24, 2025): @kenwoodjw i'm sorry, I selected the wrong Dify version. I found this error on the main branch.
yindo commented 2026-02-21 19:25:58 -05:00
Author
Owner
Copy Link
Copy Source

@kenwoodjw commented on GitHub (Aug 24, 2025):

@9527MrLi Could you try to checkout to v1.7.2 version, have same issue?

@kenwoodjw commented on GitHub (Aug 24, 2025): @9527MrLi Could you try to checkout to v1.7.2 version, have same issue?
yindo commented 2026-02-21 19:25:58 -05:00
Author
Owner
Copy Link
Copy Source

@9527MrLi commented on GitHub (Aug 25, 2025):

@kenwoodjw We deployed a version earlier than v1.7.2, but the code logic is the same as that of the v1.7.2 version. Currently, the handling logic on the main branch still lacks the mechanism to break out of the infinite loop when a task fails.

@9527MrLi commented on GitHub (Aug 25, 2025): @kenwoodjw We deployed a version earlier than v1.7.2, but the code logic is the same as that of the v1.7.2 version. Currently, the handling logic on the main branch still lacks the mechanism to break out of the infinite loop when a task fails.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#16396
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?