langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Worker often suddenly stops working. #15544

New Issue
Closed
opened 2026-02-21 19:22:13 -05:00 by yindo · 12 comments
yindo commented 2026-02-21 19:22:13 -05:00
Owner
Copy Link
Copy Source

Originally created by @AllenHu2024 on GitHub (Jul 21, 2025).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.4.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. I uploaded over 800 files to my knowledge using parent mode at about 9 AM, then the worker started to work but slowly. I restarted the worker then re-upload.
  2. worker started to work at about 11 AM.
  3. Worker log showed that missed heartbeat about 2 PM.
  4. I find worker didn't work at 8 PM. But the log stopped as below(2025-07-21 19:34:06.251 UTC+8)
    BTW, unstructured service alway shows no log when worker run.
    Image

2025-07-21 11:34:06,195 INFO sqlalchemy.engine.Engine [cached since 2.175e+04s ago] {'enabled': True, 'status': 'completed', 'completed_at': datetime.datetime(2025, 7, 21, 11, 34, 6, 194293), 'document_id_1': '97fb5921-0cdc-488e-9de7-071a52a08200', 'dataset_id_1': '4697e4da-51f2-4e4c-8270-195b20649904', 'status_1': 'indexing', 'index_node_id_1_1': '5a851e22-9a58-4ee7-a399-89dc2886494c'}
2025-07-21 11:34:06,251 INFO sqlalchemy.engine.Engine COMMIT

✔️ Expected Behavior

Worker should keep working and when worker restarted, int can continue to work.
How to restart the embedding process?

❌ Actual Behavior

Worked stopped after a few hours. No log, I can see the log as below before It didn't work.

2025-07-21 14:05:49.063 2025-07-21 14:05:49.063 INFO [Dummy-7] [gossip.py:145] - missed heartbeat from celery@shared-nezha-worker-596575b98-72bm4  
    2025-07-21 14:05:49.063
    2025-07-21 14:05:49.063
    2025-07-21 14:05:49.062
    2025-07-21 14:05:49.062
Originally created by @AllenHu2024 on GitHub (Jul 21, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.4.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. I uploaded over 800 files to my knowledge using parent mode at about 9 AM, then the worker started to work but slowly. I restarted the worker then re-upload. 2. worker started to work at about 11 AM. 3. Worker log showed that missed heartbeat about 2 PM. 4. I find worker didn't work at 8 PM. But the log stopped as below(2025-07-21 19:34:06.251 UTC+8) BTW, unstructured service alway shows no log when worker run. <img width="1466" height="783" alt="Image" src="https://github.com/user-attachments/assets/6bb8c2e7-70a3-49a9-b4fc-f9d4cc08e3f9" /> 2025-07-21 11:34:06,195 INFO sqlalchemy.engine.Engine [cached since 2.175e+04s ago] {'enabled': True, 'status': 'completed', 'completed_at': datetime.datetime(2025, 7, 21, 11, 34, 6, 194293), 'document_id_1': '97fb5921-0cdc-488e-9de7-071a52a08200', 'dataset_id_1': '4697e4da-51f2-4e4c-8270-195b20649904', 'status_1': 'indexing', 'index_node_id_1_1': '5a851e22-9a58-4ee7-a399-89dc2886494c'} 2025-07-21 11:34:06,251 INFO sqlalchemy.engine.Engine COMMIT ### ✔️ Expected Behavior Worker should keep working and when worker restarted, int can continue to work. How to restart the embedding process? ### ❌ Actual Behavior Worked stopped after a few hours. No log, I can see the log as below before It didn't work. 2025-07-21 14:05:49.063 | 2025-07-21 14:05:49.063 INFO [Dummy-7] [gossip.py:145] - missed heartbeat from celery@shared-nezha-worker-596575b98-72bm4 |   -- | -- | --   |   | 2025-07-21 14:05:49.063 | 2025-07-21 14:05:49.063 INFO [Dummy-7] [gossip.py:145] - missed heartbeat from celery@shared-nezha-worker-596575b98-zjvcb |     |   | 2025-07-21 14:05:49.063 | 2025-07-21 14:05:49.062 INFO [Dummy-7] [gossip.py:145] - missed heartbeat from celery@shared-nezha-worker-596575b98-4p4ml |     |   | 2025-07-21 14:05:49.062 | 2025-07-21 14:05:49.062 INFO [Dummy-7] [gossip.py:145] - missed heartbeat from celery@shared-nezha-worker-596575b98-4967k |     |   | 2025-07-21 14:05:49.062 | 2025-07-21 14:05:49.062 INFO [Dummy-7] [gossip.py:145] - missed heartbeat from celery@shared-nezha-worker-596575b98-xbfsk
yindo added the 🐞 bug label 2026-02-21 19:22:13 -05:00
yindo closed this issue 2026-02-21 19:22:13 -05:00
yindo commented 2026-02-21 19:22:13 -05:00
Author
Owner
Copy Link
Copy Source

@AllenHu2024 commented on GitHub (Jul 21, 2025):

Worker config: 1 Core 2G memory
Enterprise Dify.

@AllenHu2024 commented on GitHub (Jul 21, 2025): Worker config: 1 Core 2G memory Enterprise Dify.
yindo commented 2026-02-21 19:22:13 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jul 21, 2025):

Your worker is stopping due to Celery "missed heartbeat" issues, which usually means the worker is too busy (all concurrency slots are filled) to send heartbeat signals to the broker. This is common with large batch jobs like your 800+ file upload. When this happens, Celery marks the worker as offline and stops processing tasks, which matches your logs and symptoms. You can address this by increasing the number of Celery workers or adjusting concurrency settings—specifically, set a higher value for the CELERY_WORKER_AMOUNT environment variable in your Docker Compose or deployment config. This allows more concurrent tasks and helps prevent workers from becoming unresponsive during heavy loads [issue discussion].

To resume or restart embedding/indexing after a worker crash, Dify tracks the state of each document in the database. You can manually trigger recovery or retry tasks using Celery: for example, use the recover_document_indexing_task or retry_document_indexing_task to resume processing for affected documents. These tasks check the last known state and continue from where the process left off. There’s no automatic checkpoint-based resume, so manual intervention is needed if a worker stops mid-job, but the system is designed to be robust for recovery [code reference].

For large jobs, monitor CPU/memory usage and Docker resource limits to ensure your environment can handle the workload. Tuning worker concurrency and heartbeat settings is recommended for production-scale ingestion. If you need to manually resume embedding, use the Celery tasks mentioned above with the appropriate dataset and document IDs.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Join Discord Share on X

@dosubot[bot] commented on GitHub (Jul 21, 2025): <!-- Answer --> Your worker is stopping due to Celery "missed heartbeat" issues, which usually means the worker is too busy (all concurrency slots are filled) to send heartbeat signals to the broker. This is common with large batch jobs like your 800+ file upload. When this happens, Celery marks the worker as offline and stops processing tasks, which matches your logs and symptoms. You can address this by increasing the number of Celery workers or adjusting concurrency settings—specifically, set a higher value for the `CELERY_WORKER_AMOUNT` environment variable in your Docker Compose or deployment config. This allows more concurrent tasks and helps prevent workers from becoming unresponsive during heavy loads [[issue discussion](https://github.com/langgenius/dify/issues/12325)]. To resume or restart embedding/indexing after a worker crash, Dify tracks the state of each document in the database. You can manually trigger recovery or retry tasks using Celery: for example, use the `recover_document_indexing_task` or `retry_document_indexing_task` to resume processing for affected documents. These tasks check the last known state and continue from where the process left off. There’s no automatic checkpoint-based resume, so manual intervention is needed if a worker stops mid-job, but the system is designed to be robust for recovery [[code reference](https://github.com/langgenius/dify/blob/308f1340dd4ff2012815d7ad788fec7d2b438a2e/api/tasks/recover_document_indexing_task.py)]. For large jobs, monitor CPU/memory usage and Docker resource limits to ensure your environment can handle the workload. Tuning worker concurrency and heartbeat settings is recommended for production-scale ingestion. If you need to manually resume embedding, use the Celery tasks mentioned above with the appropriate dataset and document IDs. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/fee8296e-9e16-479e-83c2-80bf618d9249?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/fee8296e-9e16-479e-83c2-80bf618d9249?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/fee8296e-9e16-479e-83c2-80bf618d9249?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/fee8296e-9e16-479e-83c2-80bf618d9249?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/fee8296e-9e16-479e-83c2-80bf618d9249?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/fee8296e-9e16-479e-83c2-80bf618d9249?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/fee8296e-9e16-479e-83c2-80bf618d9249?feedback_type=other)</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/22735)
yindo commented 2026-02-21 19:22:13 -05:00
Author
Owner
Copy Link
Copy Source

@AllenHu2024 commented on GitHub (Jul 21, 2025):

@docu
what number should CELERY_WORKER_AMOUNT be set?
How to run retry_document_indexing_task/recover_document_indexing_task? In api service?

@AllenHu2024 commented on GitHub (Jul 21, 2025): @docu what number should CELERY_WORKER_AMOUNT be set? How to run retry_document_indexing_task/recover_document_indexing_task? In api service?
yindo commented 2026-02-21 19:22:13 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jul 21, 2025):

For a 1 core, 2GB RAM worker, start with CELERY_WORKER_AMOUNT=1. If you see the worker is underutilized (CPU/memory usage is low and no missed heartbeats), you can try increasing to 2, but with large batch jobs and limited resources, higher values may cause instability. In a similar scenario, CELERY_WORKER_AMOUNT=3 was used, but that was likely on a larger machine—monitor your resource usage and logs closely if you increase it source.

To manually resume or retry document indexing, use the Celery tasks from within your Python application context (not via API). For retrying, use:

from api.tasks.retry_document_indexing_task import retry_document_indexing_task
retry_document_indexing_task.delay(dataset_id, [document_id1, document_id2, ...])

For recovery:

from api.tasks.recover_document_indexing_task import recover_document_indexing_task
recover_document_indexing_task.delay(dataset_id, document_id)

Replace dataset_id and document_id with your actual IDs. These are standard Celery async tasks and can be triggered from a Python shell or script where your Celery app context is available—they are not direct API endpoints retry recover.

Monitor your worker logs and system resources after making changes to ensure stability.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Join Discord Share on X

@dosubot[bot] commented on GitHub (Jul 21, 2025): <!-- Answer --> For a 1 core, 2GB RAM worker, start with CELERY_WORKER_AMOUNT=1. If you see the worker is underutilized (CPU/memory usage is low and no missed heartbeats), you can try increasing to 2, but with large batch jobs and limited resources, higher values may cause instability. In a similar scenario, CELERY_WORKER_AMOUNT=3 was used, but that was likely on a larger machine—monitor your resource usage and logs closely if you increase it [source](https://github.com/langgenius/dify/issues/21437). To manually resume or retry document indexing, use the Celery tasks from within your Python application context (not via API). For retrying, use: ```python from api.tasks.retry_document_indexing_task import retry_document_indexing_task retry_document_indexing_task.delay(dataset_id, [document_id1, document_id2, ...]) ``` For recovery: ```python from api.tasks.recover_document_indexing_task import recover_document_indexing_task recover_document_indexing_task.delay(dataset_id, document_id) ``` Replace `dataset_id` and `document_id` with your actual IDs. These are standard Celery async tasks and can be triggered from a Python shell or script where your Celery app context is available—they are not direct API endpoints [retry](https://github.com/langgenius/dify/blob/29f0a9ab94a7a34a4e9aecef7e6d273e3f869a01/api/tasks/retry_document_indexing_task.py) [recover](https://github.com/langgenius/dify/blob/29f0a9ab94a7a34a4e9aecef7e6d273e3f869a01/api/tasks/recover_document_indexing_task.py). Monitor your worker logs and system resources after making changes to ensure stability. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/414ee16e-a17d-476e-9756-00a84ed5dbfc?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/414ee16e-a17d-476e-9756-00a84ed5dbfc?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/414ee16e-a17d-476e-9756-00a84ed5dbfc?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/414ee16e-a17d-476e-9756-00a84ed5dbfc?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/414ee16e-a17d-476e-9756-00a84ed5dbfc?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/414ee16e-a17d-476e-9756-00a84ed5dbfc?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/414ee16e-a17d-476e-9756-00a84ed5dbfc?feedback_type=other)</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/22735)
yindo commented 2026-02-21 19:22:13 -05:00
Author
Owner
Copy Link
Copy Source

@AllenHu2024 commented on GitHub (Jul 21, 2025):

but how to run the code in docker? I deploy workers to Azure K8s.
can it re-work after restart?
@ducu

@AllenHu2024 commented on GitHub (Jul 21, 2025): but how to run the code in docker? I deploy workers to Azure K8s. can it re-work after restart? @ducu
yindo commented 2026-02-21 19:22:13 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jul 21, 2025):

Dify does not automatically resume interrupted embedding/indexing jobs after a worker restart—you need to manually trigger recovery or retry tasks.

To do this in Docker/Kubernetes (including Azure AKS):

  1. Use kubectl exec (for K8s) or docker exec (for Docker) to get a shell inside your running Dify API or worker pod/container.
  2. Start a Python shell in the container (e.g., uv run python or just python).
  3. Import the Celery task and call its delay method: 
    from api.tasks.retry_document_indexing_task import retry_document_indexing_task
retry_document_indexing_task.delay(dataset_id, [document_id1, document_id2])
# or
from api.tasks.recover_document_indexing_task import recover_document_indexing_task
recover_document_indexing_task.delay(dataset_id, document_id)
    Replace dataset_id and document_id with your actual IDs.

New jobs will be picked up automatically after a worker restart, but any interrupted or errored jobs require this manual recovery step—there is no automatic checkpoint-based resume for in-progress jobs after a crash or restart. This is standard for Dify's current design and applies to containerized deployments like Azure AKS as well.
Reference: retry/recover task usage | Docker/K8s worker setup

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Join Discord Share on X

@dosubot[bot] commented on GitHub (Jul 21, 2025): <!-- Answer --> Dify does not automatically resume interrupted embedding/indexing jobs after a worker restart—you need to manually trigger recovery or retry tasks. To do this in Docker/Kubernetes (including Azure AKS): 1. Use kubectl exec (for K8s) or docker exec (for Docker) to get a shell inside your running Dify API or worker pod/container. 2. Start a Python shell in the container (e.g., uv run python or just python). 3. Import the Celery task and call its delay method: ```python from api.tasks.retry_document_indexing_task import retry_document_indexing_task retry_document_indexing_task.delay(dataset_id, [document_id1, document_id2]) # or from api.tasks.recover_document_indexing_task import recover_document_indexing_task recover_document_indexing_task.delay(dataset_id, document_id) ``` Replace dataset_id and document_id with your actual IDs. New jobs will be picked up automatically after a worker restart, but any interrupted or errored jobs require this manual recovery step—there is no automatic checkpoint-based resume for in-progress jobs after a crash or restart. This is standard for Dify's current design and applies to containerized deployments like Azure AKS as well. [Reference: retry/recover task usage](https://github.com/langgenius/dify/blob/29f0a9ab94a7a34a4e9aecef7e6d273e3f869a01/api/tasks/retry_document_indexing_task.py) | [Docker/K8s worker setup](https://github.com/langgenius/dify/blob/29f0a9ab94a7a34a4e9aecef7e6d273e3f869a01/api/docker/entrypoint.sh) <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/a47a284b-035e-414b-87ac-70fabcbf9b6d?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/a47a284b-035e-414b-87ac-70fabcbf9b6d?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/a47a284b-035e-414b-87ac-70fabcbf9b6d?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/a47a284b-035e-414b-87ac-70fabcbf9b6d?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/a47a284b-035e-414b-87ac-70fabcbf9b6d?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/a47a284b-035e-414b-87ac-70fabcbf9b6d?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/a47a284b-035e-414b-87ac-70fabcbf9b6d?feedback_type=other)</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/22735)
yindo commented 2026-02-21 19:22:13 -05:00
Author
Owner
Copy Link
Copy Source

@jubinsoni commented on GitHub (Jul 24, 2025):

hi @AllenHu2024

you can also try autoscaling
in api/docker/entrypoint.sh

add below lines

export CELERY_AUTO_SCALE="true"
echo ${CELERY_AUTO_SCALE}
Image
@jubinsoni commented on GitHub (Jul 24, 2025): hi @AllenHu2024 you can also try autoscaling in api/docker/entrypoint.sh add below lines ``` export CELERY_AUTO_SCALE="true" echo ${CELERY_AUTO_SCALE} ``` <img width="623" height="475" alt="Image" src="https://github.com/user-attachments/assets/4b49f48b-29be-44b4-812b-c59d8bbbc534" />
yindo commented 2026-02-21 19:22:13 -05:00
Author
Owner
Copy Link
Copy Source

@sabo-li commented on GitHub (Jul 29, 2025):

from api.tasks.retry_document_indexing_task import retry_document_indexing_task
retry_document_indexing_task.delay(dataset_id, [document_id1, document_id2])
# or
from api.tasks.recover_document_indexing_task import recover_document_indexing_task
recover_document_indexing_task.delay(dataset_id, document_id)

The same problem occurred here, restarting the worker did not continue processing the document.
I see that the provided method requires the document_id to be provided. How to obtain all pending document_id.
@dosu

@sabo-li commented on GitHub (Jul 29, 2025): ```python from api.tasks.retry_document_indexing_task import retry_document_indexing_task retry_document_indexing_task.delay(dataset_id, [document_id1, document_id2]) # or from api.tasks.recover_document_indexing_task import recover_document_indexing_task recover_document_indexing_task.delay(dataset_id, document_id) ``` The same problem occurred here, restarting the worker did not continue processing the document. I see that the provided method requires the `document_id` to be provided. How to obtain all pending `document_id`. [@dosu](https://go.dosu.dev/dosubot)
yindo commented 2026-02-21 19:22:13 -05:00
Author
Owner
Copy Link
Copy Source

@sabo-li commented on GitHub (Jul 30, 2025):

I recover using the following method,but it's unfriendly

Use docker compose run:

docker compose exec -it worker python

Then enter:

from time import sleep
from app_factory import create_migrations_app
from extensions.ext_database import db
from models.dataset import Dataset, Document, DocumentSegment
from tasks.retry_document_indexing_task import retry_document_indexing_task
from tasks.recover_document_indexing_task import recover_document_indexing_task

app = create_migrations_app()

dataset_id = 'not completed dataset id'

def recover():
    with app.app_context():
        document_ids = [item.id for item in db.session.query(Document).where(Document.dataset_id == dataset_id, Document.indexing_status != "completed")]
        db.session.close()
        for document_id in document_ids:
            sleep(2)
            recover_document_indexing_task.delay(dataset_id, document_id)
            print(document_id)

def retry():
    with app.app_context():
        retry_document_indexing_task.delay(dataset_id, [item.id for item in db.session.query(Document).where(Document.dataset_id == dataset_id, Document.indexing_status != "completed")])
        db.session.close()

def not_completed_count():
    with app.app_context():
        print(db.session.query(Document).where(Document.dataset_id == dataset_id, Document.indexing_status != "completed").count())
        db.session.close()

def completed_count():
    with app.app_context():
        print(db.session.query(Document).where(Document.dataset_id == dataset_id, Document.indexing_status == "completed").count())
        db.session.close()

Execute recover() will restore all not completed documents, other definen method look method name.

@sabo-li commented on GitHub (Jul 30, 2025): I recover using the following method,but it's unfriendly Use docker compose run: ```bash docker compose exec -it worker python ``` Then enter: ```python from time import sleep from app_factory import create_migrations_app from extensions.ext_database import db from models.dataset import Dataset, Document, DocumentSegment from tasks.retry_document_indexing_task import retry_document_indexing_task from tasks.recover_document_indexing_task import recover_document_indexing_task app = create_migrations_app() dataset_id = 'not completed dataset id' def recover(): with app.app_context(): document_ids = [item.id for item in db.session.query(Document).where(Document.dataset_id == dataset_id, Document.indexing_status != "completed")] db.session.close() for document_id in document_ids: sleep(2) recover_document_indexing_task.delay(dataset_id, document_id) print(document_id) def retry(): with app.app_context(): retry_document_indexing_task.delay(dataset_id, [item.id for item in db.session.query(Document).where(Document.dataset_id == dataset_id, Document.indexing_status != "completed")]) db.session.close() def not_completed_count(): with app.app_context(): print(db.session.query(Document).where(Document.dataset_id == dataset_id, Document.indexing_status != "completed").count()) db.session.close() def completed_count(): with app.app_context(): print(db.session.query(Document).where(Document.dataset_id == dataset_id, Document.indexing_status == "completed").count()) db.session.close() ``` Execute `recover()` will restore all not completed documents, other definen method look method name.
yindo commented 2026-02-21 19:22:13 -05:00
Author
Owner
Copy Link
Copy Source

@shiyang1026 commented on GitHub (Jul 30, 2025):

I’ve also encountered this issue. Our dataset contains over 100,000 documents, and when building the index, I noticed that the logs in the worker service often stop automatically.

I’ve tried increasing the number of Celery workers, but it didn’t help much. For now, the only workaround is to periodically restart the worker service. (The service is deployed using Docker Compose.)

@shiyang1026 commented on GitHub (Jul 30, 2025): I’ve also encountered this issue. Our dataset contains over 100,000 documents, and when building the index, I noticed that the logs in the worker service often stop automatically. I’ve tried increasing the number of Celery workers, but it didn’t help much. For now, the only workaround is to periodically restart the worker service. (The service is deployed using Docker Compose.)
yindo commented 2026-02-21 19:22:13 -05:00
Author
Owner
Copy Link
Copy Source

@AllenHu2024 commented on GitHub (Jul 30, 2025):

It is recommended that the worker service incorporate a transaction processing mechanism, such that when a file or chunk is not processed successfully, it is retried via a dead-letter queue (or similar).

@AllenHu2024 commented on GitHub (Jul 30, 2025): It is recommended that the worker service incorporate a transaction processing mechanism, such that when a file or chunk is not processed successfully, it is retried via a dead-letter queue (or similar).
yindo commented 2026-02-21 19:22:14 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 1, 2025):

Hi, @AllenHu2024. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You reported worker process stops with missed heartbeat errors in Dify v1.4.3 due to worker overload after uploading many files.
  • The root cause was identified as Celery workers being too busy, with a recommendation to increase CELERY_WORKER_AMOUNT.
  • Community members shared example scripts to manually trigger retry or recovery Celery tasks in Docker/Kubernetes environments.
  • Transaction processing with retries for failed chunks was suggested to improve stability.
  • Periodic worker restarts were used as a temporary workaround.

Next Steps:

  • Please confirm if this issue is still relevant with the latest version of Dify; if so, you can keep the discussion open by commenting here.
  • Otherwise, I will automatically close this issue in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Sep 1, 2025): Hi, @AllenHu2024. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You reported worker process stops with missed heartbeat errors in Dify v1.4.3 due to worker overload after uploading many files. - The root cause was identified as Celery workers being too busy, with a recommendation to increase CELERY_WORKER_AMOUNT. - Community members shared example scripts to manually trigger retry or recovery Celery tasks in Docker/Kubernetes environments. - Transaction processing with retries for failed chunks was suggested to improve stability. - Periodic worker restarts were used as a temporary workaround. **Next Steps:** - Please confirm if this issue is still relevant with the latest version of Dify; if so, you can keep the discussion open by commenting here. - Otherwise, I will automatically close this issue in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#15544
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?