langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-23 13:05:55 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

The celery beat container is missing in docker-compose.yaml, causing scheduled tasks to fail to be executed. #927

New Issue
Closed
opened 2026-02-21 17:29:04 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 17:29:04 -05:00
Owner
Copy Link
Copy Source

Originally created by @lujin1 on GitHub (Jan 18, 2024).

Self Checks

Dify version

0.4.8

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

did not execute the tasks in beat_schedule

  • clean_embedding_cache_tas
  • clean_unused_datasets_task

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @lujin1 on GitHub (Jan 18, 2024). ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to file this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). ### Dify version 0.4.8 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce did not execute the tasks in beat_schedule - clean_embedding_cache_tas - clean_unused_datasets_task ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug label 2026-02-21 17:29:04 -05:00
yindo closed this issue 2026-02-21 17:29:04 -05:00
yindo commented 2026-02-21 17:29:04 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 18, 2024):

🤖

Hello @lujin1! I'm Dosu, a friendly bot here to assist you with your issue. I'm equipped to help you troubleshoot bugs, answer questions about dify, and even guide you on how to become a contributor. While we await a human maintainer, I'm here to do my best to assist you. Let's get started, shall we?

Thank you for bringing this to our attention. The issue you're experiencing is due to the absence of a Celery Beat service in the docker-compose.yaml file. Celery Beat is a scheduler; it kicks off tasks at regular intervals, which are then executed by available worker nodes in the cluster. The tasks 'clean_embedding_cache_task' and 'clean_unused_datasets_task' are likely scheduled tasks that require Celery Beat to run.

To resolve this issue, you need to add a Celery Beat service to your docker-compose.yaml file. Here's an example of how you can do this:

  beat:
    image: langgenius/dify-api:0.4.8
    restart: always
    environment:
      # Startup mode, 'beat' starts the Celery beat for scheduling tasks.
      MODE: beat
      # --- All the configurations below are the same as those in the 'api' service. ---
      LOG_LEVEL: INFO
      SECRET_KEY: sk-9f73s3ljTXVcMT3Blb3ljTqtsKiGHXVcMT3BlbkFJLK7U
      DB_USERNAME: postgres
      DB_PASSWORD: difyai123456
      DB_HOST: db
      DB_PORT: 5432
      DB_DATABASE: dify
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_USERNAME: ''
      REDIS_PASSWORD: difyai123456
      REDIS_DB: 0
      REDIS_USE_SSL: 'false'
      CELERY_BROKER_URL: redis://:difyai123456@redis:6379/1
      STORAGE_TYPE: local
      STORAGE_LOCAL_PATH: storage
      VECTOR_STORE: weaviate
      WEAVIATE_ENDPOINT: http://weaviate:8080
      WEAVIATE_API_KEY: WVF5YThaHlkYwhGUSmCRgsX3tD5ngdN8pkih
      QDRANT_URL: http://qdrant:6333
      QDRANT_API_KEY: difyai123456
      QDRANT_CLIENT_TIMEOUT: 20
      MILVUS_HOST: 127.0.0.1
      MILVUS_PORT: 19530
      MILVUS_USER: root
      MILVUS_PASSWORD: Milvus
      MILVUS_SECURE: 'false'
      MAIL_TYPE: ''
      MAIL_DEFAULT_SEND_FROM: 'YOUR EMAIL FROM (eg: no-reply <no-reply@dify.ai>)'
      RESEND_API_KEY: ''
      RESEND_API_URL: https://api.resend.com
    depends_on:
      - db
      - redis
    volumes:
      # Mount the storage directory to the container, for storing user files.
      - ./volumes/app/storage:/app/api/storage

This service will start a Celery Beat instance that will schedule the tasks. Please replace the environment variables with your actual values.

As for the tasks 'clean_embedding_cache_task' and 'clean_unused_datasets_task', they are scheduled tasks designed to maintain the efficiency of the system. The 'clean_unused_datasets_task' removes datasets that have not been used for a specified number of days, as determined by the 'CLEAN_DAY_SETTING' configuration value. This task helps maintain the efficiency of the system by cleaning up resources that are no longer active or needed. You can find more details about these tasks in the dify codebase here and here.

I hope this helps! If you have any other questions or run into any issues, please let me know.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

@dosubot[bot] commented on GitHub (Jan 18, 2024): _🤖_ <!-- Greeting --> Hello @lujin1! I'm [Dosu](https://dosu.dev), a friendly bot here to assist you with your issue. I'm equipped to help you troubleshoot bugs, answer questions about dify, and even guide you on how to become a contributor. While we await a human maintainer, I'm here to do my best to assist you. Let's get started, shall we? <!-- Answer --> Thank you for bringing this to our attention. The issue you're experiencing is due to the absence of a Celery Beat service in the docker-compose.yaml file. Celery Beat is a scheduler; it kicks off tasks at regular intervals, which are then executed by available worker nodes in the cluster. The tasks 'clean_embedding_cache_task' and 'clean_unused_datasets_task' are likely scheduled tasks that require Celery Beat to run. To resolve this issue, you need to add a Celery Beat service to your docker-compose.yaml file. Here's an example of how you can do this: ```yaml beat: image: langgenius/dify-api:0.4.8 restart: always environment: # Startup mode, 'beat' starts the Celery beat for scheduling tasks. MODE: beat # --- All the configurations below are the same as those in the 'api' service. --- LOG_LEVEL: INFO SECRET_KEY: sk-9f73s3ljTXVcMT3Blb3ljTqtsKiGHXVcMT3BlbkFJLK7U DB_USERNAME: postgres DB_PASSWORD: difyai123456 DB_HOST: db DB_PORT: 5432 DB_DATABASE: dify REDIS_HOST: redis REDIS_PORT: 6379 REDIS_USERNAME: '' REDIS_PASSWORD: difyai123456 REDIS_DB: 0 REDIS_USE_SSL: 'false' CELERY_BROKER_URL: redis://:difyai123456@redis:6379/1 STORAGE_TYPE: local STORAGE_LOCAL_PATH: storage VECTOR_STORE: weaviate WEAVIATE_ENDPOINT: http://weaviate:8080 WEAVIATE_API_KEY: WVF5YThaHlkYwhGUSmCRgsX3tD5ngdN8pkih QDRANT_URL: http://qdrant:6333 QDRANT_API_KEY: difyai123456 QDRANT_CLIENT_TIMEOUT: 20 MILVUS_HOST: 127.0.0.1 MILVUS_PORT: 19530 MILVUS_USER: root MILVUS_PASSWORD: Milvus MILVUS_SECURE: 'false' MAIL_TYPE: '' MAIL_DEFAULT_SEND_FROM: 'YOUR EMAIL FROM (eg: no-reply <no-reply@dify.ai>)' RESEND_API_KEY: '' RESEND_API_URL: https://api.resend.com depends_on: - db - redis volumes: # Mount the storage directory to the container, for storing user files. - ./volumes/app/storage:/app/api/storage ``` This service will start a Celery Beat instance that will schedule the tasks. Please replace the environment variables with your actual values. As for the tasks 'clean_embedding_cache_task' and 'clean_unused_datasets_task', they are scheduled tasks designed to maintain the efficiency of the system. The 'clean_unused_datasets_task' removes datasets that have not been used for a specified number of days, as determined by the 'CLEAN_DAY_SETTING' configuration value. This task helps maintain the efficiency of the system by cleaning up resources that are no longer active or needed. You can find more details about these tasks in the dify codebase [here](https://github.com/langgenius/dify/blob/main/api/schedule/clean_embedding_cache_task.py) and [here](https://github.com/langgenius/dify/blob/main/api/schedule/clean_unused_datasets_task.py). I hope this helps! If you have any other questions or run into any issues, please let me know. ## Sources - [celery_app 没有执行 beat_schedule 中的任务，clean_embedding_cache_task 和 clean_unused_datasets_task](https://github.com/langgenius/dify/issues/2077) - [docker/docker-compose.yaml](https://github.com/langgenius/dify/blob/main/docker/docker-compose.yaml) - [api/schedule/clean_embedding_cache_task.py](https://github.com/langgenius/dify/blob/main/api/schedule/clean_embedding_cache_task.py) - [api/schedule/clean_unused_datasets_task.py](https://github.com/langgenius/dify/blob/main/api/schedule/clean_unused_datasets_task.py) --- This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.
yindo commented 2026-02-21 17:29:04 -05:00
Author
Owner
Copy Link
Copy Source

@takatost commented on GitHub (Jan 18, 2024):

Currently, these two scheduled tasks are specifically serving Dify SaaS to clear out expired data.
Just curious, do you have any requirements in this field?

@takatost commented on GitHub (Jan 18, 2024): Currently, these two scheduled tasks are specifically serving Dify SaaS to clear out expired data. Just curious, do you have any requirements in this field?
yindo commented 2026-02-21 17:29:04 -05:00
Author
Owner
Copy Link
Copy Source

@lujin1 commented on GitHub (Jan 18, 2024):

What is expired data?
I just want to clear the embedings table data in the postgres database.
The data is already stored in weaviate vector database.

@lujin1 commented on GitHub (Jan 18, 2024): What is expired data? I just want to clear the embedings table data in the postgres database. The data is already stored in weaviate vector database.
yindo commented 2026-02-21 17:29:04 -05:00
Author
Owner
Copy Link
Copy Source

@takatost commented on GitHub (Jan 18, 2024):

The next version v0.4.9 will deprecate the embeddings table, using redis cache for 1 hour as an alternative. Just FYI, you'll be able to directly truncate this table then.

@takatost commented on GitHub (Jan 18, 2024): The next version v0.4.9 will deprecate the `embeddings` table, using redis cache for 1 hour as an alternative. Just FYI, you'll be able to directly truncate this table then.
yindo commented 2026-02-21 17:29:04 -05:00
Author
Owner
Copy Link
Copy Source

@takatost commented on GitHub (Jan 20, 2024):

v0.4.9 is out. No more using the embeddings table.

@takatost commented on GitHub (Jan 20, 2024): v0.4.9 is out. No more using the embeddings table.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#927
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?