langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 03:55:25 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

worker disconnect broker redis when index large file #14083

New Issue
Closed
opened 2026-02-21 19:15:28 -05:00 by yindo · 2 comments
yindo commented 2026-02-21 19:15:28 -05:00
Owner
Copy Link
Copy Source

Originally created by @huangzhuo1949 on GitHub (May 21, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.15.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

2025-05-21 14:56:11,457.457 WARNING [MainThread] [consumer.py:378] - consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/app/api/.venv/lib/python3.12/site-packages/celery/worker/consumer/consumer.py", line 340, in start
    blueprint.start(self)
  File "/app/api/.venv/lib/python3.12/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/app/api/.venv/lib/python3.12/site-packages/celery/worker/consumer/consumer.py", line 746, in start
    c.loop(*c.loop_args())
  File "/app/api/.venv/lib/python3.12/site-packages/celery/worker/loops.py", line 130, in synloop
    connection.drain_events(timeout=2.0)
  File "/app/api/.venv/lib/python3.12/site-packages/kombu/connection.py", line 341, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 997, in drain_events
    get(self._deliver, timeout=timeout)
  File "/app/api/.venv/lib/python3.12/site-packages/kombu/transport/redis.py", line 584, in get
    self._register_BRPOP(channel)
  File "/app/api/.venv/lib/python3.12/site-packages/kombu/transport/redis.py", line 525, in _register_BRPOP
    channel._brpop_start()
  File "/app/api/.venv/lib/python3.12/site-packages/kombu/transport/redis.py", line 962, in _brpop_start
    self.client.connection.send_command(*command_args)
  File "/app/api/.venv/lib/python3.12/site-packages/redis/connection.py", line 476, in send_command
    self.send_packed_command(
  File "/app/api/.venv/lib/python3.12/site-packages/redis/connection.py", line 449, in send_packed_command
    self.check_health()
  File "/app/api/.venv/lib/python3.12/site-packages/redis/connection.py", line 441, in check_health
    self.retry.call_with_retry(self._send_ping, self._ping_failed)
  File "/app/api/.venv/lib/python3.12/site-packages/redis/retry.py", line 67, in call_with_retry
    raise error
  File "/app/api/.venv/lib/python3.12/site-packages/redis/retry.py", line 62, in call_with_retry
    return do()
           ^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/redis/connection.py", line 431, in _send_ping
    if str_if_bytes(self.read_response()) != "PONG":
                    ^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/redis/connection.py", line 512, in read_response
    response = self._parser.read_response(disable_decoding=disable_decoding)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/redis/_parsers/hiredis.py", line 130, in read_response
    self.read_from_socket()
  File "/app/api/.venv/lib/python3.12/site-packages/redis/_parsers/hiredis.py", line 92, in read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @huangzhuo1949 on GitHub (May 21, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.15.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce ``` 2025-05-21 14:56:11,457.457 WARNING [MainThread] [consumer.py:378] - consumer: Connection to broker lost. Trying to re-establish the connection... Traceback (most recent call last): File "/app/api/.venv/lib/python3.12/site-packages/celery/worker/consumer/consumer.py", line 340, in start blueprint.start(self) File "/app/api/.venv/lib/python3.12/site-packages/celery/bootsteps.py", line 116, in start step.start(parent) File "/app/api/.venv/lib/python3.12/site-packages/celery/worker/consumer/consumer.py", line 746, in start c.loop(*c.loop_args()) File "/app/api/.venv/lib/python3.12/site-packages/celery/worker/loops.py", line 130, in synloop connection.drain_events(timeout=2.0) File "/app/api/.venv/lib/python3.12/site-packages/kombu/connection.py", line 341, in drain_events return self.transport.drain_events(self.connection, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 997, in drain_events get(self._deliver, timeout=timeout) File "/app/api/.venv/lib/python3.12/site-packages/kombu/transport/redis.py", line 584, in get self._register_BRPOP(channel) File "/app/api/.venv/lib/python3.12/site-packages/kombu/transport/redis.py", line 525, in _register_BRPOP channel._brpop_start() File "/app/api/.venv/lib/python3.12/site-packages/kombu/transport/redis.py", line 962, in _brpop_start self.client.connection.send_command(*command_args) File "/app/api/.venv/lib/python3.12/site-packages/redis/connection.py", line 476, in send_command self.send_packed_command( File "/app/api/.venv/lib/python3.12/site-packages/redis/connection.py", line 449, in send_packed_command self.check_health() File "/app/api/.venv/lib/python3.12/site-packages/redis/connection.py", line 441, in check_health self.retry.call_with_retry(self._send_ping, self._ping_failed) File "/app/api/.venv/lib/python3.12/site-packages/redis/retry.py", line 67, in call_with_retry raise error File "/app/api/.venv/lib/python3.12/site-packages/redis/retry.py", line 62, in call_with_retry return do() ^^^^ File "/app/api/.venv/lib/python3.12/site-packages/redis/connection.py", line 431, in _send_ping if str_if_bytes(self.read_response()) != "PONG": ^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/redis/connection.py", line 512, in read_response response = self._parser.read_response(disable_decoding=disable_decoding) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/redis/_parsers/hiredis.py", line 130, in read_response self.read_from_socket() File "/app/api/.venv/lib/python3.12/site-packages/redis/_parsers/hiredis.py", line 92, in read_from_socket raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR) redis.exceptions.ConnectionError: Connection closed by server. ``` ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the 🐞 bugoutdated labels 2026-02-21 19:15:28 -05:00
yindo closed this issue 2026-02-21 19:15:28 -05:00
yindo commented 2026-02-21 19:15:29 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (May 21, 2025):

A similar issue was resolved by addressing the Celery worker's concurrency settings. The problem was that the worker node, started with the celery worker command and parameters like -c 5, was too busy to send heartbeat information to Redis when it received the maximum number of tasks it could handle concurrently. The solution was to increase the CELERY_WORKER_AMOUNT environment variable to allow the worker to handle more tasks [1].

Additionally, Redis is used as a broker for Celery workers in Dify, facilitating communication and task queue management. Ensuring that Redis is properly configured and that the Celery workers are not overwhelmed with tasks can help maintain stable connections [2].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (May 21, 2025): <!-- Answer --> A similar issue was resolved by addressing the Celery worker's concurrency settings. The problem was that the worker node, started with the `celery worker` command and parameters like `-c 5`, was too busy to send heartbeat information to Redis when it received the maximum number of tasks it could handle concurrently. The solution was to increase the `CELERY_WORKER_AMOUNT` environment variable to allow the worker to handle more tasks <sup>[[1]](https://github.com/langgenius/dify/issues/12325)</sup>. Additionally, Redis is used as a broker for Celery workers in Dify, facilitating communication and task queue management. Ensuring that Redis is properly configured and that the Celery workers are not overwhelmed with tasks can help maintain stable connections <sup>[[2]](https://github.com/crazywoola/dify-docs-mintlify/blob/main/zh-hans/getting-started/install-self-hosted/environments.mdx)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/2012c7bb-d1ba-4cc5-ae2c-499ec2454bba?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/2012c7bb-d1ba-4cc5-ae2c-499ec2454bba?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/2012c7bb-d1ba-4cc5-ae2c-499ec2454bba?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/2012c7bb-d1ba-4cc5-ae2c-499ec2454bba?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/2012c7bb-d1ba-4cc5-ae2c-499ec2454bba?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/2012c7bb-d1ba-4cc5-ae2c-499ec2454bba?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/2012c7bb-d1ba-4cc5-ae2c-499ec2454bba?feedback_type=other)</sup>
yindo commented 2026-02-21 19:15:29 -05:00
Author
Owner
Copy Link
Copy Source

@Zhujing1215 commented on GitHub (Jun 11, 2025):

CELERY_WORKER_AMOUNT is the default value. If it is empty, the default value is 1. I guess CELERY_AUTO_SCALE=false is not set to true. If you set it to true, then set CELERY_MAX_WORKERS: maximum number of workers
CELERY_MIN_WORKERS: minimum number of workers simultaneously. You can enable automatic expansion. This can solve the problem you mentioned.

@Zhujing1215 commented on GitHub (Jun 11, 2025): CELERY_WORKER_AMOUNT is the default value. If it is empty, the default value is 1. I guess CELERY_AUTO_SCALE=false is not set to true. If you set it to true, then set CELERY_MAX_WORKERS: maximum number of workers CELERY_MIN_WORKERS: minimum number of workers simultaneously. You can enable automatic expansion. This can solve the problem you mentioned.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label outdated 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#14083
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?