langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Possible database connection leak found #217

New Issue
Closed
opened 2026-02-21 17:26:25 -05:00 by yindo · 12 comments
yindo commented 2026-02-21 17:26:25 -05:00
Owner
Copy Link
Copy Source

Originally created by @DataScientistSamChan on GitHub (Jun 27, 2023).

Originally assigned to: @takatost, @crazywoola on GitHub.

We are running a hosted version of dify in production, and we almost always end up getting the same error below and the service will go offline.

sqlalchemy.exc.TimeoutError: QueuePool limit of size 30 overflow 30 reached, connection timed out, timeout 10.00 (Background on this error at: https://sqlalche.me/e/14/3o7r)

Dify version: Self Host

Steps To Reproduce

  1. Just set SQLALCHEMY_POOL_SIZE to 1.
  2. Keep browsing and play around until the error comes up

The current behavior

QueuePool limit will always end up exceeding 30 and make the service unavailble.

The expected behavior

QueuePool size of 30 should work just fine.

Originally created by @DataScientistSamChan on GitHub (Jun 27, 2023). Originally assigned to: @takatost, @crazywoola on GitHub. We are running a hosted version of dify in production, and we almost always end up getting the same error below and the service will go offline. ``` sqlalchemy.exc.TimeoutError: QueuePool limit of size 30 overflow 30 reached, connection timed out, timeout 10.00 (Background on this error at: https://sqlalche.me/e/14/3o7r) ``` Dify version: Self Host ## Steps To Reproduce 1. Just set `SQLALCHEMY_POOL_SIZE` to 1. 2. Keep browsing and play around until the error comes up ## The current behavior QueuePool limit will always end up exceeding 30 and make the service unavailble. ## The expected behavior QueuePool size of 30 should work just fine.
yindo added the 🐞 bug🙋‍♂️ question labels 2026-02-21 17:26:25 -05:00
yindo closed this issue 2026-02-21 17:26:25 -05:00
yindo commented 2026-02-21 17:26:25 -05:00
Author
Owner
Copy Link
Copy Source

@takatost commented on GitHub (Jun 28, 2023):

You can adjust the environment variable SQLALCHEMY_POOL_SIZE for both the api and worker services to fit the connection pool configuration of pgsql.
This is the configuration for working with pgsql

@takatost commented on GitHub (Jun 28, 2023): You can adjust the environment variable SQLALCHEMY_POOL_SIZE for both the api and worker services to fit the connection pool configuration of pgsql. This is the configuration for working with pgsql
yindo commented 2026-02-21 17:26:25 -05:00
Author
Owner
Copy Link
Copy Source

@DataScientistSamChan commented on GitHub (Jun 28, 2023):

Weird，since we do not have many active users and the error is only raised when simultaneous connection exceeds 30. Database connection leak is more likely the reason for this error，but I am not sure. I will dig into sqlalchemy logs to identify if this is a connection leak.

@DataScientistSamChan commented on GitHub (Jun 28, 2023): Weird，since we do not have many active users and the error is only raised when simultaneous connection exceeds 30. Database connection leak is more likely the reason for this error，but I am not sure. I will dig into sqlalchemy logs to identify if this is a connection leak.
yindo commented 2026-02-21 17:26:25 -05:00
Author
Owner
Copy Link
Copy Source

@takatost commented on GitHub (Jun 28, 2023):

SQLALCHEMY_POOL_SIZE is the size of the connection pool, which will be initialized with this many connections when the service is initialized, and will maintain this many connections afterwards.
What you said might be SQLALCHEMY_MAX_OVERFLOW, the size of the overflow connections in the pool i think.å

@takatost commented on GitHub (Jun 28, 2023): SQLALCHEMY_POOL_SIZE is the size of the connection pool, which will be initialized with this many connections when the service is initialized, and will maintain this many connections afterwards. What you said might be SQLALCHEMY_MAX_OVERFLOW, the size of the overflow connections in the pool i think.å
yindo commented 2026-02-21 17:26:25 -05:00
Author
Owner
Copy Link
Copy Source

@github-actions[bot] commented on GitHub (Jul 28, 2023):

Close due to it's no longer active, if you have any questions, you can reopen it.

@github-actions[bot] commented on GitHub (Jul 28, 2023): Close due to it's no longer active, if you have any questions, you can reopen it.
yindo commented 2026-02-21 17:26:25 -05:00
Author
Owner
Copy Link
Copy Source

@iamsk commented on GitHub (Oct 21, 2023):

TimeoutError
sqlalchemy.orm.session in merge
QueuePool limit of size 30 overflow 10 reached, connection timed out, timeout 30.00

I face the same issue and timeout is in close_pubsub:
image

I find this code:
image

Does it mean If we have 30+ users in 10 minutes, we will exceed the SQLALCHEMY_POOL_SIZE?
As it's already closed, @takatost can you share some information about this?

@iamsk commented on GitHub (Oct 21, 2023): TimeoutError sqlalchemy.orm.session in merge QueuePool limit of size 30 overflow 10 reached, connection timed out, timeout 30.00 I face the same issue and timeout is in close_pubsub: <img width="471" alt="image" src="https://github.com/langgenius/dify/assets/170050/b7767b1a-51a6-499d-9ed5-819320c5c2bf"> I find this code: <img width="515" alt="image" src="https://github.com/langgenius/dify/assets/170050/288191c3-7f0d-4c5e-baaf-e9bc29022350"> Does it mean If we have 30+ users in 10 minutes, we will exceed the SQLALCHEMY_POOL_SIZE? As it's already closed, @takatost can you share some information about this?
yindo commented 2026-02-21 17:26:25 -05:00
Author
Owner
Copy Link
Copy Source

@iamsk commented on GitHub (Oct 22, 2023):

image
SELECT end_users.id AS end_users_id, end_users.tenant_id AS end_users_tenant_id, end_users.app_id AS end_users_app_id, end_users.type AS end_users_type, end_users.external_user_id AS end_users_external_user_id, end_users.name AS end_users_name, end_users.is_anonymous AS end_users_is_anonymous, end_users.session_id AS end_users_session_id, end_users.created_at AS end_users_created_at, end_users.updated_at AS end_users_updated_at
FROM end_users
WHERE end_users.id = '59e7af1c-f22f-43e1-92cf-f6d510cacde5'::uuid
LIMIT 1

sqlalchemy.exc.TimeoutError: QueuePool limit of size 30 overflow 30 reached 问题发生在系统运行一段时间后，所以是不是这个持续增长引起的？

@iamsk commented on GitHub (Oct 22, 2023): ![image](https://github.com/langgenius/dify/assets/170050/94ab5cd0-926d-40db-a19d-0328674de4d4) SELECT end_users.id AS end_users_id, end_users.tenant_id AS end_users_tenant_id, end_users.app_id AS end_users_app_id, end_users.type AS end_users_type, end_users.external_user_id AS end_users_external_user_id, end_users.name AS end_users_name, end_users.is_anonymous AS end_users_is_anonymous, end_users.session_id AS end_users_session_id, end_users.created_at AS end_users_created_at, end_users.updated_at AS end_users_updated_at FROM end_users WHERE end_users.id = '59e7af1c-f22f-43e1-92cf-f6d510cacde5'::uuid LIMIT 1 --- sqlalchemy.exc.TimeoutError: QueuePool limit of size 30 overflow 30 reached 问题发生在系统运行一段时间后，所以是不是这个持续增长引起的？
yindo commented 2026-02-21 17:26:25 -05:00
Author
Owner
Copy Link
Copy Source

@iamsk commented on GitHub (Oct 22, 2023):

My dify version is tag:0.3.28, 数据库用的 aws rds postgresql

@iamsk commented on GitHub (Oct 22, 2023): My dify version is tag:0.3.28, 数据库用的 aws rds postgresql
yindo commented 2026-02-21 17:26:25 -05:00
Author
Owner
Copy Link
Copy Source

@tandangfei commented on GitHub (Nov 12, 2023):

Has the problem been solved？

@tandangfei commented on GitHub (Nov 12, 2023): Has the problem been solved？
yindo commented 2026-02-21 17:26:25 -05:00
Author
Owner
Copy Link
Copy Source

@iamsk commented on GitHub (Nov 12, 2023):

It's ok now, but I can't reproduce that.

@iamsk commented on GitHub (Nov 12, 2023): It's ok now, but I can't reproduce that.
yindo commented 2026-02-21 17:26:25 -05:00
Author
Owner
Copy Link
Copy Source

@zhangever commented on GitHub (Jun 12, 2024):

The same error still occurs with dify version 0.6.10.
Once this error occurs, nothing can be done any more and can't be recovered.
Could anyone help on this?

@zhangever commented on GitHub (Jun 12, 2024): The same error still occurs with dify version 0.6.10. Once this error occurs, nothing can be done any more and can't be recovered. Could anyone help on this?
yindo commented 2026-02-21 17:26:26 -05:00
Author
Owner
Copy Link
Copy Source

@13241308289 commented on GitHub (Mar 13, 2025):

QueuePool limit of size 10 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)

The same error still occurs with dify version 0.11.1.1

@13241308289 commented on GitHub (Mar 13, 2025): QueuePool limit of size 10 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r) The same error still occurs with dify version 0.11.1.1
yindo commented 2026-02-21 17:26:26 -05:00
Author
Owner
Copy Link
Copy Source

@yangguangpin commented on GitHub (Nov 20, 2025):

dify version 0.14.2 has same problem.

@yangguangpin commented on GitHub (Nov 20, 2025): dify version 0.14.2 has same problem.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug 🙋‍♂️ question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#217
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?