langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

psycopg2.errors.DeadlockDetected: deadlock detected #7583

New Issue
Closed
opened 2026-02-21 18:21:20 -05:00 by yindo · 2 comments
yindo commented 2026-02-21 18:21:20 -05:00
Owner
Copy Link
Copy Source

Originally created by @huangzhuo1949 on GitHub (Jan 8, 2025).

Originally assigned to: @laipz8200, @JohnJyong on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.14.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

2025-01-03 11:17:04,324.324 ERROR [ThreadPoolExecutor-77_5] [cached_embedding.py:137] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2118, in _exec_insertmany_context
    dialect.do_execute(
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.DeadlockDetected: deadlock detected
DETAIL:  Process 25534 waits for ShareLock on transaction 160056165; blocked by process 25538.
Process 25538 waits for ShareLock on transaction 160056149; blocked by process 25534.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (81420,15) in relation "embedding_hash_idx"


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 132, in embed_documents
    db.session.commit()
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 597, in commit
    return self._proxied.commit()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2028, in commit
    trans.commit(_to_root=True)
  File "<string>", line 2, in commit
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
                ^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1313, in commit
    self._prepare_impl()
  File "<string>", line 2, in _prepare_impl
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
                ^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1288, in _prepare_impl
    self.session.flush()
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4352, in flush
    self._flush(objects)
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4487, in _flush
    with util.safe_reraise():
         ^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4448, in _flush
    flush_context.execute()
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
    rec.execute(self)
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
    _emit_insert_statements(
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 1143, in _emit_insert_statements
    result = connection.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
    return meth(
           ^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context
    return self._exec_insertmany_context(dialect, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2126, in _exec_insertmany_context
    self._handle_dbapi_exception(
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2118, in _exec_insertmany_context
    dialect.do_execute(
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.errors.DeadlockDetected) deadlock detected
DETAIL:  Process 25534 waits for ShareLock on transaction 160056165; blocked by process 25538.
Process 25538 waits for ShareLock on transaction 160056149; blocked by process 25534.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (81420,15) in relation "embedding_hash_idx"

[SQL: INSERT INTO embeddings (model_name, hash, embedding, provider_name) VALUES (%(model_name)s, %(hash)s, %(embedding)s, %(provider_name)s) RETURNING embeddings.id, embeddings.created_at]
[parameters: {'model_name': 'text-embedding-3-large', 'hash': 'cb4db2492344433d5eeda9c4c974e9cdfa8ef1439e150aadf77c46d89fc3a94d', 'embedding': <psycopg2.extensions.Binary object at 0x7f9422f536f0>, 'provider_name': 'azure_openai'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @huangzhuo1949 on GitHub (Jan 8, 2025). Originally assigned to: @laipz8200, @JohnJyong on GitHub. ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.14.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce ``` 2025-01-03 11:17:04,324.324 ERROR [ThreadPoolExecutor-77_5] [cached_embedding.py:137] - Failed to embed documents: %s Traceback (most recent call last): File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2118, in _exec_insertmany_context dialect.do_execute( File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute cursor.execute(statement, parameters) psycopg2.errors.DeadlockDetected: deadlock detected DETAIL: Process 25534 waits for ShareLock on transaction 160056165; blocked by process 25538. Process 25538 waits for ShareLock on transaction 160056149; blocked by process 25534. HINT: See server log for query details. CONTEXT: while inserting index tuple (81420,15) in relation "embedding_hash_idx" The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/app/api/core/rag/embedding/cached_embedding.py", line 132, in embed_documents db.session.commit() File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 597, in commit return self._proxied.commit() ^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2028, in commit trans.commit(_to_root=True) File "<string>", line 2, in commit File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go ret_value = fn(self, *arg, **kw) ^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1313, in commit self._prepare_impl() File "<string>", line 2, in _prepare_impl File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go ret_value = fn(self, *arg, **kw) ^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1288, in _prepare_impl self.session.flush() File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4352, in flush self._flush(objects) File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4487, in _flush with util.safe_reraise(): ^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__ raise exc_value.with_traceback(exc_tb) File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4448, in _flush flush_context.execute() File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute rec.execute(self) File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute util.preloaded.orm_persistence.save_obj( File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj _emit_insert_statements( File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 1143, in _emit_insert_statements result = connection.execute( ^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1418, in execute return meth( ^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection return connection._execute_clauseelement( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement ret = self._execute_context( ^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context return self._exec_insertmany_context(dialect, context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2126, in _exec_insertmany_context self._handle_dbapi_exception( File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2118, in _exec_insertmany_context dialect.do_execute( File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (psycopg2.errors.DeadlockDetected) deadlock detected DETAIL: Process 25534 waits for ShareLock on transaction 160056165; blocked by process 25538. Process 25538 waits for ShareLock on transaction 160056149; blocked by process 25534. HINT: See server log for query details. CONTEXT: while inserting index tuple (81420,15) in relation "embedding_hash_idx" [SQL: INSERT INTO embeddings (model_name, hash, embedding, provider_name) VALUES (%(model_name)s, %(hash)s, %(embedding)s, %(provider_name)s) RETURNING embeddings.id, embeddings.created_at] [parameters: {'model_name': 'text-embedding-3-large', 'hash': 'cb4db2492344433d5eeda9c4c974e9cdfa8ef1439e150aadf77c46d89fc3a94d', 'embedding': <psycopg2.extensions.Binary object at 0x7f9422f536f0>, 'provider_name': 'azure_openai'}] (Background on this error at: https://sqlalche.me/e/20/e3q8) ``` ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug label 2026-02-21 18:21:20 -05:00
yindo closed this issue 2026-02-21 18:21:20 -05:00
yindo commented 2026-02-21 18:21:20 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 8, 2025):

@huangzhuo1949 Hello there! I'm here to help you with any bugs, questions, or becoming a contributor. Let's squash those bugs together!

The deadlock you're encountering is related to concurrent transactions trying to access the same resources in the embeddings table, specifically involving the embedding_hash_idx index. This can happen when multiple processes are trying to insert or update rows with the same hash value simultaneously, leading to contention and deadlocks [1].

To address this issue, consider the following strategies:

  1. Retry Logic: Implement a retry mechanism for transactions that fail due to deadlocks. This involves catching the DeadlockDetected exception and retrying the transaction after a short delay.

  2. Transaction Isolation Levels: Adjust the transaction isolation level to reduce the likelihood of deadlocks. However, be cautious as this might affect the consistency of your data.

  3. Batch Processing: If possible, reduce the number of concurrent transactions by batching operations. This can help minimize the contention for the same resources.

  4. Index Optimization: Review and optimize the indexing strategy on the embeddings table. Ensure that the indexes are necessary and efficient for the queries being executed.

  5. Database Configuration: Check your database configuration for any settings that might help mitigate deadlocks, such as lock timeouts or deadlock detection settings.

Implementing these strategies can help reduce the occurrence of deadlocks in your application.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Jan 8, 2025): <!-- Greeting --> @huangzhuo1949 Hello there! I'm here to help you with any bugs, questions, or becoming a contributor. Let's squash those bugs together! <!-- Answer --> The deadlock you're encountering is related to concurrent transactions trying to access the same resources in the `embeddings` table, specifically involving the `embedding_hash_idx` index. This can happen when multiple processes are trying to insert or update rows with the same hash value simultaneously, leading to contention and deadlocks <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/migrations/versions/64b051264f32_init.py)</sup>. To address this issue, consider the following strategies: 1. **Retry Logic**: Implement a retry mechanism for transactions that fail due to deadlocks. This involves catching the `DeadlockDetected` exception and retrying the transaction after a short delay. 2. **Transaction Isolation Levels**: Adjust the transaction isolation level to reduce the likelihood of deadlocks. However, be cautious as this might affect the consistency of your data. 3. **Batch Processing**: If possible, reduce the number of concurrent transactions by batching operations. This can help minimize the contention for the same resources. 4. **Index Optimization**: Review and optimize the indexing strategy on the `embeddings` table. Ensure that the indexes are necessary and efficient for the queries being executed. 5. **Database Configuration**: Check your database configuration for any settings that might help mitigate deadlocks, such as lock timeouts or deadlock detection settings. Implementing these strategies can help reduce the occurrence of deadlocks in your application. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/c68b10cd-82c6-4e8e-9c37-4d3943b09b6d?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/c68b10cd-82c6-4e8e-9c37-4d3943b09b6d?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/c68b10cd-82c6-4e8e-9c37-4d3943b09b6d?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/c68b10cd-82c6-4e8e-9c37-4d3943b09b6d?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/c68b10cd-82c6-4e8e-9c37-4d3943b09b6d?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/c68b10cd-82c6-4e8e-9c37-4d3943b09b6d?feedback_type=other)</sup>
yindo commented 2026-02-21 18:21:20 -05:00
Author
Owner
Copy Link
Copy Source

@huangzhuo1949 commented on GitHub (Jan 8, 2025):

fix in #12502

@huangzhuo1949 commented on GitHub (Jan 8, 2025): fix in #12502
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7583
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?