langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Redis lock timeout error in tool provider credential operations #21278

New Issue
Closed
opened 2026-02-21 20:11:44 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 20:11:44 -05:00
Owner
Copy Link
Copy Source

Originally created by @Mairuis on GitHub (Dec 25, 2025).

Originally assigned to: @fatelei on GitHub.

Self Checks

  • This is only a request for a single feature. Do not combine multiple feature requests.
  • I have searched the existing issues and confirmed there are no similar reports.

Describe the bug

After the tool provider list caching optimization was introduced in #29101, users are experiencing lock timeout errors when adding or deleting tool provider credentials. The error message is:

{"code":"invalid_param","message":"Cannot release a lock that's no longer owned","status":400}

This error occurs during OAuth callback or credential management operations.

Steps to reproduce

  1. Configure a tool provider with OAuth authentication (e.g., Google, Slack)
  2. Initiate the OAuth flow
  3. Complete the OAuth authorization on the external provider
  4. On callback, the error occurs intermittently

Expected Behavior

Credential operations should complete successfully without lock timeout errors.

Root Cause Analysis

The issue has two contributing factors:

1. invalidate_cache uses scan_iter which is slow

In tool_provider_cache.py, the invalidate_cache method uses redis_client.scan_iter(pattern) to find matching keys:

pattern = f"tool_providers:tenant_id:{tenant_id}:*"
keys = list(redis_client.scan_iter(pattern))

Problem: Redis SCAN is O(N) where N is the total number of keys in Redis, not just matching keys. Even with few matching keys, it iterates through the entire keyspace. This becomes slow when Redis has many keys.

Better approach: Since there are only a fixed set of cache key types (builtin, api, workflow, mcp, model, all), we should delete them directly without scanning:

KNOWN_TYPES = ["builtin", "api", "workflow", "mcp", "model", "all"]
keys = [self._generate_cache_key(tenant_id, t) for t in KNOWN_TYPES]
redis_client.delete(*keys)

2. invalidate_cache is called inside the lock

In builtin_tools_manage_service.py, the add_builtin_tool_provider method acquires a lock with a 20-second timeout:

with redis_client.lock(lock, timeout=20):
    # ... operations including validate_credentials() ...
    session.commit()
    ToolProviderListCache.invalidate_cache(tenant_id)  # Called inside lock!

Combined with:

  • validate_credentials() making external API calls (slow for OAuth)
  • scan_iter scanning a large Redis keyspace

The 20-second timeout can easily be exceeded, causing the lock to expire before the block completes. When Python's context manager tries to release the lock, it fails with "Cannot release a lock that's no longer owned".

Suggested Fix

  1. Move invalidate_cache outside the lock (cache invalidation doesn't need to be atomic with DB write)
  2. Replace scan_iter with direct deletion of known cache key types

Environment

  • Dify version: latest main branch
  • Deployment: Cloud/Production
Originally created by @Mairuis on GitHub (Dec 25, 2025). Originally assigned to: @fatelei on GitHub. ## Self Checks - [x] This is only a request for a single feature. Do not combine multiple feature requests. - [x] I have searched the existing issues and confirmed there are no similar reports. ## Describe the bug After the tool provider list caching optimization was introduced in #29101, users are experiencing lock timeout errors when adding or deleting tool provider credentials. The error message is: ```json {"code":"invalid_param","message":"Cannot release a lock that's no longer owned","status":400} ``` This error occurs during OAuth callback or credential management operations. ## Steps to reproduce 1. Configure a tool provider with OAuth authentication (e.g., Google, Slack) 2. Initiate the OAuth flow 3. Complete the OAuth authorization on the external provider 4. On callback, the error occurs intermittently ## Expected Behavior Credential operations should complete successfully without lock timeout errors. ## Root Cause Analysis The issue has two contributing factors: ### 1. `invalidate_cache` uses `scan_iter` which is slow In `tool_provider_cache.py`, the `invalidate_cache` method uses `redis_client.scan_iter(pattern)` to find matching keys: ```python pattern = f"tool_providers:tenant_id:{tenant_id}:*" keys = list(redis_client.scan_iter(pattern)) ``` **Problem:** Redis `SCAN` is O(N) where N is the **total number of keys in Redis**, not just matching keys. Even with few matching keys, it iterates through the entire keyspace. This becomes slow when Redis has many keys. **Better approach:** Since there are only a fixed set of cache key types (`builtin`, `api`, `workflow`, `mcp`, `model`, `all`), we should delete them directly without scanning: ```python KNOWN_TYPES = ["builtin", "api", "workflow", "mcp", "model", "all"] keys = [self._generate_cache_key(tenant_id, t) for t in KNOWN_TYPES] redis_client.delete(*keys) ``` ### 2. `invalidate_cache` is called inside the lock In `builtin_tools_manage_service.py`, the `add_builtin_tool_provider` method acquires a lock with a 20-second timeout: ```python with redis_client.lock(lock, timeout=20): # ... operations including validate_credentials() ... session.commit() ToolProviderListCache.invalidate_cache(tenant_id) # Called inside lock! ``` Combined with: - `validate_credentials()` making external API calls (slow for OAuth) - `scan_iter` scanning a large Redis keyspace The 20-second timeout can easily be exceeded, causing the lock to expire before the block completes. When Python's context manager tries to release the lock, it fails with "Cannot release a lock that's no longer owned". ## Suggested Fix 1. Move `invalidate_cache` outside the lock (cache invalidation doesn't need to be atomic with DB write) 2. Replace `scan_iter` with direct deletion of known cache key types ## Environment - Dify version: latest main branch - Deployment: Cloud/Production
yindo added the 🐞 bug label 2026-02-21 20:11:44 -05:00
yindo closed this issue 2026-02-21 20:11:44 -05:00
yindo commented 2026-02-21 20:11:45 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Dec 25, 2025):

Your root cause analysis is correct: the lock timeout error is due to slow cache invalidation using scan_iter inside the Redis lock, especially when combined with slow external credential validation. The cache keys for tool provider lists are predictable and limited to a fixed set of types, so you can safely refactor ToolProviderListCache.invalidate_cache to delete these keys directly instead of scanning the entire Redis keyspace reference.

You should also move the cache invalidation call outside the lock in add_builtin_tool_provider. Other credential management operations (update, delete) already do this, and cache invalidation is idempotent and only needs to follow the DB commit reference. This change will not introduce consistency issues and will resolve the lock timeout problem.

No other usages of scan_iter for cache invalidation were found in the codebase, so this is the only place that needs fixing. There is no documentation in dify-docs that contradicts this approach.

Recommended fix:

  • Refactor invalidate_cache to delete all known cache keys for the tenant directly.
  • Move the cache invalidation call outside the Redis lock in add_builtin_tool_provider.

This will make credential operations reliable and performant.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Dec 25, 2025): <!-- Answer --> Your root cause analysis is correct: the lock timeout error is due to slow cache invalidation using `scan_iter` inside the Redis lock, especially when combined with slow external credential validation. The cache keys for tool provider lists are predictable and limited to a fixed set of types, so you can safely refactor `ToolProviderListCache.invalidate_cache` to delete these keys directly instead of scanning the entire Redis keyspace [reference](https://github.com/langgenius/dify/blob/fb14644a79fbf70680ee2facd2017d5ef66fe40d/api/core/helper/tool_provider_cache.py#L45-L57). You should also move the cache invalidation call outside the lock in `add_builtin_tool_provider`. Other credential management operations (update, delete) already do this, and cache invalidation is idempotent and only needs to follow the DB commit [reference](https://github.com/langgenius/dify/blob/fb14644a79fbf70680ee2facd2017d5ef66fe40d/api/services/tools/builtin_tools_manage_service.py#L209-L413). This change will not introduce consistency issues and will resolve the lock timeout problem. No other usages of `scan_iter` for cache invalidation were found in the codebase, so this is the only place that needs fixing. There is no documentation in dify-docs that contradicts this approach. **Recommended fix:** - Refactor `invalidate_cache` to delete all known cache keys for the tenant directly. - Move the cache invalidation call outside the Redis lock in `add_builtin_tool_provider`. This will make credential operations reliable and performant. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/804ce385-98dc-47b0-83c2-42f3c39bc231?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/804ce385-98dc-47b0-83c2-42f3c39bc231?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/804ce385-98dc-47b0-83c2-42f3c39bc231?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/804ce385-98dc-47b0-83c2-42f3c39bc231?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/804ce385-98dc-47b0-83c2-42f3c39bc231?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/804ce385-98dc-47b0-83c2-42f3c39bc231?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/804ce385-98dc-47b0-83c2-42f3c39bc231?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=Your%20root%20cause%20analysis%20is%20correct%3A%20the%20lock%20timeout%20error%20is%20due%20to%20slow%20cache%20invalidation%20using%20%60scan_iter%60%20inside%20the%20Redis%20lock%2C%20especially%20when%20combined%20with%20slow%20external%20credential%20validation.%20The%20cache%20keys%20for%20tool%20provider%20lists%20are%20predictable%20and%20limited%20to%20a%20fixed%20set%20of%20types%2C%20so%20you%20can%20safely%20refactor%20%60ToolProviderListCache.invalidate_cache%60%20to%20delete%20these%20keys%20directly%20instead%20of%20scanning%20the%20entire%20Redis%20keyspace%C2%A0%5Breference%5D%28https%3A//github.com/langgenius/dify/blob/fb14644a79fbf70680ee2facd2017d5ef66fe40d/api/core/helper/tool_provider_cache.py%23L45-L57%29.%0A%0AYou%20should%20also%20move%20the%20cache%20invalidation%20call%20outside%20the%20lock%20in%20%60add_builtin_tool_provider%60.%20Other%20credential%20management%20operations%20%28update%2C%20delete%29%20already%20do%20this%2C%20and%20cache%20invalidation%20is%20idempotent%20and%20only%20needs%20to%20follow%20the%20DB%20commit%C2%A0%5Breference%5D%28https%3A//github.com/langgenius/dify/blob/fb14644a79fbf70680ee2facd2017d5ef66fe40d/api/services/tools/builtin_tools_manage_service.py%23L209-L413%29.%20This%20change%20will%20not%20introduce%20consistency%20issues%20and%20will%20resolve%20the%20lock%20timeout%20problem.%0A%0ANo%20other%20usages%20of%20%60scan_iter%60%20for%20cache%20invalidation%20were%20found%20in%20the%20codebase%2C%20so%20this%20is%20the%20only%20place%20that%20needs%20fixing.%20There%20is%20no%20documentation%20in%20dify-docs%20that%20contradicts%20this%20approach.%0A%0A%2A%2ARecommended%20fix%3A%2A%2A%0A-%20Refactor%20%60invalidate_cache%60%20to%20delete%20all%20known%20cache%20keys%20for%20the%20tenant%20directly.%0A-%20Move%20the%20cache%20invalidation%20call%20outside%20the%20Redis%20lock%20in%20%60add_builtin_tool_provider%60.%0A%0AThis%20will%20make%20credential%20operations%20reliable%20and%20performant.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/30149)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#21278
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?