langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-20 19:50:06 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Agent node intermittently fails with "key not found" PluginInvokeError (ValidationError) #22201

New Issue
Open
opened 2026-02-21 20:16:08 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 20:16:08 -05:00
Owner
Copy Link
Copy Source

Originally created by @Wondery6 on GitHub (Feb 12, 2026).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.12.1

Cloud or Self Hosted

Cloud

Steps to reproduce

Environment

  • Platform: Dify Cloud (official hosted version)

Bug Description

Agent node in Chatflow intermittently fails with a PluginInvokeError. The error contains 'key not found' as an invalid JSON input.

Failure rate: Approximately 20% (1 in 5 requests)

Error Message

Failed to transform agent message: req_id: 36f366a7fd PluginInvokeError: {
  "args": {},
  "error_type": "ValidationError",
  "message": "1 validation error for dict[str,any]\n Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='key not found', input_type=str]"
}

Steps to Reproduce

  1. Create a Chatflow with an Agent node
  2. Configure Agent with:
    • Model: gemini-3-flash-preview (also reproduced with GPT-4)
    • Tools: 2 workflow-type tools
  3. Run the chatflow multiple times (5-10 times)
  4. Observe intermittent failures (~20%)

Agent Configuration

{
  "model": {
    "model": "gemini-3-flash-preview",
    "provider": "langgenius/gemini/google"
  },
  "tools": [
    {
      "tool_name": "addAppointTimeToCrm",
      "type": "workflow",
      "parameters": {
        "appoint_time": null,
        "conversation_id": null,
        "visitor_id": null
      }
    },
    {
      "tool_name": "addRemarkToCrm",
      "type": "workflow",
      "parameters": {
        "conversation_id": null,
        "remark": null,
        "visitor_id": null
      }
    }
  ],
  "maximum_iterations": 10
}

Key Observations

  1. Intermittent: Same configuration succeeds ~80% of the time, fails ~20%
  2. Model-independent: Reproduced with both Gemini and GPT-4
  3. Tool-type specific: Both failing tools are workflow type (custom workflows as tools)
  4. Error source: Plugin Daemon returns 'key not found' string instead of valid JSON

Related Error

When a tool with typo in name was used, a clearer error appeared:

PluginInvokeError: {"error_type":"KeyError","message":"'addRemarkToCrm'"}

This suggests the error originates from a dictionary/cache lookup failure inside Plugin Daemon.

Technical Analysis

Error occurs in api/core/workflow/nodes/agent/agent_node.py:153-163:

except PluginDaemonClientSideError as e:
    transform_error = AgentMessageTransformError(
        f"Failed to transform agent message: {str(e)}", original_error=e
    )

The 'key not found' string appears to be returned by Plugin Daemon when:

  • A cache key lookup fails
  • Or an internal dictionary access fails

The intermittent nature suggests a race condition or cache expiration issue.

Expected Behavior

Agent node should execute reliably without intermittent failures.

Workaround Attempted

  • Renamed tools to correct spelling ✅
  • Tested with different models (GPT-4, Gemini) ✅
  • Issue persists

Request IDs (for internal debugging)

  • 36f366a7fd
  • f9191fba9c
  • 47c6324c70
  • 18ce2f4e05
Image

✔️ Expected Behavior

Agent node should execute reliably without intermittent failures.

❌ Actual Behavior

Agent node intermittently fails (~30% of requests) with error:

PluginInvokeError: {"error_type":"ValidationError","message":"1 validation error for dict[str,any] Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='key not found', input_type=str]"}

Plugin Daemon returns the string 'key not found' instead of valid JSON, causing Pydantic validation to fail.

Originally created by @Wondery6 on GitHub (Feb 12, 2026). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.12.1 ### Cloud or Self Hosted Cloud ### Steps to reproduce ## Environment - **Platform**: Dify Cloud (official hosted version) ## Bug Description Agent node in Chatflow **intermittently** fails with a `PluginInvokeError`. The error contains `'key not found'` as an invalid JSON input. **Failure rate**: Approximately 20% (1 in 5 requests) ## Error Message ``` Failed to transform agent message: req_id: 36f366a7fd PluginInvokeError: { "args": {}, "error_type": "ValidationError", "message": "1 validation error for dict[str,any]\n Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='key not found', input_type=str]" } ``` ## Steps to Reproduce 1. Create a Chatflow with an Agent node 2. Configure Agent with: - Model: `gemini-3-flash-preview` (also reproduced with GPT-4) - Tools: 2 workflow-type tools 3. Run the chatflow multiple times (5-10 times) 4. Observe intermittent failures (~20%) ## Agent Configuration ```json { "model": { "model": "gemini-3-flash-preview", "provider": "langgenius/gemini/google" }, "tools": [ { "tool_name": "addAppointTimeToCrm", "type": "workflow", "parameters": { "appoint_time": null, "conversation_id": null, "visitor_id": null } }, { "tool_name": "addRemarkToCrm", "type": "workflow", "parameters": { "conversation_id": null, "remark": null, "visitor_id": null } } ], "maximum_iterations": 10 } ``` ## Key Observations 1. **Intermittent**: Same configuration succeeds ~80% of the time, fails ~20% 2. **Model-independent**: Reproduced with both Gemini and GPT-4 3. **Tool-type specific**: Both failing tools are `workflow` type (custom workflows as tools) 4. **Error source**: Plugin Daemon returns `'key not found'` string instead of valid JSON ## Related Error When a tool with typo in name was used, a clearer error appeared: ``` PluginInvokeError: {"error_type":"KeyError","message":"'addRemarkToCrm'"} ``` This suggests the error originates from a dictionary/cache lookup failure inside Plugin Daemon. ## Technical Analysis Error occurs in `api/core/workflow/nodes/agent/agent_node.py:153-163`: ```python except PluginDaemonClientSideError as e: transform_error = AgentMessageTransformError( f"Failed to transform agent message: {str(e)}", original_error=e ) ``` The `'key not found'` string appears to be returned by Plugin Daemon when: - A cache key lookup fails - Or an internal dictionary access fails The intermittent nature suggests a **race condition** or **cache expiration** issue. ## Expected Behavior Agent node should execute reliably without intermittent failures. ## Workaround Attempted - Renamed tools to correct spelling ✅ - Tested with different models (GPT-4, Gemini) ✅ - Issue persists ## Request IDs (for internal debugging) - `36f366a7fd` - `f9191fba9c` - `47c6324c70` - `18ce2f4e05` --- <img width="1844" height="941" alt="Image" src="https://github.com/user-attachments/assets/15f196df-b94c-40b5-857d-f25679c00b21" /> ### ✔️ Expected Behavior Agent node should execute reliably without intermittent failures. ### ❌ Actual Behavior Agent node intermittently fails (~30% of requests) with error: PluginInvokeError: {"error_type":"ValidationError","message":"1 validation error for dict[str,any] Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='key not found', input_type=str]"} Plugin Daemon returns the string 'key not found' instead of valid JSON, causing Pydantic validation to fail.
yindo added the 🐞 bug🤖 feat:agent labels 2026-02-21 20:16:08 -05:00
yindo commented 2026-02-21 20:16:09 -05:00
Author
Owner
Copy Link
Copy Source

@lucasmagalhaes-solfacil commented on GitHub (Feb 12, 2026):

We expecting the same error..
We notice the Dify release a new version yesterday!

@lucasmagalhaes-solfacil commented on GitHub (Feb 12, 2026): We expecting the same error.. We notice the Dify release a new version yesterday!
yindo commented 2026-02-21 20:16:09 -05:00
Author
Owner
Copy Link
Copy Source

@lucasmagalhaes-solfacil commented on GitHub (Feb 12, 2026):

Any updates about the issue?

@lucasmagalhaes-solfacil commented on GitHub (Feb 12, 2026): Any updates about the issue?
yindo commented 2026-02-21 20:16:09 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Feb 12, 2026):

@dosu

@crazywoola commented on GitHub (Feb 12, 2026): @dosu
yindo commented 2026-02-21 20:16:09 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 12, 2026):

Thanks for the detailed bug report! This is a known issue tracked in Issue #32286.

Root Cause

The Plugin Daemon returns the literal string 'key not found' instead of valid JSON when it fails to look up tool information due to Redis cache inconsistency. The specific problems include:

  • Cache being trusted over database state
  • Race conditions where cache is written before database transaction commits
  • Ineffective distributed locking under concurrent requests

Your analysis is correct—the intermittent, model-independent nature confirms this is a cache/daemon layer issue, not LLM-related.

Fixes Already Merged

Several PRs have been merged to address this:

  1. PR #31689 - Migrated from process-local to Redis-backed distributed cache (most significant fix)
  2. PR #31708 - Reduced cache TTL from 24h to 1h
  3. PR #32008 - Added _infer_tool_provider_type method to handle missing/incorrect tool type info for workflow tools
  4. PR #31218 - Fixed inconsistent error message parsing from plugin daemon

Workarounds for Now

Since you're on Dify Cloud and waiting for these fixes to be deployed:

  1. Reconfigure your workflow tools in the agent node UI—this forces the frontend to properly save the provider_type field (related issue #32005)
  2. Implement retry logic in your application layer with exponential backoff
  3. Consider using built-in plugins instead of workflow-type tools if possible (less susceptible to cache issues)

The request IDs you provided should help the Dify Cloud team investigate further on their end.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Feb 12, 2026): <!-- Answer --> Thanks for the detailed bug report! This is a **known issue** tracked in [Issue #32286](https://github.com/langgenius/dify/issues/32286). ## Root Cause The Plugin Daemon returns the literal string `'key not found'` instead of valid JSON when it fails to look up tool information due to [Redis cache inconsistency](https://github.com/langgenius/dify/issues/31130). The specific problems include: - Cache being trusted over database state - Race conditions where cache is written before database transaction commits - Ineffective distributed locking under concurrent requests Your analysis is correct—the intermittent, model-independent nature confirms this is a cache/daemon layer issue, not LLM-related. ## Fixes Already Merged Several PRs have been merged to address this: 1. **[PR #31689](https://github.com/langgenius/dify/pull/31689)** - Migrated from process-local to Redis-backed distributed cache (most significant fix) 2. **[PR #31708](https://github.com/langgenius/dify/pull/31708)** - Reduced cache TTL from 24h to 1h 3. **[PR #32008](https://github.com/langgenius/dify/pull/32008)** - Added `_infer_tool_provider_type` method to handle missing/incorrect tool type info for workflow tools 4. **[PR #31218](https://github.com/langgenius/dify/pull/31218)** - Fixed inconsistent error message parsing from plugin daemon ## Workarounds for Now Since you're on Dify Cloud and waiting for these fixes to be deployed: 1. **Reconfigure your workflow tools** in the agent node UI—this forces the frontend to properly save the `provider_type` field ([related issue #32005](https://github.com/langgenius/dify/issues/32005)) 2. **Implement retry logic** in your application layer with exponential backoff 3. **Consider using built-in plugins** instead of workflow-type tools if possible (less susceptible to cache issues) The request IDs you provided should help the Dify Cloud team investigate further on their end. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/b8a5a92c-c6a5-452d-8d79-08a7ed4703cb?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/b8a5a92c-c6a5-452d-8d79-08a7ed4703cb?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/b8a5a92c-c6a5-452d-8d79-08a7ed4703cb?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/b8a5a92c-c6a5-452d-8d79-08a7ed4703cb?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/b8a5a92c-c6a5-452d-8d79-08a7ed4703cb?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/b8a5a92c-c6a5-452d-8d79-08a7ed4703cb?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/b8a5a92c-c6a5-452d-8d79-08a7ed4703cb?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=Thanks%20for%20the%20detailed%20bug%20report%21%20This%20is%20a%20%2A%2Aknown%20issue%2A%2A%20tracked%20in%20%5BIssue%20%2332286%5D%28https%3A//github.com/langgenius/dify/issues/32286%29.%0A%0A%23%23%20Root%20Cause%0A%0AThe%20Plugin%20Daemon%20returns%20the%20literal%20string%20%60%27key%20not%20found%27%60%20instead%20of%20valid%20JSON%20when%20it%20fails%20to%20look%20up%20tool%20information%20due%20to%20%5BRedis%20cache%20inconsistency%5D%28https%3A//github.com/langgenius/dify/issues/31130%29.%20The%20specific%20problems%20include%3A%0A-%20Cache%20being%20trusted%20over%20database%20state%0A-%20Race%20conditions%20where%20cache%20is%20written%20before%20database%20transaction%20commits%0A-%20Ineffective%20distributed%20locking%20under%20concurrent%20requests%0A%0AYour%20analysis%20is%20correct%E2%80%94the%20intermittent%2C%20model-independent%20nature%20confirms%20this%20is%20a%20cache/daemon%20layer%20issue%2C%20not%20LLM-related.%0A%0A%23%23%20Fixes%20Already%20Merged%0A%0ASeveral%20PRs%20have%20been%20merged%20to%20address%20this%3A%0A%0A1.%20%2A%2A%5BPR%20%2331689%5D%28https%3A//github.com/langgenius/dify/pull/31689%29%2A%2A%20-%20Migrated%20from%20process-local%20to%20Redis-backed%20distributed%20cache%20%28most%20significant%20fix%29%0A2.%20%2A%2A%5BPR%20%2331708%5D%28https%3A//github.com/langgenius/dify/pull/31708%29%2A%2A%20-%20Reduced%20cache%20TTL%20from%2024h%20to%201h%0A3.%20%2A%2A%5BPR%20%2332008%5D%28https%3A//github.com/langgenius/dify/pull/32008%29%2A%2A%20-%20Added%20%60_infer_tool_provider_type%60%20method%20to%20handle%20missing/incorrect%20tool%20type%20info%20for%20workflow%20tools%0A4.%20%2A%2A%5BPR%20%2331218%5D%28https%3A//github.com/langgenius/dify/pull/31218%29%2A%2A%20-%20Fixed%20inconsistent%20error%20message%20parsing%20from%20plugin%20daemon%0A%0A%23%23%20Workarounds%20for%20Now%0A%0ASince%20you%27re%20on%20Dify%20Cloud%20and%20waiting%20for%20these%20fixes%20to%20be%20deployed%3A%0A1.%20%2A%2AReconfigure%20your%20workflow%20tools%2A%2A%20in%20the%20agent%20node%20UI%E2%80%94this%20forces%20the%20frontend%20to%20properly%20save%20the%20%60provider_type%60%20field%20%28%5Brelated%20issue%20%2332005%5D%28https%3A//github.com/langgenius/dify/issues/32005%29%29%0A2.%20%2A%2AImplement%20retry%20logic%2A%2A%20in%20your%20application%20layer%20with%20exponential%20backoff%0A3.%20%2A%2AConsider%20using%20built-in%20plugins%2A%2A%20instead%20of%20workflow-type%20tools%20if%20possible%20%28less%20susceptible%20to%20cache%20issues%29%0A%0AThe%20request%20IDs%20you%20provided%20should%20help%20the%20Dify%20Cloud%20team%20investigate%20further%20on%20their%20end.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/32266)
yindo commented 2026-02-21 20:16:09 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Feb 12, 2026):

This should be resolved in the cloud env, please try it again, feel free to leave a comment if you encounter this again.

@crazywoola commented on GitHub (Feb 12, 2026): This should be resolved in the cloud env, please try it again, feel free to leave a comment if you encounter this again.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug 🤖 feat:agent
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#22201
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?