langgenius/dify-plugin-daemon
1
0
Fork 0
mirror of https://github.com/langgenius/dify-plugin-daemon.git synced 2026-07-21 17:25:23 -04:00
Code Issues 111 Packages Projects Releases 10 Wiki Activity

[BUG]fatal error: concurrent map iteration and map write #116

New Issue
Open
opened 2026-02-16 00:19:54 -05:00 by yindo · 2 comments
yindo commented 2026-02-16 00:19:54 -05:00
Owner
Copy Link
Copy Source

Originally created by @qimingg on GitHub (May 20, 2025).

Issue Description:
I encountered the following error, after which all plugins are restarted. The worst part is that this also kills all other running workflows.
From my investigation, it seems that bindToolValidator is the root cause. This issue only occurs with DIY plugins that use outputSchema together with create_variable_message.

Steps to Reproduce:

  1. Use a DIY plugin with outputSchema and create_variable_message.
  2. Trigger the plugin.
  3. Observe that all plugins are restarted and all running workflows are killed.

Additional Information:
(If you have logs or error messages, paste them here.)

My plugin

        yield self.create_variable_message("thinking",thinking_text)
        yield self.create_variable_message("conclusion", conclusion_text)

**fatal error: concurrent map iteration and map write**

goroutine 47441 [running]:
reflect.mapiternext(0x4bcac9?)
        /usr/local/go/src/runtime/map.go:1537 +0x13
reflect.(*MapIter).Next(0xc00107d2c8?)
        /usr/local/go/src/reflect/value.go:1992 +0x74
encoding/json.mapEncoder.encode({0x7f4fe5c74170?}, 0xc00109a000, {0x1614300?, 0xc001c8dda0?, 0x1614300?}, {0x8?, 0xd5?})
        /usr/local/go/src/encoding/json/encode.go:756 +0x329
encoding/json.(*encodeState).reflectValue(0xc00109a000, {0x1614300?, 0xc001c8dda0?, 0x414da5?}, {0xc0?, 0x23?})
        /usr/local/go/src/encoding/json/encode.go:322 +0x73
encoding/json.(*encodeState).marshal(0xc00107d5e0?, {0x1614300?, 0xc001c8dda0?}, {0x0?, 0x43?})
        /usr/local/go/src/encoding/json/encode.go:298 +0xbd
encoding/json.Marshal({0x1614300, 0xc001c8dda0})
        /usr/local/go/src/encoding/json/encode.go:164 +0xbe
github.com/xeipuuv/gojsonschema.(*jsonGoLoader).LoadJSON(0x96ed85?)
        /go/pkg/mod/github.com/xeipuuv/gojsonschema@v1.2.0/jsonLoader.go:306 +0x25
github.com/xeipuuv/gojsonschema.(*Schema).Validate(0xc001908b40, {0x1ae6158?, 0xc001609060?})
        /go/pkg/mod/github.com/xeipuuv/gojsonschema@v1.2.0/validation.go:50 +0x26
github.com/langgenius/dify-plugin-daemon/internal/core/plugin_daemon.InvokeTool.bindToolValidator.func2()
        /app/internal/core/plugin_daemon/tool_service.go:109 +0x6f
github.com/langgenius/dify-plugin-daemon/internal/utils/stream.(*Stream[...]).Close(0x489f0a?)
        /app/internal/utils/stream/stream.go:186 +0x62
github.com/langgenius/dify-plugin-daemon/internal/core/plugin_daemon.GenericInvokePlugin[...].func1()
        /app/internal/core/plugin_daemon/generic.go:68 +0x8f
github.com/langgenius/dify-plugin-daemon/pkg/entities.(*Broadcast[...]).Send(0xc000373608?, {{0xc000d6ed88?, 0x3}, {0xc000d6edd0, 0x2, 0x8}})
        /app/pkg/entities/listener.go:39 +0xe7
github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager/local_runtime.(*LocalPluginRuntime).Listen.func2({0xc000373608?, 0xc001465e60?, 0xc0013240d0?})
        /app/internal/core/plugin_manager/local_runtime/io.go:24 +0x158
github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager/local_runtime.(*stdioHolder).StartStdout.func1.1()
        /app/internal/core/plugin_manager/local_runtime/stdio.go:143 +0x23
github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager/local_runtime.(*stdioHolder).StartStdout.func1({0xc000d1a420, 0x24}, {0xc000373608, 0x18, 0x18})
        /app/internal/core/plugin_manager/local_runtime/stdio.go:149 +0x282
github.com/langgenius/dify-plugin-daemon/pkg/entities/plugin_entities.ParsePluginUniversalEvent({0xc000139152, 0x67, 0xc000b4dc00?}, {0x0, 0x0}, 0xc00107dd38, 0xc00107dd28, 0xc00107dd18, 0xc00107dd08)
        /app/pkg/entities/plugin_entities/event.go:54 +0x4b6
github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager/local_runtime.(*stdioHolder).StartStdout(0xc0010ec0b0, 0x1912a50)
        /app/internal/core/plugin_manager/local_runtime/stdio.go:131 +0x28f
github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager/local_runtime.(*LocalPluginRuntime).StartPlugin.func3()
        /app/internal/core/plugin_manager/local_runtime/run.go:156 +0x55
github.com/langgenius/dify-plugin-daemon/internal/utils/routine.Submit.func1.1({0x1ae6190?, 0xc001192060?})
        /app/internal/utils/routine/pool.go:58 +0x34
runtime/pprof.Do({0x1ae5f60?, 0x26789a0?}, {{0xc00096e080?, 0x18b448d?, 0x1593a40?}}, 0xc000b4df00)
        /usr/local/go/src/runtime/pprof/runtime.go:51 +0x8c
github.com/langgenius/dify-plugin-daemon/internal/utils/routine.Submit.func1()
        /app/internal/utils/routine/pool.go:56 +0x38b
github.com/panjf2000/ants.(*goWorker).run.func1()
        /go/pkg/mod/github.com/panjf2000/ants@v1.3.0/worker.go:71 +0x8a
created by github.com/panjf2000/ants.(*goWorker).run in goroutine 27
        /go/pkg/mod/github.com/panjf2000/ants@v1.3.0/worker.go:49 +0x5c

goroutine 1 [select (no cases), 966 minutes]:
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).Run(0xc00011a1c8, 0xc000209008)
        /app/internal/server/server.go:101 +0x1ef
main.main()
        /app/cmd/server/main.go:28 +0x125

goroutine 6 [chan receive]:
github.com/panjf2000/ants.(*Pool).periodicallyPurge(0xc0000fe000)
        /go/pkg/mod/github.com/panjf2000/ants@v1.3.0/pool.go:84 +0xbe
created by github.com/panjf2000/ants.NewPool in goroutine 1
        /go/pkg/mod/github.com/panjf2000/ants@v1.3.0/pool.go:163 +0x20c

goroutine 12 [chan receive]:
github.com/panjf2000/ants.(*Pool).periodicallyPurge(0xc0000fe5a0)
        /go/pkg/mod/github.com/panjf2000/ants@v1.3.0/pool.go:84 +0xbe
created by github.com/panjf2000/ants.NewPool in goroutine 1
        /go/pkg/mod/github.com/panjf2000/ants@v1.3.0/pool.go:163 +0x20c

goroutine 13 [select, 966 minutes]:
database/sql.(*DB).connectionOpener(0xc000122dd0, {0x1ae61c8, 0xc000289bd0})
        /usr/local/go/src/database/sql/sql.go:1253 +0x87
created by database/sql.OpenDB in goroutine 1
        /usr/local/go/src/database/sql/sql.go:833 +0x130

goroutine 49 [select]:
database/sql.(*DB).connectionCleaner(0xc000122dd0, 0x0?)
        /usr/local/go/src/database/sql/sql.go:1096 +0x9c
created by database/sql.(*DB).startCleanerLocked in goroutine 1
        /usr/local/go/src/database/sql/sql.go:1083 +0x105

goroutine 54 [chan receive]:
github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager.(*PluginManager).startLocalWatcher.func1()
        /app/internal/core/plugin_manager/watcher.go:18 +0xd4
created by github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager.(*PluginManager).startLocalWatcher in goroutine 1
        /app/internal/core/plugin_manager/watcher.go:15 +0x4f

goroutine 55 [chan receive, 966 minutes]:
github.com/panjf2000/gnet/v2.(*engine).stop(0xc0000feb40, {0xc000594450?})
        /go/pkg/mod/github.com/panjf2000/gnet/v2@v2.5.5/engine_unix.go:208 +0x34
github.com/panjf2000/gnet/v2.run({0x1aeaf40, 0xc0001e16e0}, {0xc000324048, 0x1, 0x2?}, 0xc0000fe510, {0xc000161e90, 0x1, 0xc000025e70?})
        /go/pkg/mod/github.com/panjf2000/gnet/v2@v2.5.5/engine_unix.go:306 +0x69d
github.com/panjf2000/gnet/v2.Run({0x1aeaf40, 0xc0001e16e0}, {0xc00034e240, 0x12}, {0xc000025f30?, 0x0?, 0x0?})
        /go/pkg/mod/github.com/panjf2000/gnet/v2@v2.5.5/gnet.go:551 +0x12b
github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager/debugging_runtime.(*RemotePluginServer).Launch(0xc000120128)
        /app/internal/core/plugin_manager/debugging_runtime/server.go:80 +0x16c
github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager.(*PluginManager).startRemoteWatcher.func1()
        /app/internal/core/plugin_manager/watcher.go:37 +0x2a
created by github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager.(*PluginManager).startRemoteWatcher in goroutine 1
        /app/internal/core/plugin_manager/watcher.go:36 +0xc5
Originally created by @qimingg on GitHub (May 20, 2025). **Issue Description:** I encountered the following error, after which all plugins are restarted. _The worst part is that this also kills all other running workflows_. From my investigation, it seems that `bindToolValidator` is the root cause. This issue only occurs with DIY plugins that use `outputSchema` together with `create_variable_message`. **Steps to Reproduce:** 1. Use a DIY plugin with `outputSchema` and `create_variable_message`. 2. Trigger the plugin. 3. Observe that all plugins are restarted and all running workflows are killed. **Additional Information:** (If you have logs or error messages, paste them here.) My plugin ``` yield self.create_variable_message("thinking",thinking_text) yield self.create_variable_message("conclusion", conclusion_text) ``` ``` **fatal error: concurrent map iteration and map write** goroutine 47441 [running]: reflect.mapiternext(0x4bcac9?) /usr/local/go/src/runtime/map.go:1537 +0x13 reflect.(*MapIter).Next(0xc00107d2c8?) /usr/local/go/src/reflect/value.go:1992 +0x74 encoding/json.mapEncoder.encode({0x7f4fe5c74170?}, 0xc00109a000, {0x1614300?, 0xc001c8dda0?, 0x1614300?}, {0x8?, 0xd5?}) /usr/local/go/src/encoding/json/encode.go:756 +0x329 encoding/json.(*encodeState).reflectValue(0xc00109a000, {0x1614300?, 0xc001c8dda0?, 0x414da5?}, {0xc0?, 0x23?}) /usr/local/go/src/encoding/json/encode.go:322 +0x73 encoding/json.(*encodeState).marshal(0xc00107d5e0?, {0x1614300?, 0xc001c8dda0?}, {0x0?, 0x43?}) /usr/local/go/src/encoding/json/encode.go:298 +0xbd encoding/json.Marshal({0x1614300, 0xc001c8dda0}) /usr/local/go/src/encoding/json/encode.go:164 +0xbe github.com/xeipuuv/gojsonschema.(*jsonGoLoader).LoadJSON(0x96ed85?) /go/pkg/mod/github.com/xeipuuv/gojsonschema@v1.2.0/jsonLoader.go:306 +0x25 github.com/xeipuuv/gojsonschema.(*Schema).Validate(0xc001908b40, {0x1ae6158?, 0xc001609060?}) /go/pkg/mod/github.com/xeipuuv/gojsonschema@v1.2.0/validation.go:50 +0x26 github.com/langgenius/dify-plugin-daemon/internal/core/plugin_daemon.InvokeTool.bindToolValidator.func2() /app/internal/core/plugin_daemon/tool_service.go:109 +0x6f github.com/langgenius/dify-plugin-daemon/internal/utils/stream.(*Stream[...]).Close(0x489f0a?) /app/internal/utils/stream/stream.go:186 +0x62 github.com/langgenius/dify-plugin-daemon/internal/core/plugin_daemon.GenericInvokePlugin[...].func1() /app/internal/core/plugin_daemon/generic.go:68 +0x8f github.com/langgenius/dify-plugin-daemon/pkg/entities.(*Broadcast[...]).Send(0xc000373608?, {{0xc000d6ed88?, 0x3}, {0xc000d6edd0, 0x2, 0x8}}) /app/pkg/entities/listener.go:39 +0xe7 github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager/local_runtime.(*LocalPluginRuntime).Listen.func2({0xc000373608?, 0xc001465e60?, 0xc0013240d0?}) /app/internal/core/plugin_manager/local_runtime/io.go:24 +0x158 github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager/local_runtime.(*stdioHolder).StartStdout.func1.1() /app/internal/core/plugin_manager/local_runtime/stdio.go:143 +0x23 github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager/local_runtime.(*stdioHolder).StartStdout.func1({0xc000d1a420, 0x24}, {0xc000373608, 0x18, 0x18}) /app/internal/core/plugin_manager/local_runtime/stdio.go:149 +0x282 github.com/langgenius/dify-plugin-daemon/pkg/entities/plugin_entities.ParsePluginUniversalEvent({0xc000139152, 0x67, 0xc000b4dc00?}, {0x0, 0x0}, 0xc00107dd38, 0xc00107dd28, 0xc00107dd18, 0xc00107dd08) /app/pkg/entities/plugin_entities/event.go:54 +0x4b6 github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager/local_runtime.(*stdioHolder).StartStdout(0xc0010ec0b0, 0x1912a50) /app/internal/core/plugin_manager/local_runtime/stdio.go:131 +0x28f github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager/local_runtime.(*LocalPluginRuntime).StartPlugin.func3() /app/internal/core/plugin_manager/local_runtime/run.go:156 +0x55 github.com/langgenius/dify-plugin-daemon/internal/utils/routine.Submit.func1.1({0x1ae6190?, 0xc001192060?}) /app/internal/utils/routine/pool.go:58 +0x34 runtime/pprof.Do({0x1ae5f60?, 0x26789a0?}, {{0xc00096e080?, 0x18b448d?, 0x1593a40?}}, 0xc000b4df00) /usr/local/go/src/runtime/pprof/runtime.go:51 +0x8c github.com/langgenius/dify-plugin-daemon/internal/utils/routine.Submit.func1() /app/internal/utils/routine/pool.go:56 +0x38b github.com/panjf2000/ants.(*goWorker).run.func1() /go/pkg/mod/github.com/panjf2000/ants@v1.3.0/worker.go:71 +0x8a created by github.com/panjf2000/ants.(*goWorker).run in goroutine 27 /go/pkg/mod/github.com/panjf2000/ants@v1.3.0/worker.go:49 +0x5c goroutine 1 [select (no cases), 966 minutes]: github.com/langgenius/dify-plugin-daemon/internal/server.(*App).Run(0xc00011a1c8, 0xc000209008) /app/internal/server/server.go:101 +0x1ef main.main() /app/cmd/server/main.go:28 +0x125 goroutine 6 [chan receive]: github.com/panjf2000/ants.(*Pool).periodicallyPurge(0xc0000fe000) /go/pkg/mod/github.com/panjf2000/ants@v1.3.0/pool.go:84 +0xbe created by github.com/panjf2000/ants.NewPool in goroutine 1 /go/pkg/mod/github.com/panjf2000/ants@v1.3.0/pool.go:163 +0x20c goroutine 12 [chan receive]: github.com/panjf2000/ants.(*Pool).periodicallyPurge(0xc0000fe5a0) /go/pkg/mod/github.com/panjf2000/ants@v1.3.0/pool.go:84 +0xbe created by github.com/panjf2000/ants.NewPool in goroutine 1 /go/pkg/mod/github.com/panjf2000/ants@v1.3.0/pool.go:163 +0x20c goroutine 13 [select, 966 minutes]: database/sql.(*DB).connectionOpener(0xc000122dd0, {0x1ae61c8, 0xc000289bd0}) /usr/local/go/src/database/sql/sql.go:1253 +0x87 created by database/sql.OpenDB in goroutine 1 /usr/local/go/src/database/sql/sql.go:833 +0x130 goroutine 49 [select]: database/sql.(*DB).connectionCleaner(0xc000122dd0, 0x0?) /usr/local/go/src/database/sql/sql.go:1096 +0x9c created by database/sql.(*DB).startCleanerLocked in goroutine 1 /usr/local/go/src/database/sql/sql.go:1083 +0x105 goroutine 54 [chan receive]: github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager.(*PluginManager).startLocalWatcher.func1() /app/internal/core/plugin_manager/watcher.go:18 +0xd4 created by github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager.(*PluginManager).startLocalWatcher in goroutine 1 /app/internal/core/plugin_manager/watcher.go:15 +0x4f goroutine 55 [chan receive, 966 minutes]: github.com/panjf2000/gnet/v2.(*engine).stop(0xc0000feb40, {0xc000594450?}) /go/pkg/mod/github.com/panjf2000/gnet/v2@v2.5.5/engine_unix.go:208 +0x34 github.com/panjf2000/gnet/v2.run({0x1aeaf40, 0xc0001e16e0}, {0xc000324048, 0x1, 0x2?}, 0xc0000fe510, {0xc000161e90, 0x1, 0xc000025e70?}) /go/pkg/mod/github.com/panjf2000/gnet/v2@v2.5.5/engine_unix.go:306 +0x69d github.com/panjf2000/gnet/v2.Run({0x1aeaf40, 0xc0001e16e0}, {0xc00034e240, 0x12}, {0xc000025f30?, 0x0?, 0x0?}) /go/pkg/mod/github.com/panjf2000/gnet/v2@v2.5.5/gnet.go:551 +0x12b github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager/debugging_runtime.(*RemotePluginServer).Launch(0xc000120128) /app/internal/core/plugin_manager/debugging_runtime/server.go:80 +0x16c github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager.(*PluginManager).startRemoteWatcher.func1() /app/internal/core/plugin_manager/watcher.go:37 +0x2a created by github.com/langgenius/dify-plugin-daemon/internal/core/plugin_manager.(*PluginManager).startRemoteWatcher in goroutine 1 /app/internal/core/plugin_manager/watcher.go:36 +0xc5 ```
yindo added the plugin-daemon label 2026-02-16 00:19:54 -05:00
yindo commented 2026-02-16 00:19:54 -05:00
Author
Owner
Copy Link
Copy Source

@qimingg commented on GitHub (May 20, 2025):

It will be effectively alleviated when I add random sleep between the create_variable_message

@qimingg commented on GitHub (May 20, 2025): It will be effectively alleviated when I add random sleep between the `create_variable_message `
yindo commented 2026-02-16 00:19:54 -05:00
Author
Owner
Copy Link
Copy Source

@wodeshijie33941 commented on GitHub (Jun 15, 2025):

The problem analysis is as follows:
However, the problem is that the read() and close() of the stream are executed by different goroutines. At this point, the plugin daemon receives an error message from the plugin ("SESSION_MESSAGE_TYPE_ERROR"), and then executes the stream's close(). Close() internally calls the stream's before Close callback function, while stream's read() calls the filter callback function. The problem lies here, the stream's before Close callback function and read () callback function access the critical resource variables (a map variable). This critical resource variable was accessed in multiple goroutines but was not protected, resulting in a 'concurrent map iteration and map write' error and ultimately causing the daemon to crash.

Simply put, the error message sent by dify-plugin triggered the crash bug of dify-plugin-daemon

Solution:
Use locks to protect critical resources

The code to fix the bug has been submitted and is waiting to be merged

@wodeshijie33941 commented on GitHub (Jun 15, 2025): The problem analysis is as follows: However, the problem is that the read() and close() of the stream are executed by different goroutines. At this point, the plugin daemon receives an error message from the plugin ("SESSION_MESSAGE_TYPE_ERROR"), and then executes the stream's close(). Close() internally calls the stream's before Close callback function, while stream's read() calls the filter callback function. The problem lies here, the stream's before Close callback function and read () callback function access the critical resource variables (a map variable). This critical resource variable was accessed in multiple goroutines but was not protected, resulting in a 'concurrent map iteration and map write' error and ultimately causing the daemon to crash. Simply put, the error message sent by dify-plugin triggered the crash bug of dify-plugin-daemon Solution: Use locks to protect critical resources The code to fix the bug has been submitted and is waiting to be merged
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Windows
bug
cmd
documentation
enhancement
go
good first issue
plugin-daemon
pull-request
Mirrored from GitHub Pull Request
 question
No Label plugin-daemon
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify-plugin-daemon#116
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?