langgenius/dify-plugin-daemon
1
0
Fork 0
mirror of https://github.com/langgenius/dify-plugin-daemon.git synced 2026-07-21 17:25:23 -04:00
Code Issues 111 Packages Projects Releases 10 Wiki Activity

Model Invocation is there a non-blocking Socket? #223

New Issue
Closed
opened 2026-02-16 01:15:01 -05:00 by yindo · 4 comments
yindo commented 2026-02-16 01:15:01 -05:00
Owner
Copy Link
Copy Source

Originally created by @SugarFatFree on GitHub (Dec 23, 2025).

Although this code is a streaming request, the internal implementation is a blocking Socket. If many large model requests are initiated at the same time, the performance will be very poor. Is there any request that supports non-blocking Socket? Aiohttp, for example

self.session.model.llm.invoke(
                model_config=LLMModelConfig(
                    provider=llm_info.get('provider'),
                    model=llm_info.get('model'),
                    mode=llm_info.get('mode'),
                    completion_params=llm_info.get('completion_params')
                ),
                prompt_messages=[
                    SystemPromptMessage(content=user_prompt),
                ],
                stream=True
            )

I found that httpx.Client() is used, can I use httpx.AsyncClient instead?

Originally created by @SugarFatFree on GitHub (Dec 23, 2025). Although this code is a streaming request, the internal implementation is a blocking Socket. If many large model requests are initiated at the same time, the performance will be very poor. Is there any request that supports non-blocking Socket? Aiohttp, for example ``` self.session.model.llm.invoke( model_config=LLMModelConfig( provider=llm_info.get('provider'), model=llm_info.get('model'), mode=llm_info.get('mode'), completion_params=llm_info.get('completion_params') ), prompt_messages=[ SystemPromptMessage(content=user_prompt), ], stream=True ) ``` I found that httpx.Client() is used, can I use httpx.AsyncClient instead?
yindo added the question label 2026-02-16 01:15:01 -05:00
yindo closed this issue 2026-02-16 01:15:01 -05:00
yindo commented 2026-02-16 01:15:02 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Dec 23, 2025):

The codebase is implemented in Go and uses Go's standard net/http.Client for HTTP requests, which are synchronous and blocking at the socket level. There is no support for Python async clients like httpx.AsyncClient or aiohttp, and no references to these libraries in the implementation. Streaming is handled by processing the response in a separate goroutine, but the underlying socket remains blocking and concurrency is managed with Go's goroutines, not non-blocking sockets or async HTTP clients reference.

If you need true non-blocking IO or async HTTP, you would need to redesign the networking layer using Go's async libraries or leverage lower-level APIs, but this would be a significant architectural change. For improving concurrency and throughput, consider tuning goroutine usage, connection pooling, and timeouts in the current Go-based implementation.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Dec 23, 2025): <!-- Answer --> The codebase is implemented in Go and uses Go's standard net/http.Client for HTTP requests, which are synchronous and blocking at the socket level. There is no support for Python async clients like httpx.AsyncClient or aiohttp, and no references to these libraries in the implementation. Streaming is handled by processing the response in a separate goroutine, but the underlying socket remains blocking and concurrency is managed with Go's goroutines, not non-blocking sockets or async HTTP clients [reference](https://github.com/langgenius/dify-plugin-daemon/blob/bf9e6e9b9b7a97bccee21d14d6733028bdd49b36/internal/core/dify_invocation/calldify/http_client.go#L19-L44). If you need true non-blocking IO or async HTTP, you would need to redesign the networking layer using Go's async libraries or leverage lower-level APIs, but this would be a significant architectural change. For improving concurrency and throughput, consider tuning goroutine usage, connection pooling, and timeouts in the current Go-based implementation. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/fdaeefe6-f9eb-4c06-bcfd-3d8e0e986487?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/fdaeefe6-f9eb-4c06-bcfd-3d8e0e986487?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/fdaeefe6-f9eb-4c06-bcfd-3d8e0e986487?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/fdaeefe6-f9eb-4c06-bcfd-3d8e0e986487?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/fdaeefe6-f9eb-4c06-bcfd-3d8e0e986487?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/fdaeefe6-f9eb-4c06-bcfd-3d8e0e986487?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/fdaeefe6-f9eb-4c06-bcfd-3d8e0e986487?feedback_type=other)</sup> [![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=The%20codebase%20is%20implemented%20in%20Go%20and%20uses%20Go%27s%20standard%20net/http.Client%20for%20HTTP%20requests%2C%20which%20are%20synchronous%20and%20blocking%20at%20the%20socket%20level.%20There%20is%20no%20support%20for%20Python%20async%20clients%20like%20httpx.AsyncClient%20or%20aiohttp%2C%20and%20no%20references%20to%20these%20libraries%20in%20the%20implementation.%20Streaming%20is%20handled%20by%20processing%20the%20response%20in%20a%20separate%20goroutine%2C%20but%20the%20underlying%20socket%20remains%20blocking%20and%20concurrency%20is%20managed%20with%20Go%27s%20goroutines%2C%20not%20non-blocking%20sockets%20or%20async%20HTTP%20clients%C2%A0%5Breference%5D%28https%3A//github.com/langgenius/dify-plugin-daemon/blob/bf9e6e9b9b7a97bccee21d14d6733028bdd49b36/internal/core/dify_invocation/calldify/http_client.go%23L19-L44%29.%0A%0AIf%20you%20need%20true%20non-blocking%20IO%20or%20async%20HTTP%2C%20you%20would%20need%20to%20redesign%20the%20networking%20layer%20using%20Go%27s%20async%20libraries%20or%20leverage%20lower-level%20APIs%2C%20but%20this%20would%20be%20a%20significant%20architectural%20change.%20For%20improving%20concurrency%20and%20throughput%2C%20consider%20tuning%20goroutine%20usage%2C%20connection%20pooling%2C%20and%20timeouts%20in%20the%20current%20Go-based%20implementation.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify-plugin-daemon/issues/550)
yindo commented 2026-02-16 01:15:02 -05:00
Author
Owner
Copy Link
Copy Source

@Yeuoly commented on GitHub (Dec 24, 2025):

Actually, gevent is the one who take over this, you will found that gevent was applied by default inside Dify Plugin SDK, which means everything is asynchronous

@Yeuoly commented on GitHub (Dec 24, 2025): Actually, `gevent` is the one who take over this, you will found that `gevent` was applied by default inside Dify Plugin SDK, which means everything is asynchronous
yindo commented 2026-02-16 01:15:02 -05:00
Author
Owner
Copy Link
Copy Source

@SugarFatFree commented on GitHub (Dec 25, 2025):

Maybe I posted it in the wrong place, it has been re-proposed in dify-plugin-sdks

@SugarFatFree commented on GitHub (Dec 25, 2025): Maybe I posted it in the wrong place, it has been re-proposed in dify-plugin-sdks
yindo commented 2026-02-16 01:15:02 -05:00
Author
Owner
Copy Link
Copy Source

@SugarFatFree commented on GitHub (Dec 28, 2025):

Actually, gevent is the one who take over this, you will found that gevent was applied by default inside Dify Plugin SDK, which means everything is asynchronous

Just wanted to add a quick note to clarify a key point: "Async" and "Non-blocking" are not the same thing.

Even if we wrap a task in an async function, if the underlying IO is still blocking, it will still force the system to occupy a thread until the operation is done. This doesn't actually solve the scaling issue—it just moves the "blocking" from the main thread to a worker thread.

If we have too many of these "async-blocking" tasks, we'll quickly run into thread starvation and high memory overhead. To get true high performance, we need to ensure the IO itself is non-blocking so we aren't wasting threads just to wait for data.

@SugarFatFree commented on GitHub (Dec 28, 2025): > Actually, `gevent` is the one who take over this, you will found that `gevent` was applied by default inside Dify Plugin SDK, which means everything is asynchronous Just wanted to add a quick note to clarify a key point: "Async" and "Non-blocking" are not the same thing. Even if we wrap a task in an async function, if the underlying IO is still blocking, it will still force the system to occupy a thread until the operation is done. This doesn't actually solve the scaling issue—it just moves the "blocking" from the main thread to a worker thread. If we have too many of these "async-blocking" tasks, we'll quickly run into thread starvation and high memory overhead. To get true high performance, we need to ensure the IO itself is non-blocking so we aren't wasting threads just to wait for data.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Windows
bug
cmd
documentation
enhancement
go
good first issue
plugin-daemon
pull-request
Mirrored from GitHub Pull Request
 question
No Label question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify-plugin-daemon#223
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?