mirror of
https://github.com/langgenius/dify-official-plugins.git
synced 2026-07-21 17:45:23 -04:00
Run failed: [xinference] Error: PluginInvokeError: {"args":{},"error_type":"APIError","message":"An error occurred during streaming"} #144
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @moshilangzi on GitHub (Mar 28, 2025).
Self Checks
Dify version
1.0.0
Plugin version
Xorbits Inference 0.0.2
Cloud or Self Hosted
Self Hosted (Source)
Steps to reproduce
use xinference deploy model :Qwen/Qwen2.5-VL-32B-Instruct
4*24G GPU4090
I deployed the Qwen2.5-vl-32b-instruct multimodal model using Xinference, with VLLM as the inference engine and a context length set to 32K. When I directly call the Xinference API for multimodal question answering, it responds quickly and normally. However, when I use the Xinference plugin in Dify for multimodal question answering, uploading two images results in an error.。
✔️ Error log
ERROR
`ERROR 03-31 15:19:44 [async_llm_engine.py:68] Engine background task failed
ERROR 03-31 15:19:44 [async_llm_engine.py:68] Traceback (most recent call last):
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 58, in _log_task_completion
ERROR 03-31 15:19:44 [async_llm_engine.py:68] return_value = task.result()
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 856, in run_engine_loop
ERROR 03-31 15:19:44 [async_llm_engine.py:68] result = task.result()
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 779, in engine_step
ERROR 03-31 15:19:44 [async_llm_engine.py:68] request_outputs = await self.engine.step_async(virtual_engine)
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 353, in step_async
ERROR 03-31 15:19:44 [async_llm_engine.py:68] outputs = await self.model_executor.execute_model_async(
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 354, in execute_model_async
ERROR 03-31 15:19:44 [async_llm_engine.py:68] return await self._driver_execute_model_async(execute_model_req)
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/executor/mp_distributed_executor.py", line 209, in _driver_execute_model_async
ERROR 03-31 15:19:44 [async_llm_engine.py:68] return await self.driver_exec_model(execute_model_req)
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR 03-31 15:19:44 [async_llm_engine.py:68] result = self.fn(*self.args, **self.kwargs)
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 420, in execute_model
ERROR 03-31 15:19:44 [async_llm_engine.py:68] output = self.model_runner.execute_model(
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 03-31 15:19:44 [async_llm_engine.py:68] return func(*args, **kwargs)
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1742, in execute_model
ERROR 03-31 15:19:44 [async_llm_engine.py:68] hidden_or_intermediate_states = model_executable(
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
ERROR 03-31 15:19:44 [async_llm_engine.py:68] return self._call_impl(*args, **kwargs)
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
ERROR 03-31 15:19:44 [async_llm_engine.py:68] return forward_call(*args, **kwargs)
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1080, in forward
ERROR 03-31 15:19:44 [async_llm_engine.py:68] inputs_embeds = self.get_input_embeddings_v0(
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1014, in get_input_embeddings_v0
ERROR 03-31 15:19:44 [async_llm_engine.py:68] image_embeds = self._process_image_input(image_input)
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 926, in _process_image_input
ERROR 03-31 15:19:44 [async_llm_engine.py:68] image_embeds = self.visual(pixel_values, grid_thw=grid_thw)
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
ERROR 03-31 15:19:44 [async_llm_engine.py:68] return self._call_impl(*args, **kwargs)
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
ERROR 03-31 15:19:44 [async_llm_engine.py:68] return forward_call(*args, **kwargs)
ERROR 03-31 15:19:44 [async_llm_engine.py:68] File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 644, in forward
ERROR 03-31 15:19:44 [async_llm_engine.py:68] hidden_states = hidden_states[window_index, :, :]
ERROR 03-31 15:19:44 [async_llm_engine.py:68] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 318.00 MiB. GPU 0 has a total capacity of 23.53 GiB of which 315.44 MiB is free. Including non-PyTorch memory, this process has 23.20 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, with 46.00 MiB allocated in private pools (e.g., CUDA Graphs), and 231.26 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Exception in callback _log_task_completion(error_callback=>)(<Task finishe...-variables)')>) at /home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py:48
handle: <Handle _log_task_completion(error_callback=>)(<Task finishe...-variables)')>) at /home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py:48>
Traceback (most recent call last):
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 58, in _log_task_completion
return_value = task.result()
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 856, in run_engine_loop
result = task.result()
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 779, in engine_step
request_outputs = await self.engine.step_async(virtual_engine)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 353, in step_async
outputs = await self.model_executor.execute_model_async(
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 354, in execute_model_async
return await self._driver_execute_model_async(execute_model_req)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/executor/mp_distributed_executor.py", line 209, in _driver_execute_model_async
return await self.driver_exec_model(execute_model_req)
File "/home/anaconda3/envs/xinference14/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 420, in execute_model
output = self.model_runner.execute_model(
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1742, in execute_model
hidden_or_intermediate_states = model_executable(
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1080, in forward
inputs_embeds = self.get_input_embeddings_v0(
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1014, in get_input_embeddings_v0
image_embeds = self._process_image_input(image_input)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 926, in _process_image_input
image_embeds = self.visual(pixel_values, grid_thw=grid_thw)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 644, in forward
hidden_states = hidden_states[window_index, :, :]
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 318.00 MiB. GPU 0 has a total capacity of 23.53 GiB of which 315.44 MiB is free. Including non-PyTorch memory, this process has 23.20 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, with 46.00 MiB allocated in private pools (e.g., CUDA Graphs), and 231.26 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/anaconda3/envs/xinference14/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 70, in _log_task_completion
raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on GitHub. See stack trace above for the actual cause.
2025-03-31 15:19:44,286 xinference.core.model 3187124 ERROR Model actor is out of memory, model id: Qwen2.5-VL-32B, error: CUDA out of memory. Tried to allocate 318.00 MiB. GPU 0 has a total capacity of 23.53 GiB of which 315.44 MiB is free. Including non-PyTorch memory, this process has 23.20 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, with 46.00 MiB allocated in private pools (e.g., CUDA Graphs), and 231.26 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Traceback (most recent call last):
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/xinference/core/model.py", line 569, in _to_async_gen
async for v in gen:
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/xinference/model/llm/utils.py", line 411, in _async_to_chat_completion_chunks
async for chunk in chunks:
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/xinference/model/llm/vllm/core.py", line 656, in stream_results
async for _request_output in results_generator:
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 1037, in generate
async for output in await self.add_request(
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 116, in generator
raise result
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 58, in _log_task_completion
return_value = task.result()
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 856, in run_engine_loop
result = task.result()
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 779, in engine_step
request_outputs = await self.engine.step_async(virtual_engine)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 353, in step_async
outputs = await self.model_executor.execute_model_async(
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 354, in execute_model_async
return await self._driver_execute_model_async(execute_model_req)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/executor/mp_distributed_executor.py", line 209, in _driver_execute_model_async
return await self.driver_exec_model(execute_model_req)
File "/home/anaconda3/envs/xinference14/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 420, in execute_model
output = self.model_runner.execute_model(
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1742, in execute_model
hidden_or_intermediate_states = model_executable(
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1080, in forward
inputs_embeds = self.get_input_embeddings_v0(
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1014, in get_input_embeddings_v0
image_embeds = self._process_image_input(image_input)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 926, in _process_image_input
image_embeds = self.visual(pixel_values, grid_thw=grid_thw)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/anaconda3/envs/xinference14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 644, in forward
hidden_states = hidden_states[window_index, :, :]
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 318.00 MiB. GPU 0 has a total capacity of 23.53 GiB of which 315.44 MiB is free. Including non-PyTorch memory, this process has 23.20 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, with 46.00 MiB allocated in private pools (e.g., CUDA Graphs), and 231.26 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)`
@king2088 commented on GitHub (May 28, 2025):
Same error
@SkywingsWang commented on GitHub (May 29, 2025):
same error
@hjlarry commented on GitHub (Jul 3, 2025):
can you try change the env variable
MULTIMODAL_SEND_FORMATtourlto check whether it works?@dosubot[bot] commented on GitHub (Aug 20, 2025):
Hi, @moshilangzi. I'm Dosu, and I'm helping the dify-official-plugins team manage their backlog and am marking this issue as stale.
Issue Summary:
MULTIMODAL_SEND_FORMATto"url"as a workaround.Next Steps:
Thank you for your understanding and contribution!