langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-28 05:20:07 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

mat1 and mat2 shapes cannot be multiplied when using "采用 Q&A 分段模式" #926

New Issue
Closed
opened 2026-02-21 17:29:04 -05:00 by yindo · 6 comments
yindo commented 2026-02-21 17:29:04 -05:00
Owner
Copy Link
Copy Source

Originally created by @kukuze on GitHub (Jan 18, 2024).

Self Checks

Dify version

0.4.8

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

when using xinference LLM:chatglm3 embedding:bge-large-zh-v1.5 、python:3-11 and upload csv,xlsx,xls,txt and selected"采用 Q&A 分段模式". Xinference hava the error "mat1 and mat2 shapes cannot be multiplied (1x5 and 7x4608) "

✔️ Expected Behavior

Can upload knowledge and selected"采用 Q&A 分段模式"

❌ Actual Behavior

mat1 and mat2 shapes cannot be multiplied when using "采用 Q&A 分段模式"

RuntimeError: [address=0.0.0.0:46787, pid=33993] mat1 and mat2 shapes cannot be multiplied (1x1 and 3x4096)
2024-01-18 14:35:43,597 xinference.api.restful_api 31861 ERROR [address=0.0.0.0:46787, pid=33993] mat1 and mat2 shapes cannot be multiplied (1x1 and 3x4096)
Traceback (most recent call last):
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/api/restful_api.py", line 997, in create_chat_completion
data = await model.chat(prompt, system_prompt, chat_history, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xoscar/backends/context.py", line 227, in send
return self._process_result_message(result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
raise message.as_instanceof_cause()
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xoscar/backends/pool.py", line 657, in send
result = await self._run_coro(message.message_id, coro)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xoscar/backends/pool.py", line 368, in _run_coro
return await coro
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xoscar/api.py", line 384, in on_receive
return await super().on_receive(message) # type: ignore
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 558, in on_receive
raise ex
File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive
async with self._lock:
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
result = await result
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/core/utils.py", line 35, in wrapped
ret = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/core/model.py", line 75, in wrapped_func
ret = await fn(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xoscar/api.py", line 462, in _wrapper
r = await func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/core/model.py", line 249, in chat
return await self._call_wrapper(self._model.chat, prompt, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/core/model.py", line 99, in _async_wrapper
return await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/core/model.py", line 209, in _call_wrapper
ret = await asyncio.to_thread(fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/model/llm/pytorch/chatglm.py", line 138, in chat
return super().chat(
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/model/llm/pytorch/core.py", line 484, in chat
c = self.generate(full_prompt, generate_config)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/model/llm/pytorch/core.py", line 278, in generate
for completion_chunk, completion_usage in generate_stream_chatglm(
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 56, in generator_context
response = gen.send(request)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/model/llm/pytorch/utils.py", line 548, in generate_stream_chatglm
for total_ids in model.stream_generate(
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 56, in generator_context
response = gen.send(request)
^^^^^^^^^^^^^^^^^
File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-pytorch-6b/modeling_chatglm.py", line 1156, in stream_generate
outputs = self(
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-pytorch-6b/modeling_chatglm.py", line 937, in forward
transformer_outputs = self.transformer(
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-pytorch-6b/modeling_chatglm.py", line 830, in forward
hidden_states, presents, all_hidden_states, all_self_attentions = self.encoder(
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-pytorch-6b/modeling_chatglm.py", line 640, in forward
layer_ret = layer(
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-pytorch-6b/modeling_chatglm.py", line 565, in forward
mlp_output = self.mlp(layernorm_output)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-pytorch-6b/modeling_chatglm.py", line 501, in forward
output = self.dense_4h_to_h(intermediate_parallel)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/bitsandbytes/nn/modules.py", line 450, in forward
out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py", line 562, in matmul
return MatMul8bitLt.apply(A, B, out, bias, state)
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/autograd/function.py", line 539, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py", line 421, in forward
output += torch.matmul(subA, state.subB)
^^^^^^^^^^^^^^^^^
RuntimeError: [address=0.0.0.0:46787, pid=33993] mat1 and mat2 shapes cannot be multiplied (1x1 and 3x4096)

Originally created by @kukuze on GitHub (Jan 18, 2024). ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to file this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). ### Dify version 0.4.8 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce when using xinference LLM:chatglm3 embedding:bge-large-zh-v1.5 、python:3-11 and upload csv,xlsx,xls,txt and selected"采用 Q&A 分段模式". Xinference hava the error "mat1 and mat2 shapes cannot be multiplied (1x5 and 7x4608) " ### ✔️ Expected Behavior Can upload knowledge and selected"采用 Q&A 分段模式" ### ❌ Actual Behavior mat1 and mat2 shapes cannot be multiplied when using "采用 Q&A 分段模式" RuntimeError: [address=0.0.0.0:46787, pid=33993] mat1 and mat2 shapes cannot be multiplied (1x1 and 3x4096) 2024-01-18 14:35:43,597 xinference.api.restful_api 31861 ERROR [address=0.0.0.0:46787, pid=33993] mat1 and mat2 shapes cannot be multiplied (1x1 and 3x4096) Traceback (most recent call last): File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/api/restful_api.py", line 997, in create_chat_completion data = await model.chat(prompt, system_prompt, chat_history, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xoscar/backends/context.py", line 227, in send return self._process_result_message(result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xoscar/backends/context.py", line 102, in _process_result_message raise message.as_instanceof_cause() File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xoscar/backends/pool.py", line 657, in send result = await self._run_coro(message.message_id, coro) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xoscar/backends/pool.py", line 368, in _run_coro return await coro File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xoscar/api.py", line 384, in __on_receive__ return await super().__on_receive__(message) # type: ignore ^^^^^^^^^^^^^^^^^ File "xoscar/core.pyx", line 558, in __on_receive__ raise ex File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__ async with self._lock: ^^^^^^^^^^^^^^^^^ File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__ with debug_async_timeout('actor_lock_timeout', ^^^^^^^^^^^^^^^^^ File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__ result = await result ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/core/utils.py", line 35, in wrapped ret = await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/core/model.py", line 75, in wrapped_func ret = await fn(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xoscar/api.py", line 462, in _wrapper r = await func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/core/model.py", line 249, in chat return await self._call_wrapper(self._model.chat, prompt, *args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/core/model.py", line 99, in _async_wrapper return await fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/core/model.py", line 209, in _call_wrapper ret = await asyncio.to_thread(fn, *args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/asyncio/threads.py", line 25, in to_thread return await loop.run_in_executor(None, func_call) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/model/llm/pytorch/chatglm.py", line 138, in chat return super().chat( ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/model/llm/pytorch/core.py", line 484, in chat c = self.generate(full_prompt, generate_config) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/model/llm/pytorch/core.py", line 278, in generate for completion_chunk, completion_usage in generate_stream_chatglm( ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 56, in generator_context response = gen.send(request) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/xinference/model/llm/pytorch/utils.py", line 548, in generate_stream_chatglm for total_ids in model.stream_generate( ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 56, in generator_context response = gen.send(request) ^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-pytorch-6b/modeling_chatglm.py", line 1156, in stream_generate outputs = self( File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-pytorch-6b/modeling_chatglm.py", line 937, in forward transformer_outputs = self.transformer( ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-pytorch-6b/modeling_chatglm.py", line 830, in forward hidden_states, presents, all_hidden_states, all_self_attentions = self.encoder( ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-pytorch-6b/modeling_chatglm.py", line 640, in forward layer_ret = layer( ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-pytorch-6b/modeling_chatglm.py", line 565, in forward mlp_output = self.mlp(layernorm_output) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-pytorch-6b/modeling_chatglm.py", line 501, in forward output = self.dense_4h_to_h(intermediate_parallel) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/bitsandbytes/nn/modules.py", line 450, in forward out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py", line 562, in matmul return MatMul8bitLt.apply(A, B, out, bias, state) ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/torch/autograd/function.py", line 539, in apply return super().apply(*args, **kwargs) # type: ignore[misc] ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/python-3-11/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py", line 421, in forward output += torch.matmul(subA, state.subB) ^^^^^^^^^^^^^^^^^ RuntimeError: [address=0.0.0.0:46787, pid=33993] mat1 and mat2 shapes cannot be multiplied (1x1 and 3x4096)
yindo added the 🐞 bug label 2026-02-21 17:29:04 -05:00
yindo closed this issue 2026-02-21 17:29:04 -05:00
yindo commented 2026-02-21 17:29:04 -05:00
Author
Owner
Copy Link
Copy Source

@takatost commented on GitHub (Jan 18, 2024):

Seems like there's an issue with the xinference client. Can you provide a more detailed method of reproducing the issue?

@takatost commented on GitHub (Jan 18, 2024): Seems like there's an issue with the xinference client. Can you provide a more detailed method of reproducing the issue?
yindo commented 2026-02-21 17:29:04 -05:00
Author
Owner
Copy Link
Copy Source

@kukuze commented on GitHub (Jan 18, 2024):

Seems like there's an issue with the xinference client. Can you provide a more detailed method of reproducing the issue?

Xinreference performs well when Q&A segmentation mode is not chosen，I just use xinference LLM:chatglm3 embedding:bge-large-zh-v1.5（python3-11 no use docker）。dify 0.4.8 (with docker)

@kukuze commented on GitHub (Jan 18, 2024): > Seems like there's an issue with the xinference client. Can you provide a more detailed method of reproducing the issue? Xinreference performs well when Q&A segmentation mode is not chosen，I just use xinference LLM:chatglm3 embedding:bge-large-zh-v1.5（python3-11 no use docker）。dify 0.4.8 (with docker)
yindo commented 2026-02-21 17:29:04 -05:00
Author
Owner
Copy Link
Copy Source

@kukuze commented on GitHub (Jan 18, 2024):

Seems like there's an issue with the xinference client. Can you provide a more detailed method of reproducing the issue?

613bee808d124b0f061c1cb9e2e98a3 315792c16f5d74c5a386c15662ac3c8 b91f643aae3316288b48d806f4d24c7 ![docker-compose](https://github.com/langgenius/dify/assets/67894866/29fea8b0-9acf-4081-ba64-a6253025b4f8) e9a0b1e1e3e5e9d49ce50ef2d79af9c ee6974537cfbee93a96864776e1acfa ![image](https://github.com/langgenius/dify/assets/67894866/a50a6a3a-92f1-4c2b-9c58-e11c0386e90e) 9c71518a56aecedd69557556c8b3d28
@kukuze commented on GitHub (Jan 18, 2024): > Seems like there's an issue with the xinference client. Can you provide a more detailed method of reproducing the issue? <img width="581" alt="613bee808d124b0f061c1cb9e2e98a3" src="https://github.com/langgenius/dify/assets/67894866/5638892e-0628-4567-ada0-c8f1a76050d6"> <img width="1206" alt="315792c16f5d74c5a386c15662ac3c8" src="https://github.com/langgenius/dify/assets/67894866/acf70f57-4b8c-480e-a2da-c01dee6166e8"> <img width="1520" alt="b91f643aae3316288b48d806f4d24c7" src="https://github.com/langgenius/dify/assets/67894866/39d5a9b2-8768-4451-9c59-43e66423652c"> ![docker-compose](https://github.com/langgenius/dify/assets/67894866/29fea8b0-9acf-4081-ba64-a6253025b4f8) <img width="1410" alt="e9a0b1e1e3e5e9d49ce50ef2d79af9c" src="https://github.com/langgenius/dify/assets/67894866/a6713a97-3059-466c-b506-a74873a6dfa2"> <img width="575" alt="ee6974537cfbee93a96864776e1acfa" src="https://github.com/langgenius/dify/assets/67894866/cc498a62-cb1a-4bf7-a430-6adb229f5d6b"> ![image](https://github.com/langgenius/dify/assets/67894866/a50a6a3a-92f1-4c2b-9c58-e11c0386e90e) <img width="1232" alt="9c71518a56aecedd69557556c8b3d28" src="https://github.com/langgenius/dify/assets/67894866/7c3644dd-4056-4463-b739-c2532ca8fede">
yindo commented 2026-02-21 17:29:05 -05:00
Author
Owner
Copy Link
Copy Source

@aresnow1 commented on GitHub (Jan 18, 2024):

Could you set quantization to "none" when launch chatglm3 model and try again?

@aresnow1 commented on GitHub (Jan 18, 2024): Could you set quantization to "none" when launch chatglm3 model and try again?
yindo commented 2026-02-21 17:29:05 -05:00
Author
Owner
Copy Link
Copy Source

@kukuze commented on GitHub (Jan 18, 2024):

Could you set quantization to "none" when launch chatglm3 model and try again?

Thank you for your reply，it can run as expected now. Can you tell me the reason?
cfec36142eb23399049e517c761fec8
f1d1a9bfdeaef9db8a3d136f914c572

@kukuze commented on GitHub (Jan 18, 2024): > Could you set quantization to "none" when launch chatglm3 model and try again? Thank you for your reply，it can run as expected now. Can you tell me the reason? <img width="1139" alt="cfec36142eb23399049e517c761fec8" src="https://github.com/langgenius/dify/assets/67894866/c5930990-7f86-41f9-a50f-0e32404b3298"> <img width="1143" alt="f1d1a9bfdeaef9db8a3d136f914c572" src="https://github.com/langgenius/dify/assets/67894866/ffa334e3-6666-467f-87fc-753f7db81bf5">
yindo commented 2026-02-21 17:29:05 -05:00
Author
Owner
Copy Link
Copy Source

@aresnow1 commented on GitHub (Jan 18, 2024):

Xinference uses bitsandbytes for quantization, it may have issues in concurrent scenarios, it is recommended to install vllm to handle high-concurrency situations.

@aresnow1 commented on GitHub (Jan 18, 2024): Xinference uses bitsandbytes for quantization, it may have issues in concurrent scenarios, it is recommended to install vllm to handle high-concurrency situations.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#926
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?