Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1308] [FEAT]: Qwen LLM Support #823

New Issue
Closed
opened 2026-02-22 18:21:35 -05:00 by yindo · 9 comments
yindo commented 2026-02-22 18:21:35 -05:00
Owner
Copy Link
Copy Source

Originally created by @Sunmori on GitHub (May 8, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1308

Originally assigned to: @shatfield4 on GitHub.

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

I chose Generic Openai settings in the LLM Provider settings and filled in Qwen's API .When chatting, the dialog box stops when a word appears.I looked at Qwen's documentation and it shows compatibility with the Openai SDK.Is anyone else able to use the Qwen API properly?I'm begging for your help.
WX20240508-215903@2x

WX20240508-215809@2x

Are there known steps to reproduce?

No response

Originally created by @Sunmori on GitHub (May 8, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1308 Originally assigned to: @shatfield4 on GitHub. ### How are you running AnythingLLM? AnythingLLM desktop app ### What happened? I chose Generic Openai settings in the LLM Provider settings and filled in Qwen's API .When chatting, the dialog box stops when a word appears.I looked at Qwen's documentation and it shows compatibility with the Openai SDK.Is anyone else able to use the Qwen API properly?I'm begging for your help. ![WX20240508-215903@2x](https://github.com/Mintplex-Labs/anything-llm/assets/88074818/95dbb8bd-47dc-4be2-bc4c-f05058f5b202) ![WX20240508-215809@2x](https://github.com/Mintplex-Labs/anything-llm/assets/88074818/b0824a04-6353-4c12-8a04-7b78c329c46b) ### Are there known steps to reproduce? _No response_
yindo added the enhancementIntegration Requestfeature requestblocked labels 2026-02-22 18:21:35 -05:00
yindo closed this issue 2026-02-22 18:21:35 -05:00
yindo commented 2026-02-22 18:21:36 -05:00
Author
Owner
Copy Link
Copy Source

@hdavy2002 commented on GitHub (May 8, 2024):

This happned with me too. I was using openrouter with Meta Llama 7B instruct.

@hdavy2002 commented on GitHub (May 8, 2024): This happned with me too. I was using openrouter with Meta Llama 7B instruct.
yindo commented 2026-02-22 18:21:36 -05:00
Author
Owner
Copy Link
Copy Source

@shatfield4 commented on GitHub (May 8, 2024):

@Sunmori To confirm, are you using the fastchat OpenAi API server that it shows on Qwen's documentation or are you running Qwen on another platform that has Qwen support? The issue may be that their API spec does not use the same outputs when doing chat completions and this causes issues with streaming. If this is the case we can create an LLM provider specifically for Qwen.

@shatfield4 commented on GitHub (May 8, 2024): @Sunmori To confirm, are you using the fastchat OpenAi API server that it shows on Qwen's documentation or are you running Qwen on another platform that has Qwen support? The issue may be that their API spec does not use the same outputs when doing chat completions and this causes issues with streaming. If this is the case we can create an LLM provider specifically for Qwen.
yindo commented 2026-02-22 18:21:36 -05:00
Author
Owner
Copy Link
Copy Source

@shatfield4 commented on GitHub (May 8, 2024):

This happned with me too. I was using openrouter with Meta Llama 7B instruct.

This is a known issue with some openrouter models. This has nothing to do with AnythingLLM and is a model issue where the model quantization is most likely causing it to give garbage responses and finishing responses early.

@shatfield4 commented on GitHub (May 8, 2024): > This happned with me too. I was using openrouter with Meta Llama 7B instruct. This is a known issue with some openrouter models. This has nothing to do with AnythingLLM and is a model issue where the model quantization is most likely causing it to give garbage responses and finishing responses early.
yindo commented 2026-02-22 18:21:36 -05:00
Author
Owner
Copy Link
Copy Source

@Sunmori commented on GitHub (May 8, 2024):

@Sunmori To confirm, are you using the fastchat OpenAi API server that it shows on Qwen's documentation or are you running Qwen on another platform that has Qwen support? The issue may be that their API spec does not use the same outputs when doing chat completions and this causes issues with streaming. If this is the case we can create an LLM provider specifically for Qwen.

I'm using the Openai API rule-compatible URLs and APIs given in the official qwen documentation.
I also had the same problem with another LLM provider, both compatible with Openai api rules.
I'm wondering if there's a bug in the Generic Openai part of AnythingLLM, because I've done Generic Openai configurations on other software and it works fine.
Of course, it would be great if an LLM provider could be created specifically for qwen, which is a very well-known LLM provider in China.

qwen official documentation:https://help.aliyun.com/zh/dashscope/developer-reference/compatibility-of-openai-with-dashscope/?spm=a2c4g.11186623.0.0.1d09f400wOtLdc
Zhipu AI official documentation:https://open.bigmodel.cn/dev/api#thirdparty_frame
Screenshots of other software
image
image

@Sunmori commented on GitHub (May 8, 2024): > @Sunmori To confirm, are you using the fastchat OpenAi API server that it shows on Qwen's documentation or are you running Qwen on another platform that has Qwen support? The issue may be that their API spec does not use the same outputs when doing chat completions and this causes issues with streaming. If this is the case we can create an LLM provider specifically for Qwen. I'm using the Openai API rule-compatible URLs and APIs given in the official qwen documentation. I also had the same problem with another LLM provider, both compatible with Openai api rules. I'm wondering if there's a bug in the Generic Openai part of AnythingLLM, because I've done Generic Openai configurations on other software and it works fine. Of course, it would be great if an LLM provider could be created specifically for qwen, which is a very well-known LLM provider in China. qwen official documentation:https://help.aliyun.com/zh/dashscope/developer-reference/compatibility-of-openai-with-dashscope/?spm=a2c4g.11186623.0.0.1d09f400wOtLdc Zhipu AI official documentation:https://open.bigmodel.cn/dev/api#thirdparty_frame Screenshots of other software <img width="859" alt="image" src="https://github.com/Mintplex-Labs/anything-llm/assets/88074818/1fdc63c5-1b6c-4592-b68e-aecc07a27c6d"> <img width="1024" alt="image" src="https://github.com/Mintplex-Labs/anything-llm/assets/88074818/bb9cba76-5652-4ee8-8eb5-00d318516818">
yindo commented 2026-02-22 18:21:36 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 8, 2024):

What is the LLM provider you are using with the generic OpenAI provider? Usually, the issue when using the generic is some providers will have different stop tokens and that will halt replies earlier.

If there is a way to us to signup and access whatever provider you are using we can easily add it as a connector and eliminate this issue

@timothycarambat commented on GitHub (May 8, 2024): What is the LLM provider you are using with the generic OpenAI provider? Usually, the issue when using the generic is some providers will have different stop tokens and that will halt replies earlier. If there is a way to us to signup and access whatever provider you are using we can easily add it as a connector and eliminate this issue
yindo commented 2026-02-22 18:21:36 -05:00
Author
Owner
Copy Link
Copy Source

@Sunmori commented on GitHub (May 9, 2024):

What is the LLM provider you are using with the generic OpenAI provider? Usually, the issue when using the generic is some providers will have different stop tokens and that will halt replies earlier.

If there is a way to us to signup and access whatever provider you are using we can easily add it as a connector and eliminate this issue

I'm using the Qwen API released by Aliyun, but it seems like it's only open for Chinese use at the moment. I can give you my API if you want to add connectors.

@Sunmori commented on GitHub (May 9, 2024): > What is the LLM provider you are using with the generic OpenAI provider? Usually, the issue when using the generic is some providers will have different stop tokens and that will halt replies earlier. > > If there is a way to us to signup and access whatever provider you are using we can easily add it as a connector and eliminate this issue I'm using the Qwen API released by Aliyun, but it seems like it's only open for Chinese use at the moment. I can give you my API if you want to add connectors.
yindo commented 2026-02-22 18:21:37 -05:00
Author
Owner
Copy Link
Copy Source

@jarodttt commented on GitHub (Jun 18, 2024):

use https://dashscope.aliyuncs.com/compatible-mode/v1

@jarodttt commented on GitHub (Jun 18, 2024): use https://dashscope.aliyuncs.com/compatible-mode/v1
yindo commented 2026-02-22 18:21:37 -05:00
Author
Owner
Copy Link
Copy Source

@muzzah commented on GitHub (Jan 18, 2025):

Im pretty sure this is a bug with Anything LLM. I just did the same thing. I have LM Studio setup as a server at home with one of the Qwen coder models running in server mode. LM Studio uses the OpenAPi interface and when i query that server using Anything LLM with the open api comptible mode setup, it also spits out garbage. However If LM Studio can run these models fine then why cant Anything LLM? Im pretty sure it has something to do with the prompt its sending but since this is not configurable its difficult to tell.

@muzzah commented on GitHub (Jan 18, 2025): Im pretty sure this is a bug with Anything LLM. I just did the same thing. I have LM Studio setup as a server at home with one of the Qwen coder models running in server mode. LM Studio uses the OpenAPi interface and when i query that server using Anything LLM with the open api comptible mode setup, it also spits out garbage. However If LM Studio can run these models fine then why cant Anything LLM? Im pretty sure it has something to do with the prompt its sending but since this is not configurable its difficult to tell.
yindo commented 2026-02-22 18:21:37 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 18, 2025):

@muzzah

Im pretty sure this is a bug with Anything LLM. I just did the same thing. I have LM Studio setup as a server at home with one of the Qwen coder models running in server mode.

We have an LMStudio API connector - assuming you are using that. If your model is returning garbage when over the API but not in LMStudio's interfaces its because you forgot to load the model in the server on LMStudio with a defined chat template (on right side panel in LMStudio while on inference server page). We don't, and cant, specify the template for you over the API. The template does all the mapping for your map for you in LMStudio, we just send it OpenAI compatible messages.

That is the entire issue for your specific circumstance.

@timothycarambat commented on GitHub (Jan 18, 2025): @muzzah > Im pretty sure this is a bug with Anything LLM. I just did the same thing. I have LM Studio setup as a server at home with one of the Qwen coder models running in server mode. We have an LMStudio API connector - assuming you are using that. If your model is returning garbage when over the API but _not_ in LMStudio's interfaces its because you forgot to load the model in the server on LMStudio with a defined chat template (on right side panel in LMStudio while on inference server page). We don't, and cant, specify the template for you over the API. The template does all the mapping for your map for you in LMStudio, we just send it OpenAI compatible messages. That is the entire issue for your specific circumstance.
yindo changed title from [FEAT]: Qwen LLM Support to [GH-ISSUE #1308] [FEAT]: Qwen LLM Support 2026-06-05 14:37:23 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label Integration Request blocked enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#823
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?