Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3363] [FEAT]: Support Ktransformers as LLM #2166

New Issue
Closed
opened 2026-02-22 18:28:25 -05:00 by yindo · 8 comments
yindo commented 2026-02-22 18:28:25 -05:00
Owner
Copy Link
Copy Source

Originally created by @WaterYue on GitHub (Feb 27, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3363

What would you like to see?

When will AnythingLLmm support ktransformers, the ktransformers can let the AI use less hardware to complete high quantity answers!!!

Here is the link:
https://github.com/kvcache-ai/ktransformers

Originally created by @WaterYue on GitHub (Feb 27, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3363 ### What would you like to see? When will AnythingLLmm support ktransformers, the ktransformers can let the AI use less hardware to complete high quantity answers!!! Here is the link: https://github.com/kvcache-ai/ktransformers
yindo added the enhancementIntegration Requestfeature request labels 2026-02-22 18:28:25 -05:00
yindo closed this issue 2026-02-22 18:28:25 -05:00
yindo commented 2026-02-22 18:28:26 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Feb 27, 2025):

If this is just another inference service that is OpenAI Compatible, you can just use the base URL and connection information in the Generic OpenAI Connector for the LLM and it will work the same.

@timothycarambat commented on GitHub (Feb 27, 2025): If this is just another inference service that is OpenAI Compatible, you can just use the base URL and connection information in the Generic OpenAI Connector for the LLM and it will work the same.
yindo commented 2026-02-22 18:28:26 -05:00
Author
Owner
Copy Link
Copy Source

@jiazhen-code commented on GitHub (Feb 28, 2025):

I am currently trying to run KTransformer (Deepseek-R1) + Generic OpenAI Connector with AnythingLLM on an Apple M1 computer. While the server shows normal responses, the client keeps getting stuck at the last returned character until I manually click the pause button, after which the final character is displayed.

@jiazhen-code commented on GitHub (Feb 28, 2025): I am currently trying to run KTransformer (Deepseek-R1) + Generic OpenAI Connector with AnythingLLM on an Apple M1 computer. While the server shows normal responses, the client keeps getting stuck at the last returned character until I manually click the pause button, after which the final character is displayed.
yindo commented 2026-02-22 18:28:26 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Feb 28, 2025):

Sounds like KTransformers does not return a stop reason - thus why the connection stream does not close. Can you confirm if the last chunk from KTransformer returns a stop reason?

@timothycarambat commented on GitHub (Feb 28, 2025): Sounds like KTransformers does not return a stop reason - thus why the connection stream does not close. Can you confirm if the last chunk from KTransformer returns a stop reason?
yindo commented 2026-02-22 18:28:26 -05:00
Author
Owner
Copy Link
Copy Source

@jiazhen-code commented on GitHub (Mar 2, 2025):

Image

There is a stop reason like this.

@jiazhen-code commented on GitHub (Mar 2, 2025): <img width="1108" alt="Image" src="https://github.com/user-attachments/assets/3c102e6a-0adf-41ab-9a30-9665b20252f8" /> There is a stop reason like this.
yindo commented 2026-02-22 18:28:26 -05:00
Author
Owner
Copy Link
Copy Source

@jiazhen-code commented on GitHub (Mar 2, 2025):

I'm sure it's because there is no finish_reason in the last chunk. I simply fix it by adding the reason in the last chunk as follows (in ktransformers/server/schemas/assistants/streaming.py):

Image
@jiazhen-code commented on GitHub (Mar 2, 2025): I'm sure it's because there is no finish_reason in the last chunk. I simply fix it by adding the reason in the last chunk as follows (in `ktransformers/server/schemas/assistants/streaming.py`): <img width="862" alt="Image" src="https://github.com/user-attachments/assets/511a7dc2-2fea-4a41-be5c-bc0803805af5" />
yindo commented 2026-02-22 18:28:26 -05:00
Author
Owner
Copy Link
Copy Source

@zhugh2333 commented on GitHub (Mar 3, 2025):

does it work?

@zhugh2333 commented on GitHub (Mar 3, 2025): does it work?
yindo commented 2026-02-22 18:28:26 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Mar 3, 2025):

That should work finish_reason is different from [DONE]. [DONE] is for Server side event streaming completion while finish_reason is for response streaming conclusion.

https://github.com/Mintplex-Labs/anything-llm/blob/fab74037fae86e5c11b3ad0caecdb66a0dff691e/server/utils/AiProviders/genericOpenAi/index.js#L312-L316

@timothycarambat commented on GitHub (Mar 3, 2025): That should work `finish_reason` is different from `[DONE]`. `[DONE]` is for Server side event streaming completion while `finish_reason` is for response streaming conclusion. https://github.com/Mintplex-Labs/anything-llm/blob/fab74037fae86e5c11b3ad0caecdb66a0dff691e/server/utils/AiProviders/genericOpenAi/index.js#L312-L316
yindo commented 2026-02-22 18:28:26 -05:00
Author
Owner
Copy Link
Copy Source

@zhugh2333 commented on GitHub (May 26, 2025):

Yes, this can work. Thank you for your answer.

@zhugh2333 commented on GitHub (May 26, 2025): Yes, this can work. Thank you for your answer.
yindo changed title from [FEAT]: Support Ktransformers as LLM to [GH-ISSUE #3363] [FEAT]: Support Ktransformers as LLM 2026-06-05 14:44:57 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label Integration Request enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2166
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?