Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4262] [FEAT]: OpenAI Harmony support #2713

New Issue
Closed
opened 2026-02-22 18:30:54 -05:00 by yindo · 4 comments
yindo commented 2026-02-22 18:30:54 -05:00
Owner
Copy Link
Copy Source

Originally created by @wcwong on GitHub (Aug 8, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4262

Originally assigned to: @timothycarambat on GitHub.

What would you like to see?

The OpenAI open weight models (gpt-oss-20b, gpt-oss-120b) are using the OpenAI Harmony Response Format. This does not appear to be compatible with the generic OpenAI response format as connecting AnythingLLM to these models does not result in properly parsed responses.

Originally created by @wcwong on GitHub (Aug 8, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4262 Originally assigned to: @timothycarambat on GitHub. ### What would you like to see? The OpenAI open weight models (gpt-oss-20b, gpt-oss-120b) are using the OpenAI [Harmony](https://cookbook.openai.com/articles/openai-harmony) Response Format. This does not appear to be compatible with the generic OpenAI response format as connecting AnythingLLM to these models does not result in properly parsed responses.
yindo added the enhancementfeature request labels 2026-02-22 18:30:54 -05:00
yindo closed this issue 2026-02-22 18:30:55 -05:00
yindo commented 2026-02-22 18:30:55 -05:00
Author
Owner
Copy Link
Copy Source

@Nikvothe commented on GitHub (Aug 21, 2025):

I am looking for a frontend that can support this, as OpenWebUI doesn't support it either.

@Nikvothe commented on GitHub (Aug 21, 2025): I am looking for a frontend that can support this, as OpenWebUI doesn't support it either.
yindo commented 2026-02-22 18:30:55 -05:00
Author
Owner
Copy Link
Copy Source

@chaserhkj commented on GitHub (Aug 22, 2025):

What inference back end are you guys using? I have been using llama.cpp's OAI-compatible interface and it is running gpt-oss-20b fine with both OpenWebUI and AnythingLLM since the merge of ggml-org/llama.cpp#15181

Edit: This post is a good summary of what flags need to be used with llama.cpp to makes this work, if you want to try that route.

@chaserhkj commented on GitHub (Aug 22, 2025): What inference back end are you guys using? I have been using llama.cpp's OAI-compatible interface and it is running gpt-oss-20b fine with both OpenWebUI and AnythingLLM since the merge of ggml-org/llama.cpp#15181 Edit: [This post](https://github.com/ggml-org/llama.cpp/pull/15181#issuecomment-3195263404) is a good summary of what flags need to be used with llama.cpp to makes this work, if you want to try that route.
yindo commented 2026-02-22 18:30:55 -05:00
Author
Owner
Copy Link
Copy Source

@wcwong commented on GitHub (Aug 24, 2025):

First, apologies for an error on my part. I should have looked more closely and the garbage that was being generated was directly from the model -- turns out you can't run the NVIDA NIM for oss120b on an A100 even though the self-check says that there is a supported profile...

After getting it running on an H100 and verifying that I'm getting coherent output, the issue with AnythingLLM is that it doesn't appear to be recognizing all the tags.

It looks like 'analysis' may be used instead of the other thinking tab. Sample output from AnythingLLM 1.8.5-42

analysisUser asks a tongue twister/riddle. We can answer humorously. Provide the classic answer: "A woodchuck would chuck as much wood as a woodchuck could chuck if a woodchuck could chuck wood." Or mention the study: about 700 pounds. The classic answer: "It would chuck approximately 700 pounds (or 317 kg) of wood." Provide both. Since user not asking medical, but we can answer. So respond with fun answer.assistantfinalIt’s a classic tongue‑twister, so the “official” answer is just as whimsical as the question:

**“A woodchuck would chuck as much wood as a woodchuck could chuck if a woodchuck could chuck wood!”**

That said, a playful little study by a wildlife biologist once estimated that, based on the amount of dirt a woodchuck moves when digging a burrow, a woodchuck could theoretically move roughly **700 pounds (about 317 kg) of wood** if it were inclined to do so. 

Either way, it’s a fun riddle—no need to worry about any actual wood‑chucking critters in the neighborhood!
@wcwong commented on GitHub (Aug 24, 2025): First, apologies for an error on my part. I should have looked more closely and the garbage that was being generated was directly from the model -- turns out you can't run the NVIDA NIM for oss120b on an A100 even though the self-check says that there is a supported profile... After getting it running on an H100 and verifying that I'm getting coherent output, the issue with AnythingLLM is that it doesn't appear to be recognizing all the tags. It looks like 'analysis' may be used instead of the other thinking tab. Sample output from AnythingLLM 1.8.5-42 ``` analysisUser asks a tongue twister/riddle. We can answer humorously. Provide the classic answer: "A woodchuck would chuck as much wood as a woodchuck could chuck if a woodchuck could chuck wood." Or mention the study: about 700 pounds. The classic answer: "It would chuck approximately 700 pounds (or 317 kg) of wood." Provide both. Since user not asking medical, but we can answer. So respond with fun answer.assistantfinalIt’s a classic tongue‑twister, so the “official” answer is just as whimsical as the question: **“A woodchuck would chuck as much wood as a woodchuck could chuck if a woodchuck could chuck wood!”** That said, a playful little study by a wildlife biologist once estimated that, based on the amount of dirt a woodchuck moves when digging a burrow, a woodchuck could theoretically move roughly **700 pounds (about 317 kg) of wood** if it were inclined to do so. Either way, it’s a fun riddle—no need to worry about any actual wood‑chucking critters in the neighborhood! ```
yindo commented 2026-02-22 18:30:55 -05:00
Author
Owner
Copy Link
Copy Source

@angelplusultra commented on GitHub (Feb 18, 2026):

Closing this because we couldn't reproduce the bug. Tested gpt-oss:20b through Ollama and OpenRouter/OpenAI. Thoughts and content render correctly.

@angelplusultra commented on GitHub (Feb 18, 2026): Closing this because we couldn't reproduce the bug. Tested `gpt-oss:20b` through Ollama and OpenRouter/OpenAI. Thoughts and content render correctly.
yindo changed title from [FEAT]: OpenAI Harmony support to [GH-ISSUE #4262] [FEAT]: OpenAI Harmony support 2026-06-05 14:48:08 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2713
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?