Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2928] [FEAT]: Specify whisper transcription language #1867

New Issue
Closed
opened 2026-02-22 18:26:54 -05:00 by yindo · 7 comments
yindo commented 2026-02-22 18:26:54 -05:00
Owner
Copy Link
Copy Source

Originally created by @khalilxg on GitHub (Jan 2, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2928

How are you running AnythingLLM?

Docker (local)

What happened?

I'm encountering an issue with the Whisper integration in AnythingLLM. Despite setting the language parameter to "ar" in the OpenAI Whisper API, the transcription often returns transliterated Arabic (Arabic words in Latin script) instead of Arabic script. I've tried various methods to address this, but none have worked so far.

Expected Behavior: The transcription should return Arabic text in Arabic script (e.g., "مرحبا" for "hello").

Actual Behavior: The transcription returns transliterated Arabic in Latin script (e.g., "marhaban" for "hello").

Environment:

AnythingLLM Version: docker latest
Operating System: debian
Additional Context: I've followed the Whisper documentation and confirmed that the language parameter is set correctly. This issue might be related to how the API processes Arabic audio or interprets the transcription language.

Request for Resolution: Please provide guidance or a workaround to force Whisper to transcribe Arabic speech into Arabic script. If this is a limitation of the current implementation, a feature to enforce script-based output would be appreciated.

Are there known steps to reproduce?

Steps to Reproduce:

Provide an Arabic audio file.
Configure the Whisper transcription with the following parameters:
model: "whisper-1"
language: "ar"
temperature: 0
Check the transcription output.

Originally created by @khalilxg on GitHub (Jan 2, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2928 ### How are you running AnythingLLM? Docker (local) ### What happened? I'm encountering an issue with the Whisper integration in AnythingLLM. Despite setting the language parameter to "ar" in the OpenAI Whisper API, the transcription often returns transliterated Arabic (Arabic words in Latin script) instead of Arabic script. I've tried various methods to address this, but none have worked so far. Expected Behavior: The transcription should return Arabic text in Arabic script (e.g., "مرحبا" for "hello"). Actual Behavior: The transcription returns transliterated Arabic in Latin script (e.g., "marhaban" for "hello"). Environment: AnythingLLM Version: docker latest Operating System: debian Additional Context: I've followed the Whisper documentation and confirmed that the language parameter is set correctly. This issue might be related to how the API processes Arabic audio or interprets the transcription language. Request for Resolution: Please provide guidance or a workaround to force Whisper to transcribe Arabic speech into Arabic script. If this is a limitation of the current implementation, a feature to enforce script-based output would be appreciated. ### Are there known steps to reproduce? Steps to Reproduce: Provide an Arabic audio file. Configure the Whisper transcription with the following parameters: model: "whisper-1" language: "ar" temperature: 0 Check the transcription output.
yindo added the enhancementfeature request labels 2026-02-22 18:26:54 -05:00
yindo closed this issue 2026-02-22 18:26:54 -05:00
yindo commented 2026-02-22 18:26:54 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 2, 2025):

needs appropriate supported language here:
https://github.com/Mintplex-Labs/anything-llm/blob/480c8b1c41b8f60ef01e6ca494b8cc80c837efbb/collector/utils/WhisperProviders/localWhisper.js#L205

language: "en", // ISO-code

@timothycarambat commented on GitHub (Jan 2, 2025): needs appropriate [supported language](https://github.com/huggingface/transformers.js/blob/1a9964fb09b8f54fcbeac46dc6aae8d76795809d/src/tokenizers.js#L3384-L3504) here: https://github.com/Mintplex-Labs/anything-llm/blob/480c8b1c41b8f60ef01e6ca494b8cc80c837efbb/collector/utils/WhisperProviders/localWhisper.js#L205 `language: "en", // ISO-code`
yindo commented 2026-02-22 18:26:54 -05:00
Author
Owner
Copy Link
Copy Source

@khalilxg commented on GitHub (Jan 2, 2025):

im using openAi whisper's api,

target language is arabic, first i've just added language variable but still got same issue,: so i've updated some lines in
anything-llm-master/collector/utils/WhisperProviders/OpenAiWhisper.js
to

`const fs = require("fs");

class OpenAiWhisper {
constructor({ options }) {
const { OpenAI: OpenAIApi } = require("openai");
if (!options.openAiKey) throw new Error("No OpenAI API key was set.");

this.openai = new OpenAIApi({
  apiKey: options.openAiKey,
});
this.model = "whisper-1";
this.temperature = 0;
this.#log("Initialized.");
this.language = "ar";
this.task = "transcribe";

}

#log(text, ...args) {
console.log(\x1b[32m[OpenAiWhisper]\x1b[0m ${text}, ...args);
}

async processFile(fullFilePath) {
return await this.openai.audio.transcriptions
.create({
file: fs.createReadStream(fullFilePath),
model: this.model,
prompt: "مرحبًا، اسمي جو، متحدث أصلي للغة العربية، وسأجري اليوم محادثة باللغة العربية حول موضوع قد تجده مثيرًا للاهتمام للغاية.",
temperature: this.temperature,
language: this.language,
task: this.task,
})
.then((response) => {
if (!response) {
return {
content: "",
error: "No content was able to be transcribed.",
};
}

    return { content: response.text, error: null };
  })
  .catch((error) => {
    this.#log(
      `Could not get any response from openai whisper`,
      error.message
    );
    return { content: "", error: error.message };
  });

}
}

module.exports = {
OpenAiWhisper,
};
`
adding prompt, language, and task, and still got same issue: input voice in pure arabic accent and i get instant latin alphabets

@khalilxg commented on GitHub (Jan 2, 2025): im using openAi whisper's api, target language is arabic, first i've just added language variable but still got same issue,: so i've updated some lines in anything-llm-master/collector/utils/WhisperProviders/OpenAiWhisper.js to `const fs = require("fs"); class OpenAiWhisper { constructor({ options }) { const { OpenAI: OpenAIApi } = require("openai"); if (!options.openAiKey) throw new Error("No OpenAI API key was set."); this.openai = new OpenAIApi({ apiKey: options.openAiKey, }); this.model = "whisper-1"; this.temperature = 0; this.#log("Initialized."); this.language = "ar"; this.task = "transcribe"; } #log(text, ...args) { console.log(`\x1b[32m[OpenAiWhisper]\x1b[0m ${text}`, ...args); } async processFile(fullFilePath) { return await this.openai.audio.transcriptions .create({ file: fs.createReadStream(fullFilePath), model: this.model, prompt: "مرحبًا، اسمي جو، متحدث أصلي للغة العربية، وسأجري اليوم محادثة باللغة العربية حول موضوع قد تجده مثيرًا للاهتمام للغاية.", temperature: this.temperature, language: this.language, task: this.task, }) .then((response) => { if (!response) { return { content: "", error: "No content was able to be transcribed.", }; } return { content: response.text, error: null }; }) .catch((error) => { this.#log( `Could not get any response from openai whisper`, error.message ); return { content: "", error: error.message }; }); } } module.exports = { OpenAiWhisper, }; ` adding prompt, language, and task, and still got same issue: input voice in pure arabic accent and i get instant latin alphabets
yindo commented 2026-02-22 18:26:54 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 2, 2025):

In that case: https://platform.openai.com/docs/guides/speech-to-text#prompting

Some languages can be written in different ways, such as simplified or traditional Chinese. The model might not always use the writing style that you want for your transcript by default. You can improve this by using a prompt in your preferred writing style.

The language helps the model determine the input language - not the output. If you add a prompt that is written in Arabic to then specify to output the translation in Arabic that may help but it is not foolproof.

Googling this shows this issue is pretty common among Whisper model users. Most wind up going to post-processing the output with an LLM for translation. So that is the current state of whisper 🤷

@timothycarambat commented on GitHub (Jan 2, 2025): In that case: https://platform.openai.com/docs/guides/speech-to-text#prompting > Some languages can be written in different ways, such as simplified or traditional Chinese. The model might not always use the writing style that you want for your transcript by default. You can improve this by using a prompt in your preferred writing style. The `language` helps the model determine the `input` language - not the output. If you add a `prompt` that is written in Arabic to then specify to output the translation in Arabic that _may_ help but it is not foolproof. Googling this shows this issue is pretty common among Whisper model users. Most wind up going to post-processing the output with an LLM for translation. So that is the current state of whisper 🤷
yindo commented 2026-02-22 18:26:54 -05:00
Author
Owner
Copy Link
Copy Source

@khalilxg commented on GitHub (Jan 22, 2025):

@timothycarambat librechat project is implementing a transcription voice feature, where user selects language and country, and the speech is transcribed into that target language. I see that voice interfaces in rag is awesome for b2c apps, ill try implement this feature hopefully itll not take too long 🚀

@khalilxg commented on GitHub (Jan 22, 2025): @timothycarambat librechat project is implementing a transcription voice feature, where user selects language and country, and the speech is transcribed into that target language. I see that voice interfaces in rag is awesome for b2c apps, ill try implement this feature hopefully itll not take too long 🚀
yindo commented 2026-02-22 18:26:55 -05:00
Author
Owner
Copy Link
Copy Source

@khalilxg commented on GitHub (Jan 22, 2025):

Also after extensive testing across various open-source projects, AnythingLLM stands out with the best RAG performance. This is primarily due to its integration with Cohere, which offers the most advanced embedding models. Other projects lack this seamless integration, and attempts to incorporate Cohere often result in project instability.

@khalilxg commented on GitHub (Jan 22, 2025): Also after extensive testing across various open-source projects, AnythingLLM stands out with the best RAG performance. This is primarily due to its integration with Cohere, which offers the most advanced embedding models. Other projects lack this seamless integration, and attempts to incorporate Cohere often result in project instability.
yindo commented 2026-02-22 18:26:55 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 22, 2025):

@khalilxg Have you tried Cohere + Reranking in the workspace? Might get even better results.

@timothycarambat commented on GitHub (Jan 22, 2025): @khalilxg Have you tried Cohere + [Reranking](https://docs.anythingllm.com/llm-not-using-my-docs#vector-database-settings--search-preference) in the workspace? Might get even better results.
yindo commented 2026-02-22 18:26:55 -05:00
Author
Owner
Copy Link
Copy Source

@khalilxg commented on GitHub (Jan 22, 2025):

Good job !

@khalilxg commented on GitHub (Jan 22, 2025): Good job !
yindo changed title from [FEAT]: Specify whisper transcription language to [GH-ISSUE #2928] [FEAT]: Specify whisper transcription language 2026-06-05 14:43:08 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1867
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?