anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-22 02:05:26 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

[PR #9264] feat: voice-typing using external Whisper / ALM API #13044

New Issue
Open
opened 2026-02-16 18:17:55 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 18:17:55 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/anomalyco/opencode/pull/9264

State: open
Merged: No

This pull request implements the voice typing feature. It uses external Whisper API or multimodal large models (such as gpt-4o, qwen3-omni) for voice input. It uses the last message as a prompt (context) to improve contextual recognition accuracy (key!).

Related issues and PR:

This feature follows the frontend-backend separation design of opencode.

  • TUI: Uses tools like ffmpeg, sox, rec, arecord to record the microphone (I only tested ffmpeg on Linux, as I don't have other devices)
  • APP: Uses the browser's native microphone recording interface

Two types of speech recognition services can be configured, with whisper as the default:

  • whisper: Compatible with the OpenAI /v1/audio/transcriptions interface
  • alm: Uses the speech input capability of multimodal large models for transcription

It uses the last assistant message in the current session + the text in the input box as a prompt for speech transcription. This contextual understanding ability allows you to directly voice input special terms like code paths, variable names, etc.!

For testing convenience, I have set up a web frontend with voice input functionality at https://d3ir6x3lfy3u68.cloudfront.net. And replaced the hardcoded https://app.opencode.ai in opencode (in the third commit Add web deploy skill and configurable web proxy)

Disclaimer: Most of the code was vibe coded and then roughly checked by me. So this might be a relatively rough implementation (or even just a POC). Welcome to provide various improvement suggestions, or take this idea and implement it yourself.

External resources:

Here is the demo

https://github.com/user-attachments/assets/e82fd38c-a80f-48a5-8643-062a61481e9a

https://github.com/user-attachments/assets/26768eba-67e4-4e42-abff-bfb4b9948c67

**Original Pull Request:** https://github.com/anomalyco/opencode/pull/9264 **State:** open **Merged:** No --- This pull request implements the voice typing feature. It uses external Whisper API or multimodal large models (such as gpt-4o, qwen3-omni) for voice input. It uses the last message as a prompt (context) to improve contextual recognition accuracy (key!). Related issues and PR: - #3827 - Fixes: #2425 - Fixes: #4695 This feature follows the frontend-backend separation design of opencode. - TUI: Uses tools like ffmpeg, sox, rec, arecord to record the microphone (I only tested ffmpeg on Linux, as I don't have other devices) - APP: Uses the browser's native microphone recording interface Two types of speech recognition services can be configured, with whisper as the default: - whisper: Compatible with the OpenAI /v1/audio/transcriptions interface - alm: Uses the speech input capability of multimodal large models for transcription It uses the last assistant message in the current session + the text in the input box as a prompt for speech transcription. **This contextual understanding ability allows you to directly voice input special terms like code paths, variable names, etc.!** For testing convenience, I have set up a web frontend with voice input functionality at https://d3ir6x3lfy3u68.cloudfront.net. And replaced the hardcoded https://app.opencode.ai in opencode (in the third commit [Add web deploy skill and configurable web proxy](https://github.com/anomalyco/opencode/commit/3ea0a81c9674cbf364c84f68f648d6e0e46102c8)) Disclaimer: Most of the code was vibe coded and then roughly checked by me. So this might be a relatively rough implementation (or even just a POC). Welcome to provide various improvement suggestions, or take this idea and implement it yourself. External resources: - I used https://github.com/heimoshuiyu/whisper-fastapi to set up a local whisper API interface - I used the `Qwen/Qwen3-Omni-30B-A3B-Instruct` model on https://cloud.siliconflow.cn / https://cloud.siliconflow.com Here is the demo https://github.com/user-attachments/assets/e82fd38c-a80f-48a5-8643-062a61481e9a https://github.com/user-attachments/assets/26768eba-67e4-4e42-abff-bfb4b9948c67
yindo added the pull-request label 2026-02-16 18:17:55 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#13044
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?