mirror of
https://github.com/Twigpine/zero.git
synced 2026-11-10 05:01:28 -10:00
* ## Speech-to-text dictation Adds voice dictation to Zero. Run `/voice` to enter hold-to-record mode, then hold **Space** to dictate and release to transcribe (run `/voice` again to turn it off so Space types normally). Pick a backend with `/stt-model`. ### Backends - **Local (sherpa-onnx)** — offline, no API key. Auto-downloads a model via `/stt-model`; the engine binary is fetched once and shared across models. Family auto-detection covers the mainstream offline families (Whisper, transducer/Zipformer, Moonshine, Paraformer, SenseVoice, and the CTC families FireRedASR, NeMo, TeleSpeech, WeNet, Zipformer-CTC, Dolphin, Omnilingual — plus the encoder/decoder families Canary and FireRedASR-AED). - **Cloud batch** — Groq and OpenAI Whisper. Keys resolve from a configured provider, the encrypted credential store, or the env var; a missing/invalid key is entered inline. - **Streaming** — a warm local sherpa-onnx websocket server (live transcript as you speak) on desktop, or Deepgram / OpenAI Realtime. ### Highlights - Live audio-reactive waveform while recording (real mic levels when streaming; a synthesized wave for batch). - `/stt-model` download chooser: recommended models first, Live/Batch tag per row, size + download state, and a search filter. - Streaming transcript is inserted live into the composer and committed on stop; batch inserts on stop. Insert-for-review by default (`stt.autoSubmit` to fire straight at the agent). - The warm streaming server is released when `/voice` is turned off and on exit. ### Implementation notes - No CGO and no bundled runtimes — every OS integration shells out via `exec.Command` with discrete argv (never a shell string). - The streaming server manager mirrors the LSP manager's warm-subprocess lifecycle. - OS-boundary code (recorders, transcribers, server) is unit-tested via injected fakes; real microphone / websocket behavior is covered by env-gated smoke tests. ### Testing - `go build ./...`, `go vet ./...`, and the config / dictation / cli / tui suites pass. - Verified end-to-end against real downloaded binaries + models (offline batch, streaming websocket server, and the model auto-download), including the FireRedASR-CTC and NeMo streaming transducer families. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added speech-to-text dictation with both batch and streaming modes, supporting local (sherpa-onnx) and cloud providers, including WAV/M4A detection. * Introduced voice dictation mode with STT model selection, local engine/model auto-download, and an inline prompt to save cloud API keys. * Added new TUI commands for voice/STT model selection and enabled F1–F12 key bindings. * **Bug Fixes** * Improved streaming dictation to preserve accumulated transcript partials and keep the editor region consistent. * Refined TUI status/picker rendering, including clearer loading/search behavior and more reliable selection alignment. <!-- end of auto-generated comment: release notes by coderabbit.ai --> * fix(dictation): address CodeRabbit review feedback - validateSTTConfig: add NumThreads bounds check - downloadVerifyExtract: stage extraction into a sibling temp dir, atomic-rename into destDir on success so a partial extract cannot leave a truncated engine binary that passes fileExists() on next run - startProcess: close pipe fds when cmd.Start() fails - ServerManager.Shutdown: honor the caller's context deadline - audioSpillDir: tighten any pre-existing directory to 0700 - startVariantDownload: refuse re-entry while a download is in flight - maybeOpenSTTDownloadPicker: defer Provider=local persistence to applyEngineComponents (commit-on-success), matching the cloud flow - fetchSTTModelsCmd: bound with a 30s timeout client+context - newSTTDownloadPickerFrom.isCurrent: compare path segments via filepath.Base to avoid substring false-positives * fix(dictation): harden recorder.Stop, Esc-cancel priority, streaming region Three correctness issues from PR review: 1) recorder.Stop() panicked (nil-deref) when called on a StartStreaming session because StartStreaming intentionally leaves r.proc nil. Reject that misuse with a clear error. Termux's batch path also leaves r.proc nil by design — keep that path working. 2) Esc was consumed by an active dictation session before the confirming second Esc could cancel a pending run, so a user mid-recording who double-Esc's to kill the run found the first Esc swallowed by dictation. Gate dictation-cancel on the Esc not being a pending-run confirm. 3) applyStreamingText hard-reset the cursor to regionStart on every partial, yanking focus away from users editing elsewhere in the composer, and stored regionStart/End as absolute indices that became stale the moment the user typed or pasted before the live region. Preserve the cursor when it's outside the region, and snapshot the prefix BEFORE the region so the next partial can detect external edits and shift [start,end) to stay aligned.
129 lines
3.9 KiB
Go
129 lines
3.9 KiB
Go
package dictation
|
|
|
|
import (
|
|
"bytes"
|
|
"context"
|
|
"io"
|
|
"os"
|
|
"os/exec"
|
|
"time"
|
|
)
|
|
|
|
// commandSpec describes one capture-process invocation. Argv is always
|
|
// discrete — never a shell string (the same invariant documented in
|
|
// internal/imageinput/clipboard.go).
|
|
type commandSpec struct {
|
|
name string
|
|
args []string
|
|
// wantStdout wires up a stdout pipe (streaming capture reads raw PCM).
|
|
wantStdout bool
|
|
// stopViaStdin, when non-empty, makes graceful stop write this to the
|
|
// process's stdin instead of signaling — ffmpeg finalizes cleanly on "q",
|
|
// and Windows has no os.Interrupt delivery anyway.
|
|
stopViaStdin string
|
|
}
|
|
|
|
// processHandle is a started capture process. Injectable via processStarter so
|
|
// unit tests fake OS-boundary behavior, mirroring internal/lsp's serverStarter
|
|
// seam (this package's clipboard.go-style shell-outs get the test seam that
|
|
// file lacks).
|
|
type processHandle interface {
|
|
// StopGracefully asks the tool to finish and flush output (SIGINT, or the
|
|
// spec's stdin stop text). The recorder falls back to Kill on timeout.
|
|
StopGracefully() error
|
|
Wait() error
|
|
Kill() error
|
|
}
|
|
|
|
// processStarter launches a capture process, returning its handle and — when
|
|
// the spec asks for it — its stdout pipe.
|
|
type processStarter func(spec commandSpec) (processHandle, io.ReadCloser, error)
|
|
|
|
// commandOutputRunner runs a short one-shot command to completion and returns
|
|
// its combined output. Used for helper invocations that aren't recordings
|
|
// (Termux start/stop, Windows ffmpeg device listing) and the offline
|
|
// transcription exec — ctx lets a caller cancel/timeout a wedged process.
|
|
type commandOutputRunner func(ctx context.Context, name string, args ...string) ([]byte, error)
|
|
|
|
func startProcess(spec commandSpec) (processHandle, io.ReadCloser, error) {
|
|
cmd := exec.Command(spec.name, spec.args...)
|
|
var stdout io.ReadCloser
|
|
if spec.wantStdout {
|
|
pipe, err := cmd.StdoutPipe()
|
|
if err != nil {
|
|
return nil, nil, err
|
|
}
|
|
stdout = pipe
|
|
}
|
|
var stdin io.WriteCloser
|
|
if spec.stopViaStdin != "" {
|
|
pipe, err := cmd.StdinPipe()
|
|
if err != nil {
|
|
return nil, nil, err
|
|
}
|
|
stdin = pipe
|
|
}
|
|
if err := cmd.Start(); err != nil {
|
|
// StdoutPipe/StdinPipe allocate fds the runtime only reclaims via
|
|
// Wait; on Start failure the process is never established, so close
|
|
// any pipes we already opened to avoid leaking them.
|
|
if stdout != nil {
|
|
_ = stdout.Close()
|
|
}
|
|
if stdin != nil {
|
|
_ = stdin.Close()
|
|
}
|
|
return nil, nil, err
|
|
}
|
|
return &realProcess{cmd: cmd, stdin: stdin, stopText: spec.stopViaStdin}, stdout, nil
|
|
}
|
|
|
|
type realProcess struct {
|
|
cmd *exec.Cmd
|
|
stdin io.WriteCloser
|
|
stopText string
|
|
}
|
|
|
|
func (p *realProcess) StopGracefully() error {
|
|
if p.stopText != "" && p.stdin != nil {
|
|
_, err := io.WriteString(p.stdin, p.stopText)
|
|
closeErr := p.stdin.Close()
|
|
if err != nil {
|
|
return err
|
|
}
|
|
return closeErr
|
|
}
|
|
// os.Interrupt is how arecord/sox finalize their WAV header. Unsupported
|
|
// on Windows, but the only Windows tool (ffmpeg) stops via stdin above.
|
|
return p.cmd.Process.Signal(os.Interrupt)
|
|
}
|
|
|
|
func (p *realProcess) Wait() error { return p.cmd.Wait() }
|
|
|
|
func (p *realProcess) Kill() error { return p.cmd.Process.Kill() }
|
|
|
|
func runCommandOutput(ctx context.Context, name string, args ...string) ([]byte, error) {
|
|
cmd := exec.CommandContext(ctx, name, args...)
|
|
var out bytes.Buffer
|
|
cmd.Stdout = &out
|
|
cmd.Stderr = &out
|
|
err := cmd.Run()
|
|
return out.Bytes(), err
|
|
}
|
|
|
|
// waitWithTimeout waits for the process, killing it if it ignores the graceful
|
|
// stop for longer than the grace window. Capture tools exit on SIGINT/"q"
|
|
// almost instantly; a hang here means a wedged device driver, and a leaked
|
|
// recording process holds the microphone open.
|
|
func waitWithTimeout(proc processHandle, grace time.Duration) error {
|
|
done := make(chan error, 1)
|
|
go func() { done <- proc.Wait() }()
|
|
select {
|
|
case err := <-done:
|
|
return err
|
|
case <-time.After(grace):
|
|
_ = proc.Kill()
|
|
return <-done
|
|
}
|
|
}
|