anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-22 18:25:32 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

[PR #10568] Add OpenAI Realtime API integration with voice support #13488

New Issue
Closed
opened 2026-02-16 18:18:20 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 18:18:20 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/anomalyco/opencode/pull/10568

State: closed
Merged: No

What does this PR do?

This PR adds comprehensive support for OpenAI's Realtime API, enabling true conversational voice interactions in opencode. The implementation includes:

Core Infrastructure:

  • New message part types (AudioPart, RealtimeEventPart) for audio data and VAD events
  • WebSocket protocol schemas for bidirectional audio streaming
  • Provider-agnostic RealtimeTransport abstraction for future multi-provider support
  • Server-side WebSocket endpoint at /realtime/:sessionID/connect

Server Integration:

  • RealtimeSession class bridging client WebSocket ↔ OpenAI Realtime API
  • Automatic tool execution with interruption handling via AbortSignal
  • Transcript and VAD event persistence to session history
  • Session configuration management (voice, VAD mode, temperature, etc.)

Tool Support:

  • Zod v4 → OpenAI JSON Schema converter for existing tools
  • Tool execution in realtime conversations with full error handling
  • Automatic cancellation on user interruption (speech_started VAD event)

Documentation:

  • Architecture overview with message flow diagrams
  • Design decisions (D1-D13) with rationale and trade-offs
  • OpenAI API reference for events and configuration
  • TDD-style implementation roadmap with test coverage

Key Design Decisions:

  • Server maintains WebSocket to OpenAI (protects API keys, enables tool execution)
  • PCM16 @ 24kHz audio format (OpenAI native, lossless)
  • Transcript-only persistence (no raw audio storage by default)
  • Server-side VAD for natural turn-taking
  • Exponential backoff reconnection with jitter

How did you verify your code works?

  • 118 tests across 7 test files covering:

    • Message types and config validation (27 tests)
    • Protocol event serialization/deserialization (47 tests)
    • Transport interface and OpenAI flows (14 tests)
    • Session lifecycle and message routing (15 tests)
    • Tool schema conversion and execution (15 tests)
    • Transcript and VAD event persistence (14 tests)

  • All tests follow TDD approach: schemas validated, transport mocked, session state verified

  • Protocol tests verify round-trip serialization of all OpenAI event types

  • Tool tests validate Zod→JSON Schema conversion and execution with interruption

  • Session tests verify client↔server↔OpenAI message routing and state management

Files Created:

  • src/realtime/ - 6 modules (protocol, transport, session, tools, persistence, index)
  • src/server/routes/realtime.ts - WebSocket and REST endpoints
  • test/realtime/ - 7 test files with 118 tests
  • docs/realtime/ - 8 documentation files

Files Modified:

  • src/session/message-v2.ts - Added AudioPart, RealtimeEventPart, ToolStateInterrupted
  • src/config/config.ts - Added experimental.realtime config schema
  • src/server/server.ts - Mounted realtime routes

Next Steps (Phase 6):

  • Web client: AudioWorklet for PCM16 capture, Web Audio API playback
  • CLI/TUI client: Integration with @mastra/node-audio for terminal voice support
**Original Pull Request:** https://github.com/anomalyco/opencode/pull/10568 **State:** closed **Merged:** No --- ### What does this PR do? This PR adds comprehensive support for OpenAI's Realtime API, enabling true conversational voice interactions in opencode. The implementation includes: **Core Infrastructure:** - New message part types (`AudioPart`, `RealtimeEventPart`) for audio data and VAD events - WebSocket protocol schemas for bidirectional audio streaming - Provider-agnostic `RealtimeTransport` abstraction for future multi-provider support - Server-side WebSocket endpoint at `/realtime/:sessionID/connect` **Server Integration:** - `RealtimeSession` class bridging client WebSocket ↔ OpenAI Realtime API - Automatic tool execution with interruption handling via AbortSignal - Transcript and VAD event persistence to session history - Session configuration management (voice, VAD mode, temperature, etc.) **Tool Support:** - Zod v4 → OpenAI JSON Schema converter for existing tools - Tool execution in realtime conversations with full error handling - Automatic cancellation on user interruption (speech_started VAD event) **Documentation:** - Architecture overview with message flow diagrams - Design decisions (D1-D13) with rationale and trade-offs - OpenAI API reference for events and configuration - TDD-style implementation roadmap with test coverage **Key Design Decisions:** - Server maintains WebSocket to OpenAI (protects API keys, enables tool execution) - PCM16 @ 24kHz audio format (OpenAI native, lossless) - Transcript-only persistence (no raw audio storage by default) - Server-side VAD for natural turn-taking - Exponential backoff reconnection with jitter ### How did you verify your code works? - **118 tests** across 7 test files covering: - Message types and config validation (27 tests) - Protocol event serialization/deserialization (47 tests) - Transport interface and OpenAI flows (14 tests) - Session lifecycle and message routing (15 tests) - Tool schema conversion and execution (15 tests) - Transcript and VAD event persistence (14 tests) - All tests follow TDD approach: schemas validated, transport mocked, session state verified - Protocol tests verify round-trip serialization of all OpenAI event types - Tool tests validate Zod→JSON Schema conversion and execution with interruption - Session tests verify client↔server↔OpenAI message routing and state management **Files Created:** - `src/realtime/` - 6 modules (protocol, transport, session, tools, persistence, index) - `src/server/routes/realtime.ts` - WebSocket and REST endpoints - `test/realtime/` - 7 test files with 118 tests - `docs/realtime/` - 8 documentation files **Files Modified:** - `src/session/message-v2.ts` - Added AudioPart, RealtimeEventPart, ToolStateInterrupted - `src/config/config.ts` - Added experimental.realtime config schema - `src/server/server.ts` - Mounted realtime routes **Next Steps (Phase 6):** - Web client: AudioWorklet for PCM16 capture, Web Audio API playback - CLI/TUI client: Integration with @mastra/node-audio for terminal voice support
yindo added the pull-request label 2026-02-16 18:18:20 -05:00
yindo closed this issue 2026-02-16 18:18:20 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#13488
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?