mirror of
https://github.com/HeyPuter/puter.git
synced 2026-11-13 20:12:57 -10:00
Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 jelveh
|
6b60f7da9c | ||
|
Filip Kujundžić
|
84faf5c40c | ||
|
Rasad Regmi
|
aba2666f26 | ||
|
tgarver-gdm
|
abe6882de9 | ||
|
jelveh
|
770af49528 | ||
|
jelveh
|
47b4cc1158 | ||
|
Nariman Jelveh
|
e253a7f862 |
No files matched your search
@@ -53,7 +53,7 @@ jobs:
|
||||
# step above fails; the tests skip themselves without the key.
|
||||
- name: Run live realtime conversation tests
|
||||
if: ${{ !cancelled() }}
|
||||
run: npm run setupExtensions && (cd src/puter-js && npm run build) && npx vitest run --config src/puter-js/tests/api/vitest.config.ts src/puter-js/tests/api/realtime/convoLive.test.ts
|
||||
run: npm run setupExtensions && (cd src/puter-js && npm run build) && npx vitest run --config src/puter-js/tests/api/vitest.config.ts src/puter-js/tests/api/realtime/realtimeLive.test.ts
|
||||
env:
|
||||
CI: 'true'
|
||||
PUTER_TEST_AI_OPENAI_API_KEY: ${{ secrets.PUTER_TEST_AI_OPENAI_API_KEY }}
|
||||
@@ -149,7 +149,8 @@ so the dialog's own scrim covers the host.
|
||||
|
||||
puter.js uses it on a page one label under the GUI origin when the page is
|
||||
signed in, framing it in a modal `<dialog>` that stays hidden until the embed
|
||||
is ready. The embed posts `{ msg: 'feedbackEmbed', state }` to its parent
|
||||
is ready. A session check that takes longer than 200ms shows a loading scrim
|
||||
with a spinner in its place; Escape or a click on it cancels. The embed posts `{ msg: 'feedbackEmbed', state }` to its parent
|
||||
(`loading`, then `ready` or `unavailable`) and finally
|
||||
`{ msg: 'feedbackDialogClosed', sent }`. It reports `unavailable` without the
|
||||
named user's session, or while that user's email still needs confirming, and
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# OpenAI Realtime relay
|
||||
|
||||
The relay serves `/puterai/openai/v1/realtime` on the API host. The SDK exposes
|
||||
it through [`puter.ai.convo()`](../../../docs/src/AI/convo.md).
|
||||
it through [`puter.ai.realtime()`](../../../docs/src/AI/realtime.md).
|
||||
|
||||
**Beta:** This API is incomplete. Its methods, options, events, and behavior may change abruptly, including breaking changes without notice.
|
||||
|
||||
Direct clients are documented in the [Realtime WebSocket API reference](../../../docs/src/AI/realtime.md).
|
||||
Direct clients are documented in the [Realtime WebSocket API reference](../../../docs/src/AI/realtime-websocket.md).
|
||||
|
||||
## Transport
|
||||
|
||||
@@ -27,7 +27,7 @@ The socket opens before authentication completes. Authentication and model
|
||||
failures arrive as OpenAI-shaped error events followed by a close frame.
|
||||
|
||||
```js
|
||||
const conversation = await puter.ai.convo({ output: ['text'] });
|
||||
const conversation = await puter.ai.realtime({ output: ['text'] });
|
||||
conversation.on('error', (event) => console.error(event));
|
||||
conversation.on('text', ({ delta }) => console.log(delta));
|
||||
conversation.on('turn.done', () => conversation.close());
|
||||
|
||||
+2
-2
@@ -200,8 +200,8 @@ These AI features are supported out of the box when using Puter.js:
|
||||
- **[`puter.ai.img2txt()`](/AI/img2txt/)** - Extract text from images (OCR)
|
||||
- **[`puter.ai.txt2vid()`](/AI/txt2vid/)** - Generate short video clips from text or a reference image with Wan, Seedance, Veo, Kling, MiniMax H3 and other models, or edit a video you supply with Gemini Omni, FLUX 3 or Kling Omni
|
||||
- **[`puter.ai.speech2speech()`](/AI/speech2speech/)** - Convert speech in one voice to another voice
|
||||
- **[`puter.ai.convo()`](/AI/convo/)** - Prepare a beta realtime conversation with optional text, microphone, playback, and interruption helpers
|
||||
- **[Realtime WebSocket API](/AI/realtime/)** - Connect directly without puter.js. This API is in beta and may change abruptly.
|
||||
- **[`puter.ai.realtime()`](/AI/realtime/)** - Prepare a beta realtime conversation with optional text, microphone, playback, and interruption helpers
|
||||
- **[Realtime WebSocket API](/AI/realtime-websocket/)** - Connect directly without puter.js. This API is in beta and may change abruptly.
|
||||
- **[`puter.ai.listModels()`](/AI/listModels/)** - List available AI chat models (and providers) that Puter currently exposes.
|
||||
- **[`puter.ai.listModelProviders()`](/AI/listModelProviders/)** - List the AI providers that Puter currently exposes.
|
||||
- **[`puter.ai.txt2speech.listEngines()`](/AI/txt2speech.listEngines/)** - List available TTS engines/models
|
||||
|
||||
@@ -1,151 +0,0 @@
|
||||
---
|
||||
title: puter.ai.convo()
|
||||
description: Prepare an OpenAI realtime conversation with optional audio and text helpers.
|
||||
platforms: [websites, apps, nodejs, workers]
|
||||
---
|
||||
|
||||
Prepares a realtime OpenAI conversation that your app can connect to when it's ready.
|
||||
|
||||
Preparing the conversation verifies Puter sign-in. Realtime conversations require a paid Puter plan, and usage is billed to the signed-in user. Native OpenAI events and convenience methods use the same session. Microphone capture and speaker playback start when your app requests them.
|
||||
|
||||
In Node.js, the SDK requires a global `WebSocket`, available in Node.js 22 or later.
|
||||
|
||||
**Beta:** This API is incomplete. Its methods, options, events, and behavior may change abruptly, including breaking changes without notice.
|
||||
|
||||
To connect without puter.js, see the [Realtime WebSocket API](/AI/realtime/).
|
||||
|
||||
## Syntax
|
||||
|
||||
```js
|
||||
const conversation = await puter.ai.convo(options = {});
|
||||
await conversation.connect();
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
#### `options` (Object) (Optional)
|
||||
|
||||
The options object accepts these fields.
|
||||
|
||||
| Option | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `model` | `String` | The realtime model. Defaults to `gpt-realtime-2.1-mini`. Also accepts `gpt-realtime-2.1`, `gpt-realtime-2`, `gpt-realtime-1.5`, `gpt-realtime`, and `gpt-realtime-mini`. |
|
||||
| `instructions` | `String` | Session instructions, up to 8192 characters. Defaults to the provider's default. |
|
||||
| `output` | `Array<String>` | Accepts `['text']`, `['audio']`, or both. Defaults to `['audio']`. Audio includes a transcript. Requesting both uses audio output and emits transcript deltas through the `text` event. |
|
||||
| `voice` | `String` | OpenAI voice identifier, up to 128 characters. Defaults to the provider's default. |
|
||||
| `turnDetection` | `Boolean` | Whether the provider detects the end of spoken turns automatically. Defaults to `true`. Set to `false` for manual audio commits. |
|
||||
|
||||
## Return value
|
||||
|
||||
A Promise resolves to a prepared conversation after options and sign-in are checked. In a browser, preparation opens the Puter sign-in prompt if needed. Preparation opens no realtime connection and requests no microphone access.
|
||||
|
||||
The conversation exposes these properties and methods.
|
||||
|
||||
| Member | Behavior |
|
||||
| --- | --- |
|
||||
| `state` | `prepared`, `connecting`, `connected`, or `closed`. |
|
||||
| `model` | Resolved model name. |
|
||||
| `connect()` | Opens the connection and applies settings. Resolves after the provider acknowledges session configuration. Concurrent calls share one connection Promise, including bounded retries for transient setup failures. Calling it while connected opens no additional connection. During a reconnection, it resolves when the reconnection succeeds and rejects if the conversation closes instead. |
|
||||
| `configure(options)` | Updates the given session settings and resolves after the provider acknowledges this update. Settings it isn't given, including ones set through native events, are left alone. Accepts the factory options except `model`. One update can be pending at a time. Provider rules apply to changes such as switching a voice after audio has been generated. |
|
||||
| `sendText(text)` | Adds a user message and requests a response. Text must be nonempty and at most 8192 characters. |
|
||||
| `appendAudio(pcm)` | Appends an `Int16Array` containing PCM16, 24 kHz mono audio. A chunk contains at most 12000 samples. |
|
||||
| `commitAudio()` | Commits buffered audio and requests a response. Requires turn detection to be off, through `turnDetection: false` or a native session update. The provider requires at least 100 ms of buffered audio. |
|
||||
| `interrupt(position?)` | Stops built-in playback immediately and waits for cancellation and playback-history synchronization. A reply requested but not yet started is cancelled as soon as it starts. Custom players provide `{ itemId, contentIndex, playedMs }`. An idle call resolves immediately. Readiness failures reject the returned Promise with `session_not_ready` or `session_closed`. |
|
||||
| `send(event)` | Sends a native OpenAI JSON event. |
|
||||
| `on(type, handler)` | Registers a listener and returns an unsubscribe function. Use `'*'` to receive every native and convenience event. |
|
||||
| `off(type, handler)` | Removes a listener. |
|
||||
| `close()` | Closes the session, rejects pending operations, and releases SDK-owned capture and playback resources. Repeated calls have no additional effect. |
|
||||
|
||||
Attach listeners before connecting to receive initial session events. Closing during connection rejects the pending attempt.
|
||||
|
||||
If the connection drops because of network loss, a server restart or a provider error, the conversation reconnects on its own. Normal closes with code 1000 end the conversation. While it reconnects, `state` is `connecting`, a `reconnecting` event fires, and pending operations reject with `connection_lost`. The new session gets every setting the provider acknowledged, from the factory options, `configure()` and native `session.update` events. Then `ready` fires again with `reconnected: true`. Listeners, the microphone and the speaker carry over. Microphone audio captured while reconnecting is dropped.
|
||||
|
||||
The conversation also restores recent context. Before `ready` fires, it adds back up to 40 of the most recent conversation items, 48 KiB in total. Restored items are user text, user speech that has a transcript, assistant replies as text, and function calls with their outputs. User speech has a transcript only when input transcription is configured through native session settings. An assistant reply that was interrupted, or still playing when the connection dropped, is cut to what the user saw or heard. The cut is estimated from how much of the audio played, so it can be off by a few words. Images, speech without a transcript, deleted items, and replies still generating when the connection dropped are left out. Restored items get new item IDs, don't fire item events, and count as input tokens on the next response. Item IDs from before the reconnect don't work in later events. Restoration uses a contiguous suffix of restorable items. An oversized newest item leaves that suffix empty. Older items left out to fit the window fire a `context_trimmed` event with the count in `omittedItems`; this is routine and not an error. A provider refusal or a failure to send the restored items fires one `error` event with code `context_restore_failed` per restoration. In both cases the conversation stays connected. Items are preserved without truncating text, function arguments or function results.
|
||||
|
||||
When Puter knows ahead of time that the server a session runs on will restart, it sends a `puter.reconnect` event, which also reaches your listeners. Without that notice, a restart drops the connection and the conversation reconnects as above. Within a few seconds, the conversation opens a replacement session in the background while the current one keeps working. It switches once no reply is generating or playing, no user turn is being spoken or transcribed, no function call is waiting for its output, and nothing you sent is waiting on the provider. The switch goes through the same steps as a reconnect, and the `reconnecting` event has code 1012. If the replacement can't open yet, the conversation keeps the current session and tries again later. A refusal that retrying can't fix, such as `insufficient_funds`, stops the attempts. If the server closes the session first, the conversation switches to the replacement, or reconnects if none is open.
|
||||
|
||||
The conversation makes up to ten attempts over about a minute. In a browser that reports being offline, it first waits up to two minutes for the connection to come back. If every attempt fails, or the server refuses with an error such as `insufficient_funds`, the conversation closes. A closed conversation cannot reconnect. To start another session, prepare a new one.
|
||||
|
||||
Native event definitions are listed in the [OpenAI realtime reference](https://developers.openai.com/api/reference/resources/realtime/server-events). Native events retain their provider fields. Sending native events preserves direct control of the session. Puter refuses `conversation.item.retrieve` events and MCP tools (`"type": "mcp"`) in `session.update` and `response.create` events. Sending one fires an `error` event with code `invalid_event`, and the conversation closes with code 1008. Audio helpers expect the session to use PCM16 at 24 kHz.
|
||||
|
||||
## Errors and limits
|
||||
|
||||
Failures use objects containing `code` and `message`. Preparation can reject with `invalid_argument`, `model_not_found`, `auth_required`, `auth_canceled`, or `auth_failed`. Connection can reject with `subscription_required` or `insufficient_funds`, which also show the Puter upgrade prompt. Connection and asynchronous operations can reject with provider errors, `session_closed`, `connection_lost`, `operation_timeout`, or `operation_in_progress`. Sending before readiness, or while reconnecting, throws `session_not_ready`.
|
||||
|
||||
Invalid input throws `invalid_argument`, `invalid_event`, or `input_too_large`. A full SDK send queue throws `queue_full`. Media helpers report `unsupported_environment`, `permission_denied`, `media_error`, `media_timeout`, or `queue_full`. Runtime media errors and `context_restore_failed` arrive through the `error` event. Provider errors also arrive as native `error` events with a nested `error` object.
|
||||
|
||||
Each native event is capped at 64 KiB. The SDK bounds JSON traversal, outgoing queues, acknowledgment waits, microphone work, and playback. All numeric limits are listed on [rate limits and quotas](/rate-limits-and-quotas/).
|
||||
|
||||
## Convenience events
|
||||
|
||||
The conversation emits these events alongside native events.
|
||||
|
||||
| Event | Fields |
|
||||
| --- | --- |
|
||||
| `ready` | `model`, `provider`, and `reconnected: true` after a reconnection |
|
||||
| `reconnecting` | `code`, `reason` of the dropped connection. After a `puter.reconnect` event, `code` is 1012. |
|
||||
| `context_trimmed` | `omittedItems`, the number of restorable items left out of the restored context after a reconnection. Fires before `ready`. |
|
||||
| `text` | `delta`, `responseId` |
|
||||
| `transcript` | `role`, `text`, `final`. Assistant deltas use `final: false`. Completed assistant and user transcripts use `final: true`. User transcripts require transcription to be configured through native session settings. |
|
||||
| `audio` | `pcm`, `sampleRate`, `responseId`, `itemId`, `contentIndex`. The PCM chunk is an `Int16Array`. |
|
||||
| `interrupted` | `source` (`manual` or `speech`), `responseId` |
|
||||
| `turn.done` | `responseId`, native provider `usage`, `status` |
|
||||
| `close` | `code`, `reason` |
|
||||
|
||||
Each event includes its `type`. When the provider event leaves out an ID, `responseId` and `itemId` are empty strings and `contentIndex` is `0`. The `interrupted` event signals local interruption immediately. Its corresponding operation resolves after the provider acknowledges cancellation and any requested truncation. Late audio from a canceled response remains visible as native events and is excluded from convenience audio and built-in playback.
|
||||
|
||||
## Browser media
|
||||
|
||||
The optional microphone and speaker helpers require a browser with Web Audio support. Microphone capture also requires media permissions and a secure context. Start audio from a user action so the browser can allow playback.
|
||||
|
||||
| Member | Behavior |
|
||||
| --- | --- |
|
||||
| `microphone.start()` | Requests microphone access and streams PCM16, resampled to 24 kHz, in 20 ms chunks through an AudioWorklet. If the page or network falls behind, chunks are dropped rather than stopping capture. |
|
||||
| `microphone.mute()` | Disables capture tracks and stops forwarding captured chunks. |
|
||||
| `microphone.unmute()` | Enables capture tracks and forwarding. |
|
||||
| `microphone.stop()` | Stops capture tracks and releases capture resources. A `start()` still waiting on the permission prompt rejects with `session_closed`. |
|
||||
| `speaker.start({ bargeIn: true })` | Starts playback. Barge-in defaults to `true`. With barge-in enabled, provider speech-start events interrupt playback and synchronize played audio with conversation history. Starting the speaker requests no microphone access. |
|
||||
| `speaker.stop()` | Clears playback and releases its audio context. |
|
||||
|
||||
Both start methods require a connected session. Capture and playback are independent, so custom audio input can use the built-in speaker, and built-in capture can use custom playback.
|
||||
|
||||
## Examples
|
||||
|
||||
### Have a text conversation
|
||||
|
||||
A text conversation uses convenience methods for input and streamed output.
|
||||
|
||||
```js
|
||||
const conversation = await puter.ai.convo({
|
||||
output: ['text'],
|
||||
instructions: 'Keep replies short.',
|
||||
});
|
||||
|
||||
conversation.on('text', ({ delta }) => console.log(delta));
|
||||
conversation.on('turn.done', () => conversation.close());
|
||||
conversation.on('error', (event) => console.error(event));
|
||||
|
||||
await conversation.connect();
|
||||
conversation.sendText('Say hello in three words.');
|
||||
```
|
||||
|
||||
### Have a voice conversation
|
||||
|
||||
A voice conversation enables the microphone and speaker after connection.
|
||||
|
||||
```js
|
||||
const conversation = await puter.ai.convo();
|
||||
conversation.on('transcript', ({ role, text }) => console.log(role, text));
|
||||
conversation.on('error', (event) => console.error(event));
|
||||
|
||||
await conversation.connect();
|
||||
await conversation.speaker.start();
|
||||
await conversation.microphone.start();
|
||||
|
||||
// To interrupt a response and submit a replacement text turn:
|
||||
await conversation.interrupt();
|
||||
conversation.sendText('Actually, make it Wednesday.');
|
||||
|
||||
// When the user ends the conversation:
|
||||
conversation.close();
|
||||
```
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
title: Realtime WebSocket API (Beta)
|
||||
description: Connect to OpenAI realtime conversations through Puter without puter.js.
|
||||
platforms: [websites, apps, nodejs, workers]
|
||||
---
|
||||
|
||||
The Realtime WebSocket API lets your client send and receive native OpenAI realtime JSON events through Puter.
|
||||
|
||||
Puter authenticates the user and charges their account for usage.
|
||||
|
||||
**Beta:** This API is incomplete. Its methods, options, events, and behavior may change abruptly, including breaking changes without notice.
|
||||
|
||||
For optional text, microphone, playback, and interruption helpers, use [`puter.ai.realtime()`](/AI/realtime/).
|
||||
|
||||
## Connection and authentication
|
||||
|
||||
Connect to this URL on the Puter API host:
|
||||
|
||||
```text
|
||||
wss://api.puter.com/puterai/openai/v1/realtime?model=gpt-realtime-2.1-mini
|
||||
```
|
||||
|
||||
The `model` query parameter defaults to `gpt-realtime-2.1-mini`. The model is fixed for the connection. A session update that changes it is rejected. It also accepts `gpt-realtime-2.1`, `gpt-realtime-2`, `gpt-realtime-1.5`, `gpt-realtime`, and `gpt-realtime-mini`.
|
||||
|
||||
Authenticate with a Puter session, app, or full-access API token. Puter supplies the provider credential. Server clients pass this header:
|
||||
|
||||
```text
|
||||
Authorization: Bearer <PUTER_TOKEN>
|
||||
```
|
||||
|
||||
Browser WebSocket clients pass `realtime` and `openai-insecure-api-key.<PUTER_TOKEN>` as subprotocols. The server echoes only `realtime`. The endpoint also accepts an `auth_token` query parameter. Prefer the authorization header where supported because URLs and subprotocols can appear in logs. Authentication takes precedence in this order, header, subprotocol, then query parameter.
|
||||
|
||||
The socket can open before authentication and provider setup finish. Listen for errors and closure immediately. Wait for `session.created`, send your session settings, then wait for `session.updated` before submitting a turn.
|
||||
|
||||
To open a replacement while another connection remains healthy, set the optional `purpose=handover` query parameter. These attempts share a smaller per-credential budget, checked before the general connection-attempt budget. Initial connections and dropped-connection recovery use the URL without this parameter. The limits are listed in [rate limits and quotas](/rate-limits-and-quotas/).
|
||||
|
||||
## Text example
|
||||
|
||||
Install the `ws` package and set `PUTER_TOKEN` to your Puter token. Run this example as a Node.js ES module:
|
||||
|
||||
```js
|
||||
import WebSocket from 'ws';
|
||||
|
||||
const socket = new WebSocket(
|
||||
'wss://api.puter.com/puterai/openai/v1/realtime?model=gpt-realtime-2.1-mini',
|
||||
{ headers: { Authorization: `Bearer ${process.env.PUTER_TOKEN}` } },
|
||||
);
|
||||
const send = (event) => socket.send(JSON.stringify(event));
|
||||
|
||||
socket.on('message', (data) => {
|
||||
const event = JSON.parse(data.toString());
|
||||
|
||||
if (event.type === 'session.created') {
|
||||
send({
|
||||
type: 'session.update',
|
||||
session: {
|
||||
type: 'realtime',
|
||||
output_modalities: ['text'],
|
||||
instructions: 'Keep replies short.',
|
||||
},
|
||||
});
|
||||
}
|
||||
if (event.type === 'session.updated') {
|
||||
send({
|
||||
type: 'conversation.item.create',
|
||||
item: {
|
||||
type: 'message',
|
||||
role: 'user',
|
||||
content: [{ type: 'input_text', text: 'Hello!' }],
|
||||
},
|
||||
});
|
||||
send({ type: 'response.create' });
|
||||
}
|
||||
if (event.type === 'response.output_text.delta')
|
||||
process.stdout.write(event.delta);
|
||||
if (event.type === 'error')
|
||||
console.error(event.error);
|
||||
if (event.type === 'response.done')
|
||||
socket.close();
|
||||
});
|
||||
socket.on('error', console.error);
|
||||
socket.on('close', (code, reason) => console.log(code, reason.toString()));
|
||||
```
|
||||
|
||||
## Events, audio, and interruption
|
||||
|
||||
Each message contains one JSON object with a nonempty string `type` of at most 128 characters. Binary messages are rejected. Native events retain their provider fields. The endpoint doesn't deliver `rate_limits.updated`, which describes Puter's provider account rather than yours, and it removes provider organization IDs from error messages. The endpoint exposes the WebSocket event protocol. Your client manages capture, playback, acknowledgments, and timeouts.
|
||||
|
||||
The endpoint refuses `conversation.item.retrieve` events and MCP tools (`"type": "mcp"`) in `session.update` and `response.create` events. A refused event isn't sent to the provider. The session gets an `invalid_event` error and closes with 1008.
|
||||
|
||||
PCM16 audio configured at 24 kHz mono is sent as base64 chunks in `input_audio_buffer.append` events. Output arrives in `response.output_audio.delta` events. With manual turn detection, commit buffered input with `input_audio_buffer.commit`, then request a reply with `response.create`.
|
||||
|
||||
To interrupt, stop local playback immediately and send `response.cancel` with the active `response_id`. If some generated audio was not played, send `conversation.item.truncate` with its `item_id`, `content_index`, and `audio_end_ms` set to the amount actually played. Wait for `response.done` and any requested `conversation.item.truncated` acknowledgment before submitting a replacement turn.
|
||||
|
||||
Connection loss ends the session. Reconnecting creates a new conversation. Tool execution and automatic reconnection belong to your client.
|
||||
|
||||
## Server restarts
|
||||
|
||||
When Puter knows ahead of time that the server a session runs on will restart, it sends each open session this event, at most once a minute:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "puter.reconnect",
|
||||
"event_id": "event_puter_1767225600000",
|
||||
"reason": "server_restarting",
|
||||
"close_in_ms": 90000
|
||||
}
|
||||
```
|
||||
|
||||
The `reason` field is `server_restarting`. The `close_in_ms` field, when present, is about how long until the server closes the session itself. The session keeps working until then. Open a new connection after a short, randomized delay, and once it gets `session.created` and your client is idle, with no response generating and no user turn in progress, move to it and close the old one. A connection that reaches a server that is also restarting gets a `server_restarting` error and closes with 1012. In that case keep the old session and try again after a short, randomized delay. If the server closes the old session first, the close code is 1012, and your client reconnects the same way. A restart can also drop the connection without any notice, so treat an unexpected close the same way.
|
||||
|
||||
A new connection starts an empty conversation. To keep context, add earlier items back with `conversation.item.create` before you request a response. Event types that start with `puter.` come from Puter, not the provider.
|
||||
|
||||
## Errors, billing, and limits
|
||||
|
||||
API errors arrive as `{ "type": "error", "error": { "code": "...", "message": "..." } }`. Handle socket errors and close frames too. Invalid, missing, or revoked credentials use `invalid_api_key`. If Puter can't check a credential or the account's plan, the error is `internal_error` and the session closes with 1011, so try again after a short, randomized delay. Other relay codes include `permission_denied`, `model_not_found`, `insufficient_funds`, `subscription_required`, `rate_limit_exceeded`, `invalid_event`, `connection_timeout`, `provider_unavailable`, `upstream_error`, and `server_restarting`.
|
||||
|
||||
Policy failures generally close with 1008, oversized client messages with 1009, upstream or setup failures with 1011, server restarts with 1012, and temporary server overload with 1013. Reconnect after 1011, 1012, or 1013. Authentication and model failures can arrive after the WebSocket handshake succeeds.
|
||||
|
||||
The endpoint requires a paid Puter plan. Usage is charged after completed responses and input transcriptions. While a response generates, part of your balance is held for it, so a session opens only if your balance covers one response. If the provider connection closes before a response reports its usage, the response is charged the full amount held for it. Each account can open 20 sessions at once by default. Each JSON message you send is limited to 64 KiB, including reassembled fragments. Messages from the provider can be up to 4 MiB. Stream audio in small chunks. All message rates, queue caps, timeouts, and other bounds are published in [rate limits and quotas](/rate-limits-and-quotas/).
|
||||
|
||||
Input transcription holds a small amount of your balance per transcribed item, priced at the transcription model you select. Two items are reserved ahead of your audio and replenished as the provider commits them, so a session holds only its spare items plus the items still awaiting a transcript. Changing the transcription model reprices the spare items. If your balance can't cover another item, audio waits while earlier items settle; once none are left to settle, the session closes with `insufficient_funds`. Failed transcriptions, items without final usage, and audio whose commit the provider never acknowledges before the connection closes are each charged that model's fixed per-item amount. Unacknowledged audio is charged one item per started 160 seconds. With server turn detection, only detected speech counts, so silence is never charged. The amounts, timeout and audio limits are listed in [rate limits and quotas](/rate-limits-and-quotas/).
|
||||
|
||||
Provider-reported realtime models must have a known price. A missing model uses the last known model. An explicitly unpriced model closes the connection with `upstream_error`.
|
||||
|
||||
After recording usage, the connection can continue while settled credit is positive, including a fractional balance. Each new billable operation still needs its full reservation after accounting for other outstanding holds.
|
||||
+128
-103
@@ -1,126 +1,151 @@
|
||||
---
|
||||
title: Realtime WebSocket API (Beta)
|
||||
description: Connect to OpenAI realtime conversations through Puter without puter.js.
|
||||
title: puter.ai.realtime()
|
||||
description: Prepare an OpenAI realtime conversation with optional audio and text helpers.
|
||||
platforms: [websites, apps, nodejs, workers]
|
||||
---
|
||||
|
||||
The Realtime WebSocket API lets your client send and receive native OpenAI realtime JSON events through Puter.
|
||||
Prepares a realtime OpenAI conversation that your app can connect to when it's ready.
|
||||
|
||||
Puter authenticates the user and charges their account for usage.
|
||||
Preparing the conversation verifies Puter sign-in. Realtime conversations require a paid Puter plan, and usage is billed to the signed-in user. Native OpenAI events and convenience methods use the same session. Microphone capture and speaker playback start when your app requests them.
|
||||
|
||||
In Node.js, the SDK requires a global `WebSocket`, available in Node.js 22 or later.
|
||||
|
||||
**Beta:** This API is incomplete. Its methods, options, events, and behavior may change abruptly, including breaking changes without notice.
|
||||
|
||||
For optional text, microphone, playback, and interruption helpers, use [`puter.ai.convo()`](/AI/convo/).
|
||||
To connect without puter.js, see the [Realtime WebSocket API](/AI/realtime-websocket/).
|
||||
|
||||
## Connection and authentication
|
||||
|
||||
Connect to this URL on the Puter API host:
|
||||
|
||||
```text
|
||||
wss://api.puter.com/puterai/openai/v1/realtime?model=gpt-realtime-2.1-mini
|
||||
```
|
||||
|
||||
The `model` query parameter defaults to `gpt-realtime-2.1-mini`. The model is fixed for the connection. A session update that changes it is rejected. It also accepts `gpt-realtime-2.1`, `gpt-realtime-2`, `gpt-realtime-1.5`, `gpt-realtime`, and `gpt-realtime-mini`.
|
||||
|
||||
Authenticate with a Puter session, app, or full-access API token. Puter supplies the provider credential. Server clients pass this header:
|
||||
|
||||
```text
|
||||
Authorization: Bearer <PUTER_TOKEN>
|
||||
```
|
||||
|
||||
Browser WebSocket clients pass `realtime` and `openai-insecure-api-key.<PUTER_TOKEN>` as subprotocols. The server echoes only `realtime`. The endpoint also accepts an `auth_token` query parameter. Prefer the authorization header where supported because URLs and subprotocols can appear in logs. Authentication takes precedence in this order, header, subprotocol, then query parameter.
|
||||
|
||||
The socket can open before authentication and provider setup finish. Listen for errors and closure immediately. Wait for `session.created`, send your session settings, then wait for `session.updated` before submitting a turn.
|
||||
|
||||
To open a replacement while another connection remains healthy, set the optional `purpose=handover` query parameter. These attempts share a smaller per-credential budget, checked before the general connection-attempt budget. Initial connections and dropped-connection recovery use the URL without this parameter. The limits are listed in [rate limits and quotas](/rate-limits-and-quotas/).
|
||||
|
||||
## Text example
|
||||
|
||||
Install the `ws` package and set `PUTER_TOKEN` to your Puter token. Run this example as a Node.js ES module:
|
||||
## Syntax
|
||||
|
||||
```js
|
||||
import WebSocket from 'ws';
|
||||
const conversation = await puter.ai.realtime(options = {});
|
||||
await conversation.connect();
|
||||
```
|
||||
|
||||
const socket = new WebSocket(
|
||||
'wss://api.puter.com/puterai/openai/v1/realtime?model=gpt-realtime-2.1-mini',
|
||||
{ headers: { Authorization: `Bearer ${process.env.PUTER_TOKEN}` } },
|
||||
);
|
||||
const send = (event) => socket.send(JSON.stringify(event));
|
||||
## Parameters
|
||||
|
||||
socket.on('message', (data) => {
|
||||
const event = JSON.parse(data.toString());
|
||||
#### `options` (Object) (Optional)
|
||||
|
||||
if (event.type === 'session.created') {
|
||||
send({
|
||||
type: 'session.update',
|
||||
session: {
|
||||
type: 'realtime',
|
||||
output_modalities: ['text'],
|
||||
instructions: 'Keep replies short.',
|
||||
},
|
||||
});
|
||||
}
|
||||
if (event.type === 'session.updated') {
|
||||
send({
|
||||
type: 'conversation.item.create',
|
||||
item: {
|
||||
type: 'message',
|
||||
role: 'user',
|
||||
content: [{ type: 'input_text', text: 'Hello!' }],
|
||||
},
|
||||
});
|
||||
send({ type: 'response.create' });
|
||||
}
|
||||
if (event.type === 'response.output_text.delta')
|
||||
process.stdout.write(event.delta);
|
||||
if (event.type === 'error')
|
||||
console.error(event.error);
|
||||
if (event.type === 'response.done')
|
||||
socket.close();
|
||||
The options object accepts these fields.
|
||||
|
||||
| Option | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `model` | `String` | The realtime model. Defaults to `gpt-realtime-2.1-mini`. Also accepts `gpt-realtime-2.1`, `gpt-realtime-2`, `gpt-realtime-1.5`, `gpt-realtime`, and `gpt-realtime-mini`. |
|
||||
| `instructions` | `String` | Session instructions, up to 8192 characters. Defaults to the provider's default. |
|
||||
| `output` | `Array<String>` | Accepts `['text']`, `['audio']`, or both. Defaults to `['audio']`. Audio includes a transcript. Requesting both uses audio output and emits transcript deltas through the `text` event. |
|
||||
| `voice` | `String` | OpenAI voice identifier, up to 128 characters. Defaults to the provider's default. |
|
||||
| `turnDetection` | `Boolean` | Whether the provider detects the end of spoken turns automatically. Defaults to `true`. Set to `false` for manual audio commits. |
|
||||
|
||||
## Return value
|
||||
|
||||
A Promise resolves to a prepared conversation after options and sign-in are checked. In a browser, preparation opens the Puter sign-in prompt if needed. Preparation opens no realtime connection and requests no microphone access.
|
||||
|
||||
The conversation exposes these properties and methods.
|
||||
|
||||
| Member | Behavior |
|
||||
| --- | --- |
|
||||
| `state` | `prepared`, `connecting`, `connected`, or `closed`. |
|
||||
| `model` | Resolved model name. |
|
||||
| `connect()` | Opens the connection and applies settings. Resolves after the provider acknowledges session configuration. Concurrent calls share one connection Promise, including bounded retries for transient setup failures. Calling it while connected opens no additional connection. During a reconnection, it resolves when the reconnection succeeds and rejects if the conversation closes instead. |
|
||||
| `configure(options)` | Updates the given session settings and resolves after the provider acknowledges this update. Settings it isn't given, including ones set through native events, are left alone. Accepts the factory options except `model`. One update can be pending at a time. Provider rules apply to changes such as switching a voice after audio has been generated. |
|
||||
| `sendText(text)` | Adds a user message and requests a response. Text must be nonempty and at most 8192 characters. |
|
||||
| `appendAudio(pcm)` | Appends an `Int16Array` containing PCM16, 24 kHz mono audio. A chunk contains at most 12000 samples. |
|
||||
| `commitAudio()` | Commits buffered audio and requests a response. Requires turn detection to be off, through `turnDetection: false` or a native session update. The provider requires at least 100 ms of buffered audio. |
|
||||
| `interrupt(position?)` | Stops built-in playback immediately and waits for cancellation and playback-history synchronization. A reply requested but not yet started is cancelled as soon as it starts. Custom players provide `{ itemId, contentIndex, playedMs }`. An idle call resolves immediately. Readiness failures reject the returned Promise with `session_not_ready` or `session_closed`. |
|
||||
| `send(event)` | Sends a native OpenAI JSON event. |
|
||||
| `on(type, handler)` | Registers a listener and returns an unsubscribe function. Use `'*'` to receive every native and convenience event. |
|
||||
| `off(type, handler)` | Removes a listener. |
|
||||
| `close()` | Closes the session, rejects pending operations, and releases SDK-owned capture and playback resources. Repeated calls have no additional effect. |
|
||||
|
||||
Attach listeners before connecting to receive initial session events. Closing during connection rejects the pending attempt.
|
||||
|
||||
If the connection drops because of network loss, a server restart or a provider error, the conversation reconnects on its own. Normal closes with code 1000 end the conversation. While it reconnects, `state` is `connecting`, a `reconnecting` event fires, and pending operations reject with `connection_lost`. The new session gets every setting the provider acknowledged, from the factory options, `configure()` and native `session.update` events. Then `ready` fires again with `reconnected: true`. Listeners, the microphone and the speaker carry over. Microphone audio captured while reconnecting is dropped.
|
||||
|
||||
The conversation also restores recent context. Before `ready` fires, it adds back up to 40 of the most recent conversation items, 48 KiB in total. Restored items are user text, user speech that has a transcript, assistant replies as text, and function calls with their outputs. User speech has a transcript only when input transcription is configured through native session settings. An assistant reply that was interrupted, or still playing when the connection dropped, is cut to what the user saw or heard. The cut is estimated from how much of the audio played, so it can be off by a few words. Images, speech without a transcript, deleted items, and replies still generating when the connection dropped are left out. Restored items get new item IDs, don't fire item events, and count as input tokens on the next response. Item IDs from before the reconnect don't work in later events. Restoration uses a contiguous suffix of restorable items. An oversized newest item leaves that suffix empty. Older items left out to fit the window fire a `context_trimmed` event with the count in `omittedItems`; this is routine and not an error. A provider refusal or a failure to send the restored items fires one `error` event with code `context_restore_failed` per restoration. In both cases the conversation stays connected. Items are preserved without truncating text, function arguments or function results.
|
||||
|
||||
When Puter knows ahead of time that the server a session runs on will restart, it sends a `puter.reconnect` event, which also reaches your listeners. Without that notice, a restart drops the connection and the conversation reconnects as above. Within a few seconds, the conversation opens a replacement session in the background while the current one keeps working. It switches once no reply is generating or playing, no user turn is being spoken or transcribed, no function call is waiting for its output, and nothing you sent is waiting on the provider. The switch goes through the same steps as a reconnect, and the `reconnecting` event has code 1012. If the replacement can't open yet, the conversation keeps the current session and tries again later. A refusal that retrying can't fix, such as `insufficient_funds`, stops the attempts. If the server closes the session first, the conversation switches to the replacement, or reconnects if none is open.
|
||||
|
||||
The conversation makes up to ten attempts over about a minute. In a browser that reports being offline, it first waits up to two minutes for the connection to come back. If every attempt fails, or the server refuses with an error such as `insufficient_funds`, the conversation closes. A closed conversation cannot reconnect. To start another session, prepare a new one.
|
||||
|
||||
Native event definitions are listed in the [OpenAI realtime reference](https://developers.openai.com/api/reference/resources/realtime/server-events). Native events retain their provider fields. Sending native events preserves direct control of the session. Puter refuses `conversation.item.retrieve` events and MCP tools (`"type": "mcp"`) in `session.update` and `response.create` events. Sending one fires an `error` event with code `invalid_event`, and the conversation closes with code 1008. Audio helpers expect the session to use PCM16 at 24 kHz.
|
||||
|
||||
## Errors and limits
|
||||
|
||||
Failures use objects containing `code` and `message`. Preparation can reject with `invalid_argument`, `model_not_found`, `auth_required`, `auth_canceled`, or `auth_failed`. Connection can reject with `subscription_required` or `insufficient_funds`, which also show the Puter upgrade prompt. Connection and asynchronous operations can reject with provider errors, `session_closed`, `connection_lost`, `operation_timeout`, or `operation_in_progress`. Sending before readiness, or while reconnecting, throws `session_not_ready`.
|
||||
|
||||
Invalid input throws `invalid_argument`, `invalid_event`, or `input_too_large`. A full SDK send queue throws `queue_full`. Media helpers report `unsupported_environment`, `permission_denied`, `media_error`, `media_timeout`, or `queue_full`. Runtime media errors and `context_restore_failed` arrive through the `error` event. Provider errors also arrive as native `error` events with a nested `error` object.
|
||||
|
||||
Each native event is capped at 64 KiB. The SDK bounds JSON traversal, outgoing queues, acknowledgment waits, microphone work, and playback. All numeric limits are listed on [rate limits and quotas](/rate-limits-and-quotas/).
|
||||
|
||||
## Convenience events
|
||||
|
||||
The conversation emits these events alongside native events.
|
||||
|
||||
| Event | Fields |
|
||||
| --- | --- |
|
||||
| `ready` | `model`, `provider`, and `reconnected: true` after a reconnection |
|
||||
| `reconnecting` | `code`, `reason` of the dropped connection. After a `puter.reconnect` event, `code` is 1012. |
|
||||
| `context_trimmed` | `omittedItems`, the number of restorable items left out of the restored context after a reconnection. Fires before `ready`. |
|
||||
| `text` | `delta`, `responseId` |
|
||||
| `transcript` | `role`, `text`, `final`. Assistant deltas use `final: false`. Completed assistant and user transcripts use `final: true`. User transcripts require transcription to be configured through native session settings. |
|
||||
| `audio` | `pcm`, `sampleRate`, `responseId`, `itemId`, `contentIndex`. The PCM chunk is an `Int16Array`. |
|
||||
| `interrupted` | `source` (`manual` or `speech`), `responseId` |
|
||||
| `turn.done` | `responseId`, native provider `usage`, `status` |
|
||||
| `close` | `code`, `reason` |
|
||||
|
||||
Each event includes its `type`. When the provider event leaves out an ID, `responseId` and `itemId` are empty strings and `contentIndex` is `0`. The `interrupted` event signals local interruption immediately. Its corresponding operation resolves after the provider acknowledges cancellation and any requested truncation. Late audio from a canceled response remains visible as native events and is excluded from convenience audio and built-in playback.
|
||||
|
||||
## Browser media
|
||||
|
||||
The optional microphone and speaker helpers require a browser with Web Audio support. Microphone capture also requires media permissions and a secure context. Start audio from a user action so the browser can allow playback.
|
||||
|
||||
| Member | Behavior |
|
||||
| --- | --- |
|
||||
| `microphone.start()` | Requests microphone access and streams PCM16, resampled to 24 kHz, in 20 ms chunks through an AudioWorklet. If the page or network falls behind, chunks are dropped rather than stopping capture. |
|
||||
| `microphone.mute()` | Disables capture tracks and stops forwarding captured chunks. |
|
||||
| `microphone.unmute()` | Enables capture tracks and forwarding. |
|
||||
| `microphone.stop()` | Stops capture tracks and releases capture resources. A `start()` still waiting on the permission prompt rejects with `session_closed`. |
|
||||
| `speaker.start({ bargeIn: true })` | Starts playback. Barge-in defaults to `true`. With barge-in enabled, provider speech-start events interrupt playback and synchronize played audio with conversation history. Starting the speaker requests no microphone access. |
|
||||
| `speaker.stop()` | Clears playback and releases its audio context. |
|
||||
|
||||
Both start methods require a connected session. Capture and playback are independent, so custom audio input can use the built-in speaker, and built-in capture can use custom playback.
|
||||
|
||||
## Examples
|
||||
|
||||
### Have a text conversation
|
||||
|
||||
A text conversation uses convenience methods for input and streamed output.
|
||||
|
||||
```js
|
||||
const conversation = await puter.ai.realtime({
|
||||
output: ['text'],
|
||||
instructions: 'Keep replies short.',
|
||||
});
|
||||
socket.on('error', console.error);
|
||||
socket.on('close', (code, reason) => console.log(code, reason.toString()));
|
||||
|
||||
conversation.on('text', ({ delta }) => console.log(delta));
|
||||
conversation.on('turn.done', () => conversation.close());
|
||||
conversation.on('error', (event) => console.error(event));
|
||||
|
||||
await conversation.connect();
|
||||
conversation.sendText('Say hello in three words.');
|
||||
```
|
||||
|
||||
## Events, audio, and interruption
|
||||
### Have a voice conversation
|
||||
|
||||
Each message contains one JSON object with a nonempty string `type` of at most 128 characters. Binary messages are rejected. Native events retain their provider fields. The endpoint doesn't deliver `rate_limits.updated`, which describes Puter's provider account rather than yours, and it removes provider organization IDs from error messages. The endpoint exposes the WebSocket event protocol. Your client manages capture, playback, acknowledgments, and timeouts.
|
||||
A voice conversation enables the microphone and speaker after connection.
|
||||
|
||||
The endpoint refuses `conversation.item.retrieve` events and MCP tools (`"type": "mcp"`) in `session.update` and `response.create` events. A refused event isn't sent to the provider. The session gets an `invalid_event` error and closes with 1008.
|
||||
```js
|
||||
const conversation = await puter.ai.realtime();
|
||||
conversation.on('transcript', ({ role, text }) => console.log(role, text));
|
||||
conversation.on('error', (event) => console.error(event));
|
||||
|
||||
PCM16 audio configured at 24 kHz mono is sent as base64 chunks in `input_audio_buffer.append` events. Output arrives in `response.output_audio.delta` events. With manual turn detection, commit buffered input with `input_audio_buffer.commit`, then request a reply with `response.create`.
|
||||
await conversation.connect();
|
||||
await conversation.speaker.start();
|
||||
await conversation.microphone.start();
|
||||
|
||||
To interrupt, stop local playback immediately and send `response.cancel` with the active `response_id`. If some generated audio was not played, send `conversation.item.truncate` with its `item_id`, `content_index`, and `audio_end_ms` set to the amount actually played. Wait for `response.done` and any requested `conversation.item.truncated` acknowledgment before submitting a replacement turn.
|
||||
// To interrupt a response and submit a replacement text turn:
|
||||
await conversation.interrupt();
|
||||
conversation.sendText('Actually, make it Wednesday.');
|
||||
|
||||
Connection loss ends the session. Reconnecting creates a new conversation. Tool execution and automatic reconnection belong to your client.
|
||||
|
||||
## Server restarts
|
||||
|
||||
When Puter knows ahead of time that the server a session runs on will restart, it sends each open session this event, at most once a minute:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "puter.reconnect",
|
||||
"event_id": "event_puter_1767225600000",
|
||||
"reason": "server_restarting",
|
||||
"close_in_ms": 90000
|
||||
}
|
||||
// When the user ends the conversation:
|
||||
conversation.close();
|
||||
```
|
||||
|
||||
The `reason` field is `server_restarting`. The `close_in_ms` field, when present, is about how long until the server closes the session itself. The session keeps working until then. Open a new connection after a short, randomized delay, and once it gets `session.created` and your client is idle, with no response generating and no user turn in progress, move to it and close the old one. A connection that reaches a server that is also restarting gets a `server_restarting` error and closes with 1012. In that case keep the old session and try again after a short, randomized delay. If the server closes the old session first, the close code is 1012, and your client reconnects the same way. A restart can also drop the connection without any notice, so treat an unexpected close the same way.
|
||||
|
||||
A new connection starts an empty conversation. To keep context, add earlier items back with `conversation.item.create` before you request a response. Event types that start with `puter.` come from Puter, not the provider.
|
||||
|
||||
## Errors, billing, and limits
|
||||
|
||||
API errors arrive as `{ "type": "error", "error": { "code": "...", "message": "..." } }`. Handle socket errors and close frames too. Invalid, missing, or revoked credentials use `invalid_api_key`. If Puter can't check a credential or the account's plan, the error is `internal_error` and the session closes with 1011, so try again after a short, randomized delay. Other relay codes include `permission_denied`, `model_not_found`, `insufficient_funds`, `subscription_required`, `rate_limit_exceeded`, `invalid_event`, `connection_timeout`, `provider_unavailable`, `upstream_error`, and `server_restarting`.
|
||||
|
||||
Policy failures generally close with 1008, oversized client messages with 1009, upstream or setup failures with 1011, server restarts with 1012, and temporary server overload with 1013. Reconnect after 1011, 1012, or 1013. Authentication and model failures can arrive after the WebSocket handshake succeeds.
|
||||
|
||||
The endpoint requires a paid Puter plan. Usage is charged after completed responses and input transcriptions. While a response generates, part of your balance is held for it, so a session opens only if your balance covers one response. If the provider connection closes before a response reports its usage, the response is charged the full amount held for it. Each account can open 20 sessions at once by default. Each JSON message you send is limited to 64 KiB, including reassembled fragments. Messages from the provider can be up to 4 MiB. Stream audio in small chunks. All message rates, queue caps, timeouts, and other bounds are published in [rate limits and quotas](/rate-limits-and-quotas/).
|
||||
|
||||
Input transcription holds a small amount of your balance per transcribed item, priced at the transcription model you select. Two items are reserved ahead of your audio and replenished as the provider commits them, so a session holds only its spare items plus the items still awaiting a transcript. Changing the transcription model reprices the spare items. If your balance can't cover another item, audio waits while earlier items settle; once none are left to settle, the session closes with `insufficient_funds`. Failed transcriptions, items without final usage, and audio whose commit the provider never acknowledges before the connection closes are each charged that model's fixed per-item amount. Unacknowledged audio is charged one item per started 160 seconds. With server turn detection, only detected speech counts, so silence is never charged. The amounts, timeout and audio limits are listed in [rate limits and quotas](/rate-limits-and-quotas/).
|
||||
|
||||
Provider-reported realtime models must have a known price. A missing model uses the last known model. An explicitly unpriced model closes the connection with `upstream_error`.
|
||||
|
||||
After recording usage, the connection can continue while settled credit is positive, including a fractional balance. Each new billable operation still needs its full reservation after accounting for other outstanding holds.
|
||||
@@ -91,7 +91,7 @@ Gemini speech-to-text first counts the audio's tokens, which times out after 30
|
||||
|
||||
The OpenAI- and Anthropic-compatible endpoints (`/puterai/openai/v1/*`, `/puterai/anthropic/v1/messages`) require a paid plan; a free account gets `402 subscription_required`. The same models are available to every account through `puter.ai.*` and `/drivers/call`, and the model catalogue endpoints are open to everyone.
|
||||
|
||||
Realtime sessions ([`puter.ai.convo()`](/AI/convo/), `/puterai/openai/v1/realtime`) require a paid plan, like the other OpenAI-compatible endpoints, and are capped at 20 open at once per account. An account without a plan gets an `error` event with code `subscription_required`. A session over the cap gets `rate_limit_exceeded`. Both close with 1008.
|
||||
Realtime sessions ([`puter.ai.realtime()`](/AI/realtime/), `/puterai/openai/v1/realtime`) require a paid plan, like the other OpenAI-compatible endpoints, and are capped at 20 open at once per account. An account without a plan gets an `error` event with code `subscription_required`. A session over the cap gets `rate_limit_exceeded`. Both close with 1008.
|
||||
|
||||
| Realtime transport limit | Value | Behavior at the limit |
|
||||
| --- | --- | --- |
|
||||
@@ -131,7 +131,7 @@ Realtime sessions ([`puter.ai.convo()`](/AI/convo/), `/puterai/openai/v1/realtim
|
||||
|
||||
Send audio as a stream of small chunks. The message cap applies to text, configuration, tool output, and audio alike.
|
||||
|
||||
The [`puter.ai.convo()`](/AI/convo/) SDK also applies these limits before sending or retaining data.
|
||||
The [`puter.ai.realtime()`](/AI/realtime/) SDK also applies these limits before sending or retaining data.
|
||||
|
||||
| Conversation SDK limit | Value | Behavior at the limit |
|
||||
| --- | --- | --- |
|
||||
|
||||
@@ -124,17 +124,17 @@ let sidebar = [
|
||||
path: '/AI/speech2speech',
|
||||
},
|
||||
{
|
||||
title: '<code>convo()</code>',
|
||||
page_title: '<code>puter.ai.convo()</code>',
|
||||
title_tag: 'puter.ai.convo()',
|
||||
title: '<code>realtime()</code>',
|
||||
page_title: '<code>puter.ai.realtime()</code>',
|
||||
title_tag: 'puter.ai.realtime()',
|
||||
icon: '/assets/img/function.svg',
|
||||
source: '/AI/convo.md',
|
||||
path: '/AI/convo',
|
||||
source: '/AI/realtime.md',
|
||||
path: '/AI/realtime',
|
||||
},
|
||||
{
|
||||
title: 'Realtime WebSocket API (Beta)',
|
||||
source: '/AI/realtime.md',
|
||||
path: '/AI/realtime',
|
||||
source: '/AI/realtime-websocket.md',
|
||||
path: '/AI/realtime-websocket',
|
||||
},
|
||||
{
|
||||
title: '<code>listModels()</code>',
|
||||
|
||||
@@ -12,4 +12,6 @@ body {
|
||||
/* A scroll over the scrim must not scroll the framing page behind it. */
|
||||
html {
|
||||
overscroll-behavior: none;
|
||||
/* Matches the frame's, so it stays transparent and follows the OS theme. */
|
||||
color-scheme: light dark;
|
||||
}
|
||||
@@ -181,6 +181,20 @@ input[type=text], input[type=password], input[type=email], select {
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
@media (prefers-color-scheme: dark) {
|
||||
input[type=text], input[type=password], input[type=email], select {
|
||||
background-color: #2d2d2d;
|
||||
border: 1px solid #555;
|
||||
color: #e6e6e6;
|
||||
}
|
||||
|
||||
input[type=text]::placeholder,
|
||||
input[type=password]::placeholder,
|
||||
input[type=email]::placeholder {
|
||||
color: #999;
|
||||
}
|
||||
}
|
||||
|
||||
/* to prevent auto-zoom on input focus in mobile */
|
||||
.device-phone input[type=text], .device-phone input[type=password], .device-phone input[type=email], .device-phone select {
|
||||
font-size: 17px;
|
||||
|
||||
@@ -6,6 +6,7 @@ import { mkdtemp, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { inflateSync } from 'node:zlib';
|
||||
|
||||
let browser;
|
||||
let server;
|
||||
@@ -74,7 +75,7 @@ beforeAll(async () => {
|
||||
});
|
||||
</script>
|
||||
<iframe referrerpolicy="origin" title="Send feedback" src="${origin}/embed/feedback#user=${req.query.user ?? ''}"
|
||||
style="display:block;border:0;width:100vw;height:100vh;color-scheme:normal"></iframe>
|
||||
style="display:block;border:0;width:100vw;height:100vh;color-scheme:light dark"></iframe>
|
||||
<div style="height:3000px"></div>
|
||||
</body></html>`));
|
||||
app.get('/embed/feedback', (_req, res) => {
|
||||
@@ -102,13 +103,26 @@ afterAll(async () => {
|
||||
});
|
||||
|
||||
/** A browser whose GUI origin is signed in with `token`, if any. */
|
||||
const signedIn = token => browser.newContext({ storageState: {
|
||||
const signedIn = (token, options = {}) => browser.newContext({ ...options, storageState: {
|
||||
cookies: [],
|
||||
origins: token ? [{ origin, localStorage: [{ name: 'auth_token_v2', value: token }] }] : [],
|
||||
} });
|
||||
|
||||
const received = page => page.evaluate(() => window.received);
|
||||
|
||||
/** The [r, g, b] of one pixel of the page, from a screenshot. */
|
||||
const pixel = async (page, x, y) => {
|
||||
const png = await page.screenshot({ clip: { x, y, width: 1, height: 1 }, animations: 'disabled' });
|
||||
const data = [];
|
||||
for ( let at = 8; at < png.length; ) {
|
||||
const length = png.readUInt32BE(at);
|
||||
if ( png.toString('ascii', at + 4, at + 8) === 'IDAT' ) data.push(png.subarray(at + 8, at + 8 + length));
|
||||
at += 12 + length;
|
||||
}
|
||||
// One pixel: a filter byte, then its channels unchanged.
|
||||
return [...inflateSync(Buffer.concat(data)).subarray(1, 4)];
|
||||
};
|
||||
|
||||
describe('feedback embed', () => {
|
||||
it('sends feedback for the framing page as the user it names, then tells it', async () => {
|
||||
const context = await signedIn(sessionToken);
|
||||
@@ -162,6 +176,26 @@ describe('feedback embed', () => {
|
||||
}
|
||||
}, 30000);
|
||||
|
||||
it.each(['light', 'dark'])('lets the page show through under the scrim in %s mode', async (colorScheme) => {
|
||||
const context = await signedIn(sessionToken, { colorScheme });
|
||||
try {
|
||||
const page = await context.newPage();
|
||||
page.setDefaultTimeout(6000);
|
||||
await page.goto(`${hostOrigin}/host?user=${sessionUser}`);
|
||||
const frame = page.frameLocator('iframe');
|
||||
await frame.locator('.app-feedback-form').waitFor();
|
||||
expect(await frame.locator('body').evaluate(() => matchMedia('(prefers-color-scheme: dark)').matches))
|
||||
.toBe(colorScheme === 'dark');
|
||||
await page.waitForTimeout(300);
|
||||
// The red page, dimmed by the scrim, rather than an opaque frame.
|
||||
const [red, green, blue] = await pixel(page, 10, 10);
|
||||
expect(red).toBeGreaterThan(green + 40);
|
||||
expect(green).toBe(blue);
|
||||
} finally {
|
||||
await context.close();
|
||||
}
|
||||
}, 30000);
|
||||
|
||||
it('keeps the page behind from scrolling under the scrim', async () => {
|
||||
const context = await signedIn(sessionToken);
|
||||
try {
|
||||
|
||||
Vendored
+13
-13
@@ -44,17 +44,17 @@ export type {
|
||||
ListTTSEnginesOptions,
|
||||
ListTTSVoicesOptions,
|
||||
OcrAnnotationFormat,
|
||||
ConvoOptions,
|
||||
ConvoTextEvent,
|
||||
ConvoAudioEvent,
|
||||
ConvoTranscriptEvent,
|
||||
ConvoInterruptedEvent,
|
||||
ConvoTurnDoneEvent,
|
||||
ConvoErrorEvent,
|
||||
ConvoEventMap,
|
||||
ConvoSessionOptions,
|
||||
ConvoPlaybackPosition,
|
||||
ConvoSpeakerOptions,
|
||||
RealtimeOptions,
|
||||
RealtimeTextEvent,
|
||||
RealtimeAudioEvent,
|
||||
RealtimeTranscriptEvent,
|
||||
RealtimeInterruptedEvent,
|
||||
RealtimeTurnDoneEvent,
|
||||
RealtimeErrorEvent,
|
||||
RealtimeEventMap,
|
||||
RealtimeSessionOptions,
|
||||
RealtimePlaybackPosition,
|
||||
RealtimeSpeakerOptions,
|
||||
RealtimeServerEvent,
|
||||
RealtimeClientEvent,
|
||||
Speech2SpeechOptions,
|
||||
@@ -391,5 +391,5 @@ export type Teams = InstanceType<import('./types/modules/teams/index.js').TeamsC
|
||||
export type UI = InstanceType<import('./types/modules/UI.js').UIConstructor>;
|
||||
export type WorkersHandler = InstanceType<import('./types/modules/Workers.js').WorkersConstructor>;
|
||||
|
||||
export type { Conversation, ConvoError } from './types/modules/ai/convo.js';
|
||||
export type { ConvoMicrophone, ConvoSpeaker } from './types/modules/ai/convoMedia.js';
|
||||
export type { Conversation, RealtimeError } from './types/modules/ai/realtime.js';
|
||||
export type { RealtimeMicrophone, RealtimeSpeaker } from './types/modules/ai/realtimeMedia.js';
|
||||
@@ -8,6 +8,21 @@
|
||||
const LOAD_GRACE_MS = 1000;
|
||||
// How long the embed may take to check the session before the popup is used.
|
||||
const READY_TIMEOUT_MS = 15_000;
|
||||
// A session check slower than this shows a loading scrim over the page.
|
||||
const LOADING_DELAY_MS = 200;
|
||||
|
||||
// The loading scrim matches the embed's own, which fades in over it once ready.
|
||||
const STYLE = `
|
||||
dialog.puter-feedback-embed::backdrop{background:transparent}
|
||||
dialog.puter-feedback-embed:focus{outline:none}
|
||||
.puter-feedback-embed-loading{position:absolute;inset:0;display:flex;align-items:center;justify-content:center;background:rgba(0,0,0,.5);opacity:0;pointer-events:none;transition:opacity .16s ease-in-out}
|
||||
.puter-feedback-embed.is-loading .puter-feedback-embed-loading{opacity:1;pointer-events:auto}
|
||||
.puter-feedback-embed.is-loading iframe{visibility:hidden}
|
||||
.puter-feedback-embed-loading span{width:28px;height:28px;box-sizing:border-box;border:3px solid rgba(255,255,255,.3);border-top-color:#fff;border-radius:50%;animation:puter-feedback-embed-spin .7s linear infinite}
|
||||
@keyframes puter-feedback-embed-spin{to{transform:rotate(360deg)}}
|
||||
@media (prefers-color-scheme:dark){.puter-feedback-embed-loading{background:rgba(0,0,0,.66)}}
|
||||
@media (prefers-reduced-motion:reduce){.puter-feedback-embed-loading{transition:none}.puter-feedback-embed-loading span{animation-duration:1.6s}}
|
||||
`;
|
||||
|
||||
/**
|
||||
* Whether a page at `pageUrl` may frame an embed served by `guiOrigin`:
|
||||
@@ -35,8 +50,9 @@ export const canFrameGuiEmbed = (pageUrl, guiOrigin) => {
|
||||
|
||||
/**
|
||||
* Shows the app feedback dialog from `/embed/feedback` in a modal frame over
|
||||
* the page, as the user named. Resolves `{ sent }` once the dialog closes, or
|
||||
* null without showing anything when the embed can't serve this user (no
|
||||
* the page, as the user named, behind a loading scrim while a slow session
|
||||
* check runs. Resolves `{ sent }` once the dialog closes or the user cancels
|
||||
* the loading scrim, or null when the embed can't serve this user (no
|
||||
* matching Puter session, an email to confirm, or a refused frame), so the
|
||||
* caller can use the popup instead. Never rejects.
|
||||
*
|
||||
@@ -59,22 +75,29 @@ export const showFeedbackEmbed = ({ guiOrigin, userUuid }) => new Promise((resol
|
||||
const dialog = document.createElement('dialog');
|
||||
dialog.className = 'puter-feedback-embed';
|
||||
dialog.style.cssText = 'position:fixed;inset:0;width:100%;height:100%;max-width:none;max-height:none;margin:0;padding:0;border:0;background:transparent;overflow:hidden;';
|
||||
// The embed draws its own scrim.
|
||||
const style = document.createElement('style');
|
||||
style.textContent = 'dialog.puter-feedback-embed::backdrop{background:transparent}';
|
||||
style.textContent = STYLE;
|
||||
const frame = document.createElement('iframe');
|
||||
frame.title = 'Send feedback';
|
||||
// The embed allows a framing page it can name from the referrer.
|
||||
frame.setAttribute('referrerpolicy', 'origin');
|
||||
// A frame whose color-scheme differs from its document's is painted
|
||||
// opaque; the embed declares none.
|
||||
frame.style.cssText = 'display:block;width:100%;height:100%;border:0;background:transparent;color-scheme:normal;';
|
||||
dialog.append(style, frame);
|
||||
// opaque, so this matches the embed's. WebKit also gives the frame this
|
||||
// scheme as its preferred one, so it has to follow the OS.
|
||||
frame.style.cssText = 'display:block;width:100%;height:100%;border:0;background:transparent;color-scheme:light dark;';
|
||||
const loading = document.createElement('div');
|
||||
loading.className = 'puter-feedback-embed-loading';
|
||||
loading.setAttribute('role', 'status');
|
||||
loading.setAttribute('aria-label', 'Loading');
|
||||
loading.append(document.createElement('span'));
|
||||
dialog.append(style, frame, loading);
|
||||
|
||||
let settled = false;
|
||||
let heard = false;
|
||||
let shown = false;
|
||||
let ready = false;
|
||||
let timer = null;
|
||||
let loadingTimer = null;
|
||||
const restoreFocus = document.activeElement;
|
||||
const wait = (ms) => {
|
||||
clearTimeout(timer);
|
||||
@@ -84,11 +107,17 @@ export const showFeedbackEmbed = ({ guiOrigin, userUuid }) => new Promise((resol
|
||||
if ( settled ) return;
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
clearTimeout(loadingTimer);
|
||||
window.removeEventListener('message', onMessage);
|
||||
dialog.remove();
|
||||
if ( shown ) restoreFocus?.focus?.({ preventScroll: true });
|
||||
resolve(result);
|
||||
};
|
||||
const show = () => {
|
||||
if ( dialog.open ) return;
|
||||
shown = true;
|
||||
dialog.showModal();
|
||||
};
|
||||
const onMessage = (event) => {
|
||||
if ( event.origin !== origin || event.source !== frame.contentWindow ) return;
|
||||
const { msg, state, sent } = event.data ?? {};
|
||||
@@ -99,9 +128,11 @@ export const showFeedbackEmbed = ({ guiOrigin, userUuid }) => new Promise((resol
|
||||
wait(READY_TIMEOUT_MS);
|
||||
} else if ( msg === 'feedbackEmbed' && state === 'ready' ) {
|
||||
heard = true;
|
||||
ready = true;
|
||||
clearTimeout(timer);
|
||||
shown = true;
|
||||
dialog.showModal();
|
||||
clearTimeout(loadingTimer);
|
||||
dialog.classList.remove('is-loading');
|
||||
show();
|
||||
frame.focus();
|
||||
} else if ( msg === 'feedbackEmbed' && state === 'unavailable' ) {
|
||||
finish(null);
|
||||
@@ -111,18 +142,26 @@ export const showFeedbackEmbed = ({ guiOrigin, userUuid }) => new Promise((resol
|
||||
frame.addEventListener('load', () => {
|
||||
if ( ! heard ) wait(LOAD_GRACE_MS);
|
||||
});
|
||||
// Escape outside the frame would close the dialog with the embed's
|
||||
// dialog still open inside it; the embed owns dismissal.
|
||||
// Once ready, Escape outside the frame would close the dialog with the
|
||||
// embed's dialog still open inside it; the embed owns dismissal.
|
||||
dialog.addEventListener('cancel', (event) => {
|
||||
event.preventDefault();
|
||||
frame.focus();
|
||||
if ( ready ) frame.focus();
|
||||
else finish({ sent: false });
|
||||
});
|
||||
loading.addEventListener('click', () => {
|
||||
if ( ! ready ) finish({ sent: false });
|
||||
});
|
||||
|
||||
try {
|
||||
frame.src = url.href;
|
||||
// Hidden until the embed is ready, so a refused session shows nothing.
|
||||
// Hidden at first, so a quickly refused session shows nothing.
|
||||
document.body.append(dialog);
|
||||
wait(READY_TIMEOUT_MS);
|
||||
loadingTimer = setTimeout(() => {
|
||||
dialog.classList.add('is-loading');
|
||||
show();
|
||||
}, LOADING_DELAY_MS);
|
||||
} catch {
|
||||
finish(null);
|
||||
}
|
||||
|
||||
@@ -50,6 +50,8 @@ describe('showFeedbackEmbed', () => {
|
||||
expect(url.search).toBe('');
|
||||
expect(new URLSearchParams(url.hash.slice(1)).get('user')).toBe(USER);
|
||||
expect(frame().getAttribute('referrerpolicy')).toBe('origin');
|
||||
// WebKit hands the frame this as its preferred scheme; it must follow the OS.
|
||||
expect(frame().getAttribute('style')).toContain('color-scheme: light dark');
|
||||
expect(frame().closest('dialog').open).toBe(false);
|
||||
});
|
||||
|
||||
@@ -72,6 +74,64 @@ describe('showFeedbackEmbed', () => {
|
||||
expect(document.activeElement).toBe(trigger);
|
||||
});
|
||||
|
||||
it('shows a loading scrim over the page while a slow session check runs, then the dialog', async () => {
|
||||
vi.useFakeTimers();
|
||||
const trigger = document.createElement('button');
|
||||
document.body.append(trigger);
|
||||
trigger.focus();
|
||||
const result = showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
|
||||
const dialog = frame().closest('dialog');
|
||||
fromEmbed({ msg: 'feedbackEmbed', state: 'loading' });
|
||||
await vi.advanceTimersByTimeAsync(199);
|
||||
expect(dialog.open).toBe(false);
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect(dialog.open).toBe(true);
|
||||
expect(dialog.classList.contains('is-loading')).toBe(true);
|
||||
expect(dialog.querySelector('[role="status"]')).not.toBeNull();
|
||||
|
||||
fromEmbed({ msg: 'feedbackEmbed', state: 'ready' });
|
||||
expect(dialog.classList.contains('is-loading')).toBe(false);
|
||||
expect(document.activeElement).toBe(frame());
|
||||
fromEmbed({ msg: 'feedbackDialogClosed', sent: true });
|
||||
await expect(result).resolves.toEqual({ sent: true });
|
||||
expect(document.activeElement).toBe(trigger);
|
||||
});
|
||||
|
||||
it('never shows the loading scrim when the embed is ready quickly', async () => {
|
||||
vi.useFakeTimers();
|
||||
showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
|
||||
const dialog = frame().closest('dialog');
|
||||
fromEmbed({ msg: 'feedbackEmbed', state: 'ready' });
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(dialog.open).toBe(true);
|
||||
expect(dialog.classList.contains('is-loading')).toBe(false);
|
||||
});
|
||||
|
||||
it('takes the loading scrim down for the popup when the embed turns out unavailable', async () => {
|
||||
vi.useFakeTimers();
|
||||
const trigger = document.createElement('button');
|
||||
document.body.append(trigger);
|
||||
trigger.focus();
|
||||
const result = showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
|
||||
await vi.advanceTimersByTimeAsync(200);
|
||||
fromEmbed({ msg: 'feedbackEmbed', state: 'unavailable' });
|
||||
await expect(result).resolves.toBeNull();
|
||||
expect(document.querySelector('dialog')).toBeNull();
|
||||
expect(document.activeElement).toBe(trigger);
|
||||
});
|
||||
|
||||
it.each([
|
||||
['Escape', (dialog) => dialog.dispatchEvent(new Event('cancel', { cancelable: true }))],
|
||||
['a click on the scrim', (dialog) => dialog.querySelector('[role="status"]').click()],
|
||||
])('lets %s cancel while loading, without falling back to the popup', async (_label, cancel) => {
|
||||
vi.useFakeTimers();
|
||||
const result = showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
|
||||
await vi.advanceTimersByTimeAsync(200);
|
||||
cancel(frame().closest('dialog'));
|
||||
await expect(result).resolves.toEqual({ sent: false });
|
||||
expect(document.querySelector('dialog')).toBeNull();
|
||||
});
|
||||
|
||||
it('resolves null without showing anything when the embed is unavailable', async () => {
|
||||
const result = showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
|
||||
const dialog = frame().closest('dialog');
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { PuterModule } from '../../lib/PuterModule.js';
|
||||
import { convo } from './convo.js';
|
||||
import { realtime } from './realtime.js';
|
||||
import { chat } from './chat.js';
|
||||
import { txt2img } from './image.js';
|
||||
import { listModelProviders, listModels } from './models.js';
|
||||
@@ -49,7 +49,7 @@ export class AIModule extends PuterModule {
|
||||
// runtime so destructured calls (`const { chat } = puter.ai`) keep
|
||||
// working like the old arrow fields did.
|
||||
chat = chat;
|
||||
convo = convo;
|
||||
realtime = realtime;
|
||||
img2txt = img2txt;
|
||||
speech2txt = speech2txt;
|
||||
speech2speech = speech2speech;
|
||||
@@ -66,7 +66,7 @@ export class AIModule extends PuterModule {
|
||||
/** @type {unknown} */ (this)
|
||||
);
|
||||
for ( const name of [
|
||||
'chat', 'convo', 'img2txt', 'speech2txt', 'speech2speech',
|
||||
'chat', 'realtime', 'img2txt', 'speech2txt', 'speech2speech',
|
||||
'txt2img', 'txt2vid', 'listModels', 'listModelProviders',
|
||||
] ) {
|
||||
methods[name] = methods[name].bind(this);
|
||||
|
||||
@@ -4,17 +4,17 @@ import {
|
||||
CLOSE_MESSAGE_TOO_LARGE,
|
||||
RealtimeSession,
|
||||
createRealtimeSocket,
|
||||
} from './convoTransport.js';
|
||||
} from './realtimeTransport.js';
|
||||
import { RealtimeEvents } from './realtimeEvents.js';
|
||||
import { ConvoMicrophone, ConvoSpeaker } from './convoMedia.js';
|
||||
import { ConvoHistory } from './convoHistory.js';
|
||||
import { RealtimeMicrophone, RealtimeSpeaker } from './realtimeMedia.js';
|
||||
import { RealtimeHistory } from './realtimeHistory.js';
|
||||
|
||||
/** @typedef {import('./types.js').ConvoOptions} ConvoOptions */
|
||||
/** @typedef {import('./types.js').ConvoSessionOptions} ConvoSessionOptions */
|
||||
/** @typedef {import('./types.js').ConvoPlaybackPosition} ConvoPlaybackPosition */
|
||||
/** @typedef {import('./types.js').RealtimeOptions} RealtimeOptions */
|
||||
/** @typedef {import('./types.js').RealtimeSessionOptions} RealtimeSessionOptions */
|
||||
/** @typedef {import('./types.js').RealtimePlaybackPosition} RealtimePlaybackPosition */
|
||||
/** @typedef {import('./types.js').RealtimeServerEvent} RealtimeServerEvent */
|
||||
/** @typedef {import('./types.js').RealtimeClientEvent} RealtimeClientEvent */
|
||||
/** @typedef {{ code: string; message: string }} ConvoError */
|
||||
/** @typedef {{ code: string; message: string }} RealtimeError */
|
||||
/**
|
||||
* A pending reconnect notice. A replacement session opens right away and takes
|
||||
* over once the conversation is idle.
|
||||
@@ -29,7 +29,7 @@ import { ConvoHistory } from './convoHistory.js';
|
||||
* probe: RealtimeSession | null;
|
||||
* created: RealtimeServerEvent | null;
|
||||
* stopProbe: (() => void) | null;
|
||||
* }} ConvoHandover
|
||||
* }} RealtimeHandover
|
||||
* @internal
|
||||
*/
|
||||
|
||||
@@ -106,7 +106,7 @@ const encoder = new TextEncoder();
|
||||
/**
|
||||
* @param {string} code
|
||||
* @param {string} message
|
||||
* @returns {ConvoError}
|
||||
* @returns {RealtimeError}
|
||||
*/
|
||||
const failure = (code, message) => ({ code, message });
|
||||
|
||||
@@ -138,7 +138,7 @@ function mergeSession(base, patch) {
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {ConvoOptions} options
|
||||
* @param {RealtimeOptions} options
|
||||
* @param {boolean} [allowModel]
|
||||
* @returns {void}
|
||||
*/
|
||||
@@ -261,12 +261,12 @@ function encodeEvent(event) {
|
||||
/**
|
||||
* A prepared realtime conversation with optional media helpers.
|
||||
*
|
||||
* @extends {RealtimeEvents<import('./types.js').ConvoEventMap>}
|
||||
* @extends {RealtimeEvents<import('./types.js').RealtimeEventMap>}
|
||||
*/
|
||||
export class Conversation extends RealtimeEvents {
|
||||
/** @type {import('../../index.js').Puter} */
|
||||
#puter;
|
||||
/** @type {ConvoOptions} */
|
||||
/** @type {RealtimeOptions} */
|
||||
#options;
|
||||
/** @type {'prepared' | 'connecting' | 'connected' | 'closed'} */
|
||||
#state = 'prepared';
|
||||
@@ -285,7 +285,7 @@ export class Conversation extends RealtimeEvents {
|
||||
* type: string;
|
||||
* matches: (event: RealtimeServerEvent) => boolean;
|
||||
* resolve: (event: RealtimeServerEvent) => void;
|
||||
* reject: (error: ConvoError) => void;
|
||||
* reject: (error: RealtimeError) => void;
|
||||
* timer: ReturnType<typeof setTimeout>;
|
||||
* eventId?: string;
|
||||
* anyError?: boolean;
|
||||
@@ -313,7 +313,7 @@ export class Conversation extends RealtimeEvents {
|
||||
/**
|
||||
* @type {{
|
||||
* resolve: () => void;
|
||||
* reject: (error: ConvoError) => void;
|
||||
* reject: (error: RealtimeError) => void;
|
||||
* }[]}
|
||||
*/
|
||||
#reconnectWaiters = [];
|
||||
@@ -330,8 +330,8 @@ export class Conversation extends RealtimeEvents {
|
||||
*/
|
||||
#interruption = null;
|
||||
#cancelled = new Set();
|
||||
#history = new ConvoHistory((id) => this.#cancelled.has(id));
|
||||
/** @type {ConvoHandover | null} */
|
||||
#history = new RealtimeHistory((id) => this.#cancelled.has(id));
|
||||
/** @type {RealtimeHandover | null} */
|
||||
#handover = null;
|
||||
// Speech started and its audio isn't committed or cleared yet.
|
||||
#speaking = false;
|
||||
@@ -358,17 +358,17 @@ export class Conversation extends RealtimeEvents {
|
||||
/** @type {Set<string>} */
|
||||
#committedItems = new Set();
|
||||
// Replayed `conversation.item.create` event IDs on the current session.
|
||||
/** @type {Map<string, import('./convoHistory.js').ReplayEntry>} */
|
||||
/** @type {Map<string, import('./realtimeHistory.js').ReplayEntry>} */
|
||||
#replayEvents = new Map();
|
||||
#replayReported = false;
|
||||
/** @type {ConvoMicrophone} */
|
||||
/** @type {RealtimeMicrophone} */
|
||||
microphone;
|
||||
/** @type {ConvoSpeaker} */
|
||||
/** @type {RealtimeSpeaker} */
|
||||
speaker;
|
||||
|
||||
/**
|
||||
* @param {import('../../index.js').Puter} puter
|
||||
* @param {ConvoOptions} options
|
||||
* @param {RealtimeOptions} options
|
||||
* @internal
|
||||
*/
|
||||
constructor(puter, options) {
|
||||
@@ -380,8 +380,8 @@ export class Conversation extends RealtimeEvents {
|
||||
model: options.model ?? DEFAULT_MODEL,
|
||||
};
|
||||
this.#turnDetection = this.#options.turnDetection !== false;
|
||||
this.microphone = new ConvoMicrophone(this);
|
||||
this.speaker = new ConvoSpeaker(this);
|
||||
this.microphone = new RealtimeMicrophone(this);
|
||||
this.speaker = new RealtimeSpeaker(this);
|
||||
}
|
||||
|
||||
/** @returns {'prepared' | 'connecting' | 'connected' | 'closed'} */
|
||||
@@ -807,7 +807,7 @@ export class Conversation extends RealtimeEvents {
|
||||
* restarting, is retried with backoff while the current session keeps
|
||||
* working; any other refusal ends the attempts.
|
||||
*
|
||||
* @param {ConvoHandover} handover
|
||||
* @param {RealtimeHandover} handover
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
async #openReplacement(handover) {
|
||||
@@ -887,7 +887,7 @@ export class Conversation extends RealtimeEvents {
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {ConvoHandover} handover
|
||||
* @param {RealtimeHandover} handover
|
||||
* @param {string} [code]
|
||||
* @returns {void}
|
||||
*/
|
||||
@@ -1011,7 +1011,7 @@ export class Conversation extends RealtimeEvents {
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {ConvoSessionOptions} options
|
||||
* @param {RealtimeSessionOptions} options
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
async configure(options) {
|
||||
@@ -1025,7 +1025,7 @@ export class Conversation extends RealtimeEvents {
|
||||
* new session needs. A partial update leaves settings made through native
|
||||
* events alone.
|
||||
*
|
||||
* @param {ConvoSessionOptions} options
|
||||
* @param {RealtimeSessionOptions} options
|
||||
* @param {boolean} [full]
|
||||
* @param {number} [ackMs]
|
||||
* @returns {Promise<void>}
|
||||
@@ -1168,7 +1168,7 @@ export class Conversation extends RealtimeEvents {
|
||||
* Stops local playback immediately and waits for cancellation and
|
||||
* truncation.
|
||||
*
|
||||
* @param {ConvoPlaybackPosition} [position]
|
||||
* @param {RealtimePlaybackPosition} [position]
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
interrupt(position) {
|
||||
@@ -1181,7 +1181,7 @@ export class Conversation extends RealtimeEvents {
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {ConvoPlaybackPosition | undefined} position
|
||||
* @param {RealtimePlaybackPosition | undefined} position
|
||||
* @param {'manual' | 'speech'} source
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
@@ -1323,7 +1323,7 @@ export class Conversation extends RealtimeEvents {
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {ConvoError} error
|
||||
* @param {RealtimeError} error
|
||||
* @returns {void}
|
||||
* @internal
|
||||
*/
|
||||
@@ -1347,7 +1347,7 @@ export class Conversation extends RealtimeEvents {
|
||||
promptIfUpgradeRequired(
|
||||
error,
|
||||
{
|
||||
method: 'puter.ai.convo',
|
||||
method: 'puter.ai.realtime',
|
||||
subscriptionMessage:
|
||||
'Realtime conversations require a subscription.',
|
||||
},
|
||||
@@ -1404,7 +1404,7 @@ export class Conversation extends RealtimeEvents {
|
||||
}
|
||||
}
|
||||
if (event.type === RECONNECT_NOTICE && !this.#handover) {
|
||||
/** @type {ConvoHandover} */
|
||||
/** @type {RealtimeHandover} */
|
||||
const handover = {
|
||||
reason:
|
||||
typeof event.reason === 'string' && event.reason
|
||||
@@ -1707,7 +1707,7 @@ export class Conversation extends RealtimeEvents {
|
||||
|
||||
/** @returns {string} */
|
||||
#eventId() {
|
||||
return `puter_convo_${++this.#sequence}`;
|
||||
return `puter_realtime_${++this.#sequence}`;
|
||||
}
|
||||
/**
|
||||
* Sends events together or not at all. `limit` caps the socket's send
|
||||
@@ -1830,7 +1830,7 @@ export class Conversation extends RealtimeEvents {
|
||||
return this.#wait(type, matches, event.event_id, false, timeoutMs);
|
||||
}
|
||||
/**
|
||||
* @param {ConvoError} error
|
||||
* @param {RealtimeError} error
|
||||
* @returns {void}
|
||||
*/
|
||||
#rejectPending(error) {
|
||||
@@ -1847,10 +1847,10 @@ export class Conversation extends RealtimeEvents {
|
||||
* connection.
|
||||
*
|
||||
* @this {import('./index.js').AIModule}
|
||||
* @param {ConvoOptions} [options]
|
||||
* @param {RealtimeOptions} [options]
|
||||
* @returns {Promise<Conversation>}
|
||||
*/
|
||||
export async function convo(options = {}) {
|
||||
export async function realtime(options = {}) {
|
||||
validateOptions(options, true);
|
||||
const settings = {
|
||||
...options,
|
||||
+5
-5
@@ -1,5 +1,5 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { Conversation, convo } from './convo.js';
|
||||
import { Conversation, realtime } from './realtime.js';
|
||||
|
||||
class FakeWebSocket extends EventTarget {
|
||||
static instances = [];
|
||||
@@ -283,14 +283,14 @@ describe('conversation preparation', () => {
|
||||
'rejects invalid options before opening a socket (%j)',
|
||||
async (options) => {
|
||||
await expect(
|
||||
convo.call({ puter: owner() }, options),
|
||||
realtime.call({ puter: owner() }, options),
|
||||
).rejects.toMatchObject({ code: expect.any(String) });
|
||||
expect(FakeWebSocket.instances).toHaveLength(0);
|
||||
},
|
||||
);
|
||||
it('requires authentication outside a browser', async () => {
|
||||
await expect(
|
||||
convo.call({ puter: { ...owner(), authToken: null } }),
|
||||
realtime.call({ puter: { ...owner(), authToken: null } }),
|
||||
).rejects.toMatchObject({ code: 'auth_required' });
|
||||
});
|
||||
it('reports canceled browser authentication', async () => {
|
||||
@@ -304,7 +304,7 @@ describe('conversation preparation', () => {
|
||||
},
|
||||
},
|
||||
};
|
||||
await expect(convo.call({ puter })).rejects.toMatchObject({
|
||||
await expect(realtime.call({ puter })).rejects.toMatchObject({
|
||||
code: 'auth_canceled',
|
||||
});
|
||||
});
|
||||
@@ -988,7 +988,7 @@ it('preflights both events of a text turn before sending either', async () => {
|
||||
socket().bufferedAmount =
|
||||
192 * 1024 -
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify({ ...item, event_id: 'puter_convo_99' }),
|
||||
JSON.stringify({ ...item, event_id: 'puter_realtime_99' }),
|
||||
).byteLength;
|
||||
expect(() => c.sendText(text)).toThrow(
|
||||
expect.objectContaining({ code: 'queue_full' }),
|
||||
+4
-4
@@ -4,7 +4,7 @@
|
||||
* An item `conversation.item.create` accepts: a text message, a function call
|
||||
* or a function call's output.
|
||||
*
|
||||
* @typedef {Record<string, unknown> & { type: string }} ConvoReplayItem
|
||||
* @typedef {Record<string, unknown> & { type: string }} RealtimeReplayItem
|
||||
*/
|
||||
|
||||
/**
|
||||
@@ -29,7 +29,7 @@
|
||||
* @property {string} [output]
|
||||
*/
|
||||
|
||||
/** @typedef {{ record: HistoryRecord; item: ConvoReplayItem }} ReplayEntry */
|
||||
/** @typedef {{ record: HistoryRecord; item: RealtimeReplayItem }} ReplayEntry */
|
||||
|
||||
const ITEM_EVENTS = new Set([
|
||||
'conversation.item.added',
|
||||
@@ -99,7 +99,7 @@ function partText(part) {
|
||||
*
|
||||
* @internal
|
||||
*/
|
||||
export class ConvoHistory {
|
||||
export class RealtimeHistory {
|
||||
/** @type {HistoryRecord[]} */
|
||||
#records = [];
|
||||
/** @type {Map<string, HistoryRecord>} */
|
||||
@@ -426,7 +426,7 @@ export class ConvoHistory {
|
||||
* The `conversation.item.create` item that recreates `record`, or `null`.
|
||||
*
|
||||
* @param {HistoryRecord} record
|
||||
* @returns {ConvoReplayItem | null}
|
||||
* @returns {RealtimeReplayItem | null}
|
||||
*/
|
||||
#replayItem(record) {
|
||||
if (!record.final) return null;
|
||||
+9
-9
@@ -1,5 +1,5 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { ConvoHistory } from './convoHistory.js';
|
||||
import { RealtimeHistory } from './realtimeHistory.js';
|
||||
|
||||
const user = (id, content) => ({
|
||||
type: 'conversation.item.added',
|
||||
@@ -10,9 +10,9 @@ const texts = (history) =>
|
||||
.replay(40, 48 * 1024)
|
||||
.entries.map(({ item }) => item.content[0].text);
|
||||
|
||||
describe('ConvoHistory', () => {
|
||||
describe('RealtimeHistory', () => {
|
||||
it('cuts text without word breaks at the played share', () => {
|
||||
const history = new ConvoHistory(() => false);
|
||||
const history = new RealtimeHistory(() => false);
|
||||
history.observe({
|
||||
type: 'response.output_item.added',
|
||||
response_id: 'r',
|
||||
@@ -40,7 +40,7 @@ describe('ConvoHistory', () => {
|
||||
});
|
||||
|
||||
it('keeps a known transcript when a later copy of the item lacks it', () => {
|
||||
const history = new ConvoHistory(() => false);
|
||||
const history = new RealtimeHistory(() => false);
|
||||
history.observe(user('u', [{ type: 'input_audio', transcript: null }]));
|
||||
expect(texts(history)).toEqual([]);
|
||||
history.observe({
|
||||
@@ -57,7 +57,7 @@ describe('ConvoHistory', () => {
|
||||
});
|
||||
|
||||
it('keeps a bounded number of the newest items', () => {
|
||||
const history = new ConvoHistory(() => false);
|
||||
const history = new RealtimeHistory(() => false);
|
||||
for (let i = 0; i < 100; i++)
|
||||
history.observe(
|
||||
user(`u${i}`, [{ type: 'input_text', text: `${i}` }]),
|
||||
@@ -67,7 +67,7 @@ describe('ConvoHistory', () => {
|
||||
});
|
||||
|
||||
it('stops at the first item over the byte budget instead of leaving a gap', () => {
|
||||
const history = new ConvoHistory(() => false);
|
||||
const history = new RealtimeHistory(() => false);
|
||||
history.observe(user('u1', [{ type: 'input_text', text: 'old' }]));
|
||||
history.observe(
|
||||
user('u2', [{ type: 'input_text', text: 'x'.repeat(50_000) }]),
|
||||
@@ -79,7 +79,7 @@ describe('ConvoHistory', () => {
|
||||
it.each(['x'.repeat(49 * 1024), '😀'.repeat(13 * 1024)])(
|
||||
'reports an oversized newest item without truncating it',
|
||||
(text) => {
|
||||
const history = new ConvoHistory(() => false);
|
||||
const history = new RealtimeHistory(() => false);
|
||||
history.observe(
|
||||
user('old', [{ type: 'input_text', text: 'tiny' }]),
|
||||
);
|
||||
@@ -96,7 +96,7 @@ describe('ConvoHistory', () => {
|
||||
);
|
||||
|
||||
it('waits on every unanswered call since the last user message', () => {
|
||||
const history = new ConvoHistory(() => false);
|
||||
const history = new RealtimeHistory(() => false);
|
||||
const call = (id, status = 'completed') =>
|
||||
history.observe({
|
||||
type: 'response.output_item.done',
|
||||
@@ -133,7 +133,7 @@ describe('ConvoHistory', () => {
|
||||
});
|
||||
|
||||
it('reports the new session copies of replayed items until they are done', () => {
|
||||
const history = new ConvoHistory(() => false);
|
||||
const history = new RealtimeHistory(() => false);
|
||||
history.observe(user('old', [{ type: 'input_text', text: 'hi' }]));
|
||||
history.replayed(history.replay(40, 48 * 1024).entries);
|
||||
const copy = user('new', [{ type: 'input_text', text: 'hi' }]);
|
||||
+32
-31
@@ -1,6 +1,6 @@
|
||||
/** @typedef {import('./convo.js').Conversation} Conversation */
|
||||
/** @typedef {import('./types.js').ConvoPlaybackPosition} ConvoPlaybackPosition */
|
||||
/** @typedef {import('./types.js').ConvoSpeakerOptions} ConvoSpeakerOptions */
|
||||
/** @typedef {import('./realtime.js').Conversation} Conversation */
|
||||
/** @typedef {import('./types.js').RealtimePlaybackPosition} RealtimePlaybackPosition */
|
||||
/** @typedef {import('./types.js').RealtimeSpeakerOptions} RealtimeSpeakerOptions */
|
||||
/** @typedef {import('./types.js').RealtimeServerEvent} RealtimeServerEvent */
|
||||
|
||||
const SAMPLE_RATE = 24000;
|
||||
@@ -69,7 +69,7 @@ function audioContext(sampleRate) {
|
||||
// The worklet owns resampling. Chunks the main thread hasn't taken within
|
||||
// about half a second are dropped rather than queued.
|
||||
const CAPTURE_WORKLET = `
|
||||
class PuterConvoCapture extends AudioWorkletProcessor {
|
||||
class PuterRealtimeCapture extends AudioWorkletProcessor {
|
||||
constructor() {
|
||||
super();
|
||||
this.samples = new Int16Array(480);
|
||||
@@ -108,13 +108,13 @@ class PuterConvoCapture extends AudioWorkletProcessor {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
registerProcessor('puter-convo-capture', PuterConvoCapture);
|
||||
registerProcessor('puter-realtime-capture', PuterRealtimeCapture);
|
||||
`;
|
||||
|
||||
/** Optional browser microphone capture. */
|
||||
export class ConvoMicrophone {
|
||||
export class RealtimeMicrophone {
|
||||
/** @type {Conversation} */
|
||||
#convo;
|
||||
#conversation;
|
||||
/** @type {MediaStream | null} */
|
||||
#stream = null;
|
||||
/** @type {AudioContext | null} */
|
||||
@@ -131,11 +131,11 @@ export class ConvoMicrophone {
|
||||
#cancelStart = null;
|
||||
|
||||
/**
|
||||
* @param {Conversation} convo
|
||||
* @param {Conversation} conversation
|
||||
* @internal
|
||||
*/
|
||||
constructor(convo) {
|
||||
this.#convo = convo;
|
||||
constructor(conversation) {
|
||||
this.#conversation = conversation;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -145,7 +145,7 @@ export class ConvoMicrophone {
|
||||
*/
|
||||
start() {
|
||||
try {
|
||||
this.#convo.assertConnected();
|
||||
this.#conversation.assertConnected();
|
||||
} catch (error) {
|
||||
return Promise.reject(error);
|
||||
}
|
||||
@@ -184,7 +184,8 @@ export class ConvoMicrophone {
|
||||
this.#context = context;
|
||||
// A reconnect keeps capture running; chunks are dropped meanwhile.
|
||||
const current = () =>
|
||||
generation === this.#generation && this.#convo.state !== 'closed';
|
||||
generation === this.#generation &&
|
||||
this.#conversation.state !== 'closed';
|
||||
const permission = navigator.mediaDevices
|
||||
.getUserMedia({
|
||||
audio: {
|
||||
@@ -228,27 +229,27 @@ export class ConvoMicrophone {
|
||||
}
|
||||
if (!current())
|
||||
throw { code: 'session_closed', message: 'Capture was stopped' };
|
||||
this.#node = new AudioWorkletNode(context, 'puter-convo-capture');
|
||||
this.#node = new AudioWorkletNode(context, 'puter-realtime-capture');
|
||||
this.#source = context.createMediaStreamSource(stream);
|
||||
const node = this.#node;
|
||||
node.port.onmessage = ({ data }) => {
|
||||
if (!current()) return;
|
||||
node.port.postMessage('ack');
|
||||
if (this.#muted || this.#convo.state !== 'connected') return;
|
||||
if (this.#muted || this.#conversation.state !== 'connected') return;
|
||||
try {
|
||||
this.#convo.appendAudio(data.pcm);
|
||||
this.#conversation.appendAudio(data.pcm);
|
||||
} catch (error) {
|
||||
// A congested socket, or one closing before the conversation
|
||||
// knows it, drops the chunk instead of ending capture: a
|
||||
// reconnect resumes it and a final close stops it.
|
||||
if (DROPPED_CHUNK_ERRORS.has(error?.code)) return;
|
||||
this.stop();
|
||||
this.#convo.reportMediaError(error);
|
||||
this.#conversation.reportMediaError(error);
|
||||
}
|
||||
};
|
||||
node.onprocessorerror = () => {
|
||||
this.stop();
|
||||
this.#convo.reportMediaError({
|
||||
this.#conversation.reportMediaError({
|
||||
code: 'media_error',
|
||||
message: 'Microphone processor failed',
|
||||
});
|
||||
@@ -298,9 +299,9 @@ export class ConvoMicrophone {
|
||||
}
|
||||
|
||||
/** Optional bounded browser PCM playback. */
|
||||
export class ConvoSpeaker {
|
||||
export class RealtimeSpeaker {
|
||||
/** @type {Conversation} */
|
||||
#convo;
|
||||
#conversation;
|
||||
/** @type {AudioContext | null} */
|
||||
#context = null;
|
||||
/** @type {Promise<void> | null} */
|
||||
@@ -317,7 +318,7 @@ export class ConvoSpeaker {
|
||||
* }>}
|
||||
*/
|
||||
#sources = new Set();
|
||||
/** @type {(ConvoPlaybackPosition & { responseId: string }) | null} */
|
||||
/** @type {(RealtimePlaybackPosition & { responseId: string }) | null} */
|
||||
#last = null;
|
||||
#nextAt = 0;
|
||||
#bargeIn = true;
|
||||
@@ -325,11 +326,11 @@ export class ConvoSpeaker {
|
||||
#overflowed = '';
|
||||
|
||||
/**
|
||||
* @param {Conversation} convo
|
||||
* @param {Conversation} conversation
|
||||
* @internal
|
||||
*/
|
||||
constructor(convo) {
|
||||
this.#convo = convo;
|
||||
constructor(conversation) {
|
||||
this.#conversation = conversation;
|
||||
}
|
||||
/**
|
||||
* @returns {boolean}
|
||||
@@ -365,12 +366,12 @@ export class ConvoSpeaker {
|
||||
/**
|
||||
* Starts playback without requesting microphone access.
|
||||
*
|
||||
* @param {ConvoSpeakerOptions} [options]
|
||||
* @param {RealtimeSpeakerOptions} [options]
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
start(options = {}) {
|
||||
try {
|
||||
this.#convo.assertConnected();
|
||||
this.#conversation.assertConnected();
|
||||
} catch (error) {
|
||||
return Promise.reject(error);
|
||||
}
|
||||
@@ -424,7 +425,7 @@ export class ConvoSpeaker {
|
||||
const responseId = String(event.response_id ?? '');
|
||||
if (this.#overflowed !== responseId) {
|
||||
this.#overflowed = responseId;
|
||||
this.#convo.reportMediaError({
|
||||
this.#conversation.reportMediaError({
|
||||
code: 'queue_full',
|
||||
message: `Playback exceeds ${MAX_SCHEDULED_SECONDS} seconds`,
|
||||
});
|
||||
@@ -471,7 +472,7 @@ export class ConvoSpeaker {
|
||||
this.#last.playedMs,
|
||||
(offset + duration) * 1000,
|
||||
);
|
||||
if (!this.#sources.size) this.#convo.playbackEnded();
|
||||
if (!this.#sources.size) this.#conversation.playbackEnded();
|
||||
};
|
||||
this.#nextAt = source.end;
|
||||
node.start(start);
|
||||
@@ -483,7 +484,7 @@ export class ConvoSpeaker {
|
||||
* is scheduled, it is the end of what played for `responseId`.
|
||||
*
|
||||
* @param {string} [responseId] The response still generating, if any.
|
||||
* @returns {ConvoPlaybackPosition | undefined}
|
||||
* @returns {RealtimePlaybackPosition | undefined}
|
||||
* @internal
|
||||
*/
|
||||
playbackPosition(responseId) {
|
||||
@@ -503,7 +504,7 @@ export class ConvoSpeaker {
|
||||
* Each item with scheduled audio the listener hasn't fully heard, in play
|
||||
* order, with how much of it was heard.
|
||||
*
|
||||
* @returns {ConvoPlaybackPosition[]}
|
||||
* @returns {RealtimePlaybackPosition[]}
|
||||
* @internal
|
||||
*/
|
||||
unheard() {
|
||||
@@ -511,7 +512,7 @@ export class ConvoSpeaker {
|
||||
const heard =
|
||||
(context?.currentTime ?? 0) -
|
||||
(context?.outputLatency || context?.baseLatency || 0);
|
||||
/** @type {Map<string, ConvoPlaybackPosition>} */
|
||||
/** @type {Map<string, RealtimePlaybackPosition>} */
|
||||
const items = new Map();
|
||||
for (const source of this.#sources) {
|
||||
if (source.end <= heard || items.has(source.itemId)) continue;
|
||||
@@ -540,7 +541,7 @@ export class ConvoSpeaker {
|
||||
this.#sources.clear();
|
||||
this.#last = null;
|
||||
this.#nextAt = 0;
|
||||
if (playing) this.#convo.playbackEnded();
|
||||
if (playing) this.#conversation.playbackEnded();
|
||||
}
|
||||
|
||||
/**
|
||||
+21
-21
@@ -1,5 +1,5 @@
|
||||
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
|
||||
import { ConvoMicrophone, ConvoSpeaker } from './convoMedia.js';
|
||||
import { RealtimeMicrophone, RealtimeSpeaker } from './realtimeMedia.js';
|
||||
|
||||
class FakeContext {
|
||||
static instances = [];
|
||||
@@ -43,7 +43,7 @@ class FakeWorklet {
|
||||
connect = vi.fn();
|
||||
disconnect = vi.fn();
|
||||
}
|
||||
const makeConvo = () => ({
|
||||
const makeConversation = () => ({
|
||||
state: 'connected',
|
||||
assertConnected() {
|
||||
if (this.state !== 'connected') throw { code: 'session_closed' };
|
||||
@@ -73,7 +73,7 @@ afterEach(async () => {
|
||||
});
|
||||
|
||||
it('captures PCM, acknowledges bounded worklet chunks, mutes and releases tracks', async () => {
|
||||
const c = makeConvo();
|
||||
const c = makeConversation();
|
||||
const audioTrack = track();
|
||||
vi.stubGlobal('navigator', {
|
||||
mediaDevices: {
|
||||
@@ -83,7 +83,7 @@ it('captures PCM, acknowledges bounded worklet chunks, mutes and releases tracks
|
||||
})),
|
||||
},
|
||||
});
|
||||
const mic = new ConvoMicrophone(c);
|
||||
const mic = new RealtimeMicrophone(c);
|
||||
helpers.push(mic);
|
||||
const pending = mic.start();
|
||||
expect(mic.start()).toBe(pending);
|
||||
@@ -115,7 +115,7 @@ it('releases a stream when permission resolves after stop', async () => {
|
||||
}),
|
||||
},
|
||||
});
|
||||
const mic = new ConvoMicrophone(makeConvo());
|
||||
const mic = new RealtimeMicrophone(makeConversation());
|
||||
helpers.push(mic);
|
||||
const pending = mic.start();
|
||||
mic.stop();
|
||||
@@ -134,7 +134,7 @@ it('settles start() when stopped during the permission prompt', async () => {
|
||||
}),
|
||||
},
|
||||
});
|
||||
const mic = new ConvoMicrophone(makeConvo());
|
||||
const mic = new RealtimeMicrophone(makeConversation());
|
||||
helpers.push(mic);
|
||||
const pending = mic.start();
|
||||
mic.stop();
|
||||
@@ -154,7 +154,7 @@ it('reports browser playback failures with SDK error codes', async () => {
|
||||
}
|
||||
},
|
||||
);
|
||||
const speaker = new ConvoSpeaker(makeConvo());
|
||||
const speaker = new RealtimeSpeaker(makeConversation());
|
||||
helpers.push(speaker);
|
||||
await expect(speaker.start()).rejects.toEqual({
|
||||
code: 'media_error',
|
||||
@@ -162,14 +162,14 @@ it('reports browser playback failures with SDK error codes', async () => {
|
||||
});
|
||||
});
|
||||
/** A started microphone over fake browser media. */
|
||||
const startedMic = async (c = makeConvo()) => {
|
||||
const startedMic = async (c = makeConversation()) => {
|
||||
const audioTrack = track();
|
||||
const getUserMedia = vi.fn(async () => ({
|
||||
getTracks: () => [audioTrack],
|
||||
getAudioTracks: () => [audioTrack],
|
||||
}));
|
||||
vi.stubGlobal('navigator', { mediaDevices: { getUserMedia } });
|
||||
const mic = new ConvoMicrophone(c);
|
||||
const mic = new RealtimeMicrophone(c);
|
||||
helpers.push(mic);
|
||||
await mic.start();
|
||||
const port = FakeWorklet.instances.at(-1).port;
|
||||
@@ -269,7 +269,7 @@ it('captures at the device rate', async () => {
|
||||
return new FakeContext();
|
||||
});
|
||||
vi.stubGlobal('AudioContext', ContextSpy);
|
||||
const mic = new ConvoMicrophone(makeConvo());
|
||||
const mic = new RealtimeMicrophone(makeConversation());
|
||||
helpers.push(mic);
|
||||
await mic.start();
|
||||
expect(ContextSpy).toHaveBeenCalledWith(undefined);
|
||||
@@ -284,7 +284,7 @@ it('reports a denied permission', async () => {
|
||||
},
|
||||
},
|
||||
});
|
||||
const mic = new ConvoMicrophone(makeConvo());
|
||||
const mic = new RealtimeMicrophone(makeConversation());
|
||||
helpers.push(mic);
|
||||
await expect(mic.start()).rejects.toMatchObject({
|
||||
code: 'permission_denied',
|
||||
@@ -292,7 +292,7 @@ it('reports a denied permission', async () => {
|
||||
});
|
||||
it('rejects unsupported capture environments', async () => {
|
||||
vi.stubGlobal('navigator', {});
|
||||
const mic = new ConvoMicrophone(makeConvo());
|
||||
const mic = new RealtimeMicrophone(makeConversation());
|
||||
helpers.push(mic);
|
||||
await expect(mic.start()).rejects.toMatchObject({
|
||||
code: 'unsupported_environment',
|
||||
@@ -310,7 +310,7 @@ it('waits on the permission prompt but times out a stuck audio context', async (
|
||||
}),
|
||||
},
|
||||
});
|
||||
const mic = new ConvoMicrophone(makeConvo());
|
||||
const mic = new RealtimeMicrophone(makeConversation());
|
||||
helpers.push(mic);
|
||||
const result = expect(mic.start()).rejects.toMatchObject({
|
||||
code: 'media_timeout',
|
||||
@@ -323,7 +323,7 @@ it('waits on the permission prompt but times out a stuck audio context', async (
|
||||
expect(audioTrack.stop).toHaveBeenCalled();
|
||||
});
|
||||
it('tracks played position and clears playback immediately', async () => {
|
||||
const speaker = new ConvoSpeaker(makeConvo());
|
||||
const speaker = new RealtimeSpeaker(makeConversation());
|
||||
helpers.push(speaker);
|
||||
await speaker.start();
|
||||
speaker.enqueue(new Int16Array(2400), {
|
||||
@@ -348,8 +348,8 @@ it('tracks played position and clears playback immediately', async () => {
|
||||
expect(context.close).toHaveBeenCalled();
|
||||
});
|
||||
it('bounds scheduled playback without interrupting the reply', async () => {
|
||||
const c = makeConvo();
|
||||
const speaker = new ConvoSpeaker(c);
|
||||
const c = makeConversation();
|
||||
const speaker = new RealtimeSpeaker(c);
|
||||
helpers.push(speaker);
|
||||
await speaker.start({ bargeIn: false });
|
||||
const delta = {
|
||||
@@ -371,7 +371,7 @@ it('bounds scheduled playback without interrupting the reply', async () => {
|
||||
expect(speaker.bargeIn).toBe(false);
|
||||
});
|
||||
it('subtracts output latency and covers gaps between chunks', async () => {
|
||||
const speaker = new ConvoSpeaker(makeConvo());
|
||||
const speaker = new RealtimeSpeaker(makeConversation());
|
||||
helpers.push(speaker);
|
||||
await speaker.start();
|
||||
const context = FakeContext.instances[0];
|
||||
@@ -396,8 +396,8 @@ it('subtracts output latency and covers gaps between chunks', async () => {
|
||||
});
|
||||
|
||||
it('reports playing until the last scheduled chunk ends', async () => {
|
||||
const c = makeConvo();
|
||||
const speaker = new ConvoSpeaker(c);
|
||||
const c = makeConversation();
|
||||
const speaker = new RealtimeSpeaker(c);
|
||||
helpers.push(speaker);
|
||||
await speaker.start();
|
||||
const context = FakeContext.instances[0];
|
||||
@@ -429,7 +429,7 @@ it('reports playing until the last scheduled chunk ends', async () => {
|
||||
});
|
||||
|
||||
it('lists every item not fully heard, in play order', async () => {
|
||||
const speaker = new ConvoSpeaker(makeConvo());
|
||||
const speaker = new RealtimeSpeaker(makeConversation());
|
||||
helpers.push(speaker);
|
||||
await speaker.start();
|
||||
const context = FakeContext.instances[0];
|
||||
@@ -469,7 +469,7 @@ it('a late canceled permission request does not stop a newer capture', async ()
|
||||
: Promise.resolve({ getTracks: () => [newTrack] }),
|
||||
},
|
||||
});
|
||||
const mic = new ConvoMicrophone(makeConvo());
|
||||
const mic = new RealtimeMicrophone(makeConversation());
|
||||
helpers.push(mic);
|
||||
const oldStart = mic.start();
|
||||
mic.stop();
|
||||
File renamed without changes.
+1
-1
@@ -1,5 +1,5 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { RealtimeSession } from './convoTransport.js';
|
||||
import { RealtimeSession } from './realtimeTransport.js';
|
||||
|
||||
/** Just enough of a WebSocket to drive the session's listeners. */
|
||||
const fakeSocket = () => {
|
||||
@@ -602,7 +602,7 @@ export {};
|
||||
/**
|
||||
* Session settings for a live conversation.
|
||||
*
|
||||
* @typedef {Object} ConvoSessionOptions
|
||||
* @typedef {Object} RealtimeSessionOptions
|
||||
* @property {string} [instructions] Instructions, up to 8192 characters.
|
||||
* @property {('text' | 'audio')[]} [output] Defaults to audio. Audio includes transcript events.
|
||||
* @property {string} [voice] OpenAI voice identifier, up to 128 characters.
|
||||
@@ -611,12 +611,12 @@ export {};
|
||||
/**
|
||||
* Options for preparing a conversation.
|
||||
*
|
||||
* @typedef {ConvoSessionOptions & { model?: string }} ConvoOptions
|
||||
* @typedef {RealtimeSessionOptions & { model?: string }} RealtimeOptions
|
||||
*/
|
||||
/**
|
||||
* How much of an assistant audio item a custom player has played.
|
||||
*
|
||||
* @typedef {Object} ConvoPlaybackPosition
|
||||
* @typedef {Object} RealtimePlaybackPosition
|
||||
* @property {string} itemId Assistant message item ID.
|
||||
* @property {number} contentIndex Audio content index.
|
||||
* @property {number} playedMs Played duration in milliseconds.
|
||||
@@ -624,24 +624,24 @@ export {};
|
||||
/**
|
||||
* Built-in speaker options.
|
||||
*
|
||||
* @typedef {Object} ConvoSpeakerOptions
|
||||
* @typedef {Object} RealtimeSpeakerOptions
|
||||
* @property {boolean} [bargeIn] Stop replies when user speech is detected. Defaults to true.
|
||||
*/
|
||||
|
||||
/**
|
||||
* @typedef {{ type: 'text'; delta: string; responseId: string }} ConvoTextEvent */
|
||||
* @typedef {{ type: 'text'; delta: string; responseId: string }} RealtimeTextEvent */
|
||||
/**
|
||||
* @typedef {{ type: 'audio'; pcm: Int16Array; sampleRate: number; responseId: string; itemId: string; contentIndex: number }} ConvoAudioEvent */
|
||||
* @typedef {{ type: 'audio'; pcm: Int16Array; sampleRate: number; responseId: string; itemId: string; contentIndex: number }} RealtimeAudioEvent */
|
||||
/**
|
||||
* @typedef {{ type: 'transcript'; role: 'user' | 'assistant'; text: string; final: boolean }} ConvoTranscriptEvent */
|
||||
* @typedef {{ type: 'transcript'; role: 'user' | 'assistant'; text: string; final: boolean }} RealtimeTranscriptEvent */
|
||||
/**
|
||||
* @typedef {{ type: 'interrupted'; source: 'manual' | 'speech'; responseId: string }} ConvoInterruptedEvent */
|
||||
* @typedef {{ type: 'interrupted'; source: 'manual' | 'speech'; responseId: string }} RealtimeInterruptedEvent */
|
||||
/**
|
||||
* A finished reply. `usage` and `status` are OpenAI's, unchanged.
|
||||
* @typedef {{ type: 'turn.done'; responseId: string | undefined; usage: unknown; status: unknown }} ConvoTurnDoneEvent */
|
||||
* @typedef {{ type: 'turn.done'; responseId: string | undefined; usage: unknown; status: unknown }} RealtimeTurnDoneEvent */
|
||||
/**
|
||||
* An OpenAI `error` event, or an SDK media failure, which also carries `code` and `message` at the
|
||||
* top level.
|
||||
* @typedef {{ type: 'error'; error: { code?: string; message?: string } & Record<string, unknown>; code?: string; message?: string }} ConvoErrorEvent */
|
||||
* @typedef {{ type: 'error'; error: { code?: string; message?: string } & Record<string, unknown>; code?: string; message?: string }} RealtimeErrorEvent */
|
||||
/** Native provider events retain their provider-owned fields.
|
||||
* @typedef {Record<string, RealtimeServerEvent> & { text: ConvoTextEvent; audio: ConvoAudioEvent; transcript: ConvoTranscriptEvent; interrupted: ConvoInterruptedEvent; 'turn.done': ConvoTurnDoneEvent; error: ConvoErrorEvent; ready: { type: 'ready'; model: string; provider: 'openai'; reconnected?: boolean }; reconnecting: { type: 'reconnecting'; code: number; reason: string }; context_trimmed: { type: 'context_trimmed'; omittedItems: number }; close: { type: 'close'; code: number; reason: string } }} ConvoEventMap */
|
||||
* @typedef {Record<string, RealtimeServerEvent> & { text: RealtimeTextEvent; audio: RealtimeAudioEvent; transcript: RealtimeTranscriptEvent; interrupted: RealtimeInterruptedEvent; 'turn.done': RealtimeTurnDoneEvent; error: RealtimeErrorEvent; ready: { type: 'ready'; model: string; provider: 'openai'; reconnected?: boolean }; reconnecting: { type: 'reconnecting'; code: number; reason: string }; context_trimmed: { type: 'context_trimmed'; omittedItems: number }; close: { type: 'close'; code: number; reason: string } }} RealtimeEventMap */
|
||||
@@ -340,7 +340,12 @@ el.open();
|
||||
The account button and Puter's app browser for a first-party app on a Puter subdomain (`notes.puter.com`). It is internal and left out of the public docs, because the app browser only works one subdomain under the Puter origin. It follows the page's own puter.js sign-in, never the Puter session's.
|
||||
|
||||
- Signed out, it shows **Sign in**, which calls `puter.auth.signIn()`. A temporary account counts as signed out.
|
||||
- Signed in, it shows the user's profile picture, or the username's initial when there is none. Clicking it opens a menu with the username and **Sign out**, which calls `puter.auth.signOut()`.
|
||||
- Signed in, it shows the user's profile picture, or the username's initial when there is none. Clicking it opens a menu with:
|
||||
- The user's picture, username and email. The email line is hidden when whoami leaves it out, such as for an app that can't read it.
|
||||
- **Manage account**, **Usage** and **Open Puter**, links that open the GUI origin's `/dashboard/#account`, `/dashboard/#usage` and `/` in a new tab.
|
||||
- **Send feedback**, which calls `puter.ui.showFeedbackDialog()`.
|
||||
- **Sign out**, which calls `puter.auth.signOut()`.
|
||||
- The menu opens with focus on its first item. ArrowUp and ArrowDown move between items and wrap around, and Home and End jump to the first and last.
|
||||
- On a page exactly one label under the Puter origin, with the same scheme and port, an **Apps** button opens the app browser (`/embed/apps`) in a panel under the button. It names the signed-in user as `#user=<uuid>`, so the app browser shows the Puter session's apps only to that user (see `doc/embedding-app-browser.md`), and passes a forced `theme` on as `theme=`. Elsewhere the button is hidden, because the server refuses the framing.
|
||||
- The menu and the Apps panel hang under their button with a caret pointing at it, and close on Escape, a press outside them, or their button. They're in the browser's top layer where it has one, so a host's `transform`, `overflow` or `z-index` can't clip or cover them. On screens up to 600px wide, the Apps panel spans the width below the button.
|
||||
- Closing the Apps panel keeps the app browser, so reopening it keeps its scroll and search. Any change to the page's sign-in discards it, so it never stays loaded for the previous user.
|
||||
@@ -355,6 +360,10 @@ The account button and Puter's app browser for a first-party app on a Puter subd
|
||||
| `sign-out-label` | `Sign out` | Sign-out menu item text |
|
||||
| `close-label` | `Close` | The Apps panel's close button label |
|
||||
| `account-label` | `Account` | Account button label, read as `Account: <username>` |
|
||||
| `manage-account-label` | `Manage account` | Menu link to the account settings |
|
||||
| `usage-label` | `Usage` | Menu link to the usage page |
|
||||
| `open-puter-label` | `Open Puter` | Menu link to Puter |
|
||||
| `feedback-label` | `Send feedback` | Menu item that opens the feedback dialog |
|
||||
| `theme` | follows the OS | `light` or `dark`, for the component and the app browser in it |
|
||||
|
||||
Changing an attribute updates the component in place.
|
||||
|
||||
@@ -4,7 +4,8 @@
|
||||
*
|
||||
* Signed out (a temporary account counts as signed out), it shows a Sign in
|
||||
* button. Signed in, it shows the user's profile picture (or initial), which
|
||||
* opens a menu with the username and Sign out. On a page one subdomain under
|
||||
* opens a menu with the account, links to manage it, see its usage and open
|
||||
* Puter, Send feedback, and Sign out. On a page one subdomain under
|
||||
* the Puter origin, an Apps button opens Puter's app browser in a panel under
|
||||
* it, for the same user: signed out, or signed in as an account other than
|
||||
* the Puter session's, the browser shows only recommended apps. The browser
|
||||
@@ -12,7 +13,8 @@
|
||||
* Renders nothing inside the Puter desktop.
|
||||
*
|
||||
* Attributes: theme, apps-label, sign-in-label, sign-out-label, close-label,
|
||||
* account-label
|
||||
* account-label, manage-account-label, usage-label, open-puter-label,
|
||||
* feedback-label
|
||||
* Parts: apps-button, avatar, sign-in, menu, apps-panel
|
||||
* Events: signin (detail.user), signout
|
||||
*/
|
||||
@@ -24,6 +26,7 @@ import { canFrameGuiEmbed } from '../../lib/guiEmbed.js';
|
||||
const APPS_ICON = `<svg viewBox="0 0 24 24" width="20" height="20" aria-hidden="true" focusable="false">${
|
||||
[5, 12, 19].flatMap(y => [5, 12, 19].map(x => `<circle cx="${x}" cy="${y}" r="2"/>`)).join('')
|
||||
}</svg>`;
|
||||
const EXTERNAL_ICON = '<svg class="menu-external" viewBox="0 0 32 32" width="11" height="11" aria-hidden="true" focusable="false"><path d="m26 28h-20a2 2 0 0 1 -2-2v-20a2 2 0 0 1 2-2h10v2h-10v20h20v-10h2v10a2 2 0 0 1 -2 2z"/><path d="m20 2v2h6.586l-8.586 8.586 1.414 1.414 8.586-8.586v6.586h2v-10z"/></svg>';
|
||||
const CLOSE_ICON = '<svg viewBox="0 0 24 24" width="16" height="16" aria-hidden="true" focusable="false" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><path d="M6 6l12 12M18 6L6 18"/></svg>';
|
||||
|
||||
const LABELS = {
|
||||
@@ -32,8 +35,19 @@ const LABELS = {
|
||||
'sign-out-label': 'Sign out',
|
||||
'close-label': 'Close',
|
||||
'account-label': 'Account',
|
||||
'manage-account-label': 'Manage account',
|
||||
'usage-label': 'Usage',
|
||||
'open-puter-label': 'Open Puter',
|
||||
'feedback-label': 'Send feedback',
|
||||
};
|
||||
|
||||
// Menu links, each opening a page of the Puter GUI in a new tab.
|
||||
const MENU_LINKS = [
|
||||
{ className: 'manage-account', label: 'manage-account-label', path: '/dashboard/#account' },
|
||||
{ className: 'usage', label: 'usage-label', path: '/dashboard/#usage' },
|
||||
{ className: 'open-puter', label: 'open-puter-label', path: '/' },
|
||||
];
|
||||
|
||||
// Gap between a panel and the viewport edge, and between its caret and corners.
|
||||
const EDGE = 8;
|
||||
const CARET_INSET = 20;
|
||||
@@ -216,7 +230,8 @@ class PuterUserMenu extends PuterWebComponent {
|
||||
text-align: center;
|
||||
}
|
||||
.avatar[aria-expanded="true"] { box-shadow: 0 0 0 3px rgba(8, 142, 240, .25); }
|
||||
.avatar img {
|
||||
.avatar img,
|
||||
.menu-avatar img {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
width: 100%;
|
||||
@@ -285,31 +300,78 @@ class PuterUserMenu extends PuterWebComponent {
|
||||
}
|
||||
|
||||
.menu {
|
||||
min-width: 200px;
|
||||
min-width: 220px;
|
||||
max-width: min(300px, calc(100vw - ${EDGE * 2}px));
|
||||
padding: 4px 0;
|
||||
text-align: left;
|
||||
}
|
||||
.menu-user {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
padding: 8px 14px 10px;
|
||||
margin-bottom: 4px;
|
||||
border-bottom: 1px solid #eee;
|
||||
color: #414650;
|
||||
font-size: 14px;
|
||||
}
|
||||
.menu-avatar {
|
||||
position: relative;
|
||||
flex: none;
|
||||
width: 40px;
|
||||
height: 40px;
|
||||
overflow: hidden;
|
||||
border-radius: 50%;
|
||||
background: linear-gradient(#34a5f8, #088ef0);
|
||||
color: #fff;
|
||||
font-size: 18px;
|
||||
font-weight: 600;
|
||||
line-height: 40px;
|
||||
text-align: center;
|
||||
}
|
||||
.menu-identity {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
min-width: 0;
|
||||
}
|
||||
.menu-username,
|
||||
.menu-email {
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.menu-username {
|
||||
color: #414650;
|
||||
font-size: 14px;
|
||||
font-weight: 600;
|
||||
}
|
||||
.menu-email {
|
||||
color: #6b7280;
|
||||
font-size: 12px;
|
||||
}
|
||||
.menu-item {
|
||||
display: block;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
box-sizing: border-box;
|
||||
width: 100%;
|
||||
padding: 8px 14px;
|
||||
border: none;
|
||||
background: none;
|
||||
color: #333;
|
||||
font-size: 14px;
|
||||
line-height: normal;
|
||||
text-align: left;
|
||||
text-decoration: none;
|
||||
cursor: pointer;
|
||||
}
|
||||
.menu-external {
|
||||
flex: none;
|
||||
fill: currentColor;
|
||||
opacity: .7;
|
||||
}
|
||||
.menu-separator {
|
||||
height: 1px;
|
||||
margin: 4px 0;
|
||||
background-color: #eee;
|
||||
}
|
||||
.menu-item:focus-visible {
|
||||
outline: none;
|
||||
@@ -411,7 +473,9 @@ class PuterUserMenu extends PuterWebComponent {
|
||||
}
|
||||
:host(.puter-theme-dark) .menu-user,
|
||||
:host(.puter-theme-dark) .apps-header { border-color: #3a3a3a; }
|
||||
:host(.puter-theme-dark) .menu-user,
|
||||
:host(.puter-theme-dark) .menu-separator { background-color: #3a3a3a; }
|
||||
:host(.puter-theme-dark) .menu-email { color: #a0a0a0; }
|
||||
:host(.puter-theme-dark) .menu-username,
|
||||
:host(.puter-theme-dark) .apps-title,
|
||||
:host(.puter-theme-dark) .menu-item { color: #e6e6e6; }
|
||||
:host(.puter-theme-dark) .menu-item:focus-visible { background-color: #3a3a3a; color: #fff; }
|
||||
@@ -442,7 +506,16 @@ class PuterUserMenu extends PuterWebComponent {
|
||||
<button type="button" class="avatar" part="avatar" aria-haspopup="menu" aria-expanded="false" hidden></button>
|
||||
</div>
|
||||
<div class="panel menu" part="menu" role="menu" tabindex="-1" hidden>
|
||||
<div class="menu-user" hidden></div>
|
||||
<div class="menu-user" hidden>
|
||||
<span class="menu-avatar" aria-hidden="true"></span>
|
||||
<span class="menu-identity">
|
||||
<span class="menu-username"></span>
|
||||
<span class="menu-email" hidden></span>
|
||||
</span>
|
||||
</div>
|
||||
${MENU_LINKS.map(link => `<a class="menu-item ${link.className}" role="menuitem" target="_blank" rel="noopener"><span class="menu-label"></span>${EXTERNAL_ICON}</a>`).join('')}
|
||||
<button type="button" class="menu-item feedback" role="menuitem"></button>
|
||||
<div class="menu-separator" role="separator"></div>
|
||||
<button type="button" class="menu-item sign-out" role="menuitem"></button>
|
||||
</div>
|
||||
<div class="panel apps-panel" part="apps-panel" role="dialog" tabindex="-1" hidden>
|
||||
@@ -471,6 +544,10 @@ class PuterUserMenu extends PuterWebComponent {
|
||||
if ( this.$('.menu').hidden ) this._openMenu();
|
||||
else this._closePanels();
|
||||
});
|
||||
for ( const link of MENU_LINKS ) {
|
||||
this.$(`.${link.className}`).addEventListener('click', () => this._closePanels());
|
||||
}
|
||||
this.$('.feedback').addEventListener('click', () => this._sendFeedback());
|
||||
this.$('.sign-out').addEventListener('click', () => this._signOut());
|
||||
this.$('.apps-close').addEventListener('click', () => this._closePanels({ returnFocus: true }));
|
||||
this.shadowRoot.addEventListener('keydown', (event) => {
|
||||
@@ -482,6 +559,8 @@ class PuterUserMenu extends PuterWebComponent {
|
||||
} else if ( event.key === 'Tab' && open.classList.contains('menu') ) {
|
||||
// Tab leaves the menu: it closes, and focus moves on from its button.
|
||||
this._closePanels({ returnFocus: true });
|
||||
} else if ( open.classList.contains('menu') && this._moveMenuFocus(event.key) ) {
|
||||
event.preventDefault();
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -501,6 +580,10 @@ class PuterUserMenu extends PuterWebComponent {
|
||||
close.title = this._label('close-label');
|
||||
this.$('.sign-in').textContent = this._label('sign-in-label');
|
||||
this.$('.sign-out').textContent = this._label('sign-out-label');
|
||||
this.$('.feedback').textContent = this._label('feedback-label');
|
||||
for ( const link of MENU_LINKS ) {
|
||||
this.$(`.${link.className} .menu-label`).textContent = this._label(link.label);
|
||||
}
|
||||
this._renderAvatarLabel();
|
||||
}
|
||||
|
||||
@@ -522,19 +605,17 @@ class PuterUserMenu extends PuterWebComponent {
|
||||
|
||||
const avatar = this.$('.avatar');
|
||||
avatar.hidden = !this.#ready || !signedIn;
|
||||
avatar.textContent = username ? username[0].toUpperCase() : '?';
|
||||
if ( this.#picture ) {
|
||||
const img = document.createElement('img');
|
||||
img.alt = '';
|
||||
// The initial stays underneath for a picture the page's CSP refuses.
|
||||
img.addEventListener('error', () => img.remove());
|
||||
img.src = this.#picture;
|
||||
avatar.append(img);
|
||||
}
|
||||
this._fillAvatar(avatar, username);
|
||||
this._renderAvatarLabel();
|
||||
const menuUser = this.$('.menu-user');
|
||||
menuUser.textContent = username;
|
||||
menuUser.hidden = !username;
|
||||
this.$('.menu-user').hidden = !username;
|
||||
this.$('.menu-username').textContent = username;
|
||||
this._fillAvatar(this.$('.menu-avatar'), username);
|
||||
const email = typeof this.#user?.email === 'string' ? this.#user.email : '';
|
||||
this.$('.menu-email').textContent = email;
|
||||
this.$('.menu-email').hidden = !email;
|
||||
for ( const link of MENU_LINKS ) {
|
||||
this.$(`.${link.className}`).href = new URL(link.path, globalThis.puter.defaultGUIOrigin).href;
|
||||
}
|
||||
if ( avatar.hidden && !this.$('.menu').hidden ) this._closePanels();
|
||||
if ( this.#ready && this.#focusSignIn ) {
|
||||
this.#focusSignIn = false;
|
||||
@@ -542,6 +623,18 @@ class PuterUserMenu extends PuterWebComponent {
|
||||
}
|
||||
}
|
||||
|
||||
/** The username's initial, with the profile picture over it once there is one. */
|
||||
_fillAvatar (el, username) {
|
||||
el.textContent = username ? username[0].toUpperCase() : '?';
|
||||
if ( ! this.#picture ) return;
|
||||
const img = document.createElement('img');
|
||||
img.alt = '';
|
||||
// The initial stays underneath for a picture the page's CSP refuses.
|
||||
img.addEventListener('error', () => img.remove());
|
||||
img.src = this.#picture;
|
||||
el.append(img);
|
||||
}
|
||||
|
||||
// -- Panels --
|
||||
|
||||
_openPanel () {
|
||||
@@ -607,7 +700,23 @@ class PuterUserMenu extends PuterWebComponent {
|
||||
_openMenu () {
|
||||
const menu = this.$('.menu');
|
||||
this._showPanel(menu);
|
||||
this.$('.sign-out').focus({ preventScroll: true });
|
||||
this.$('.menu-item').focus({ preventScroll: true });
|
||||
}
|
||||
|
||||
/** Moves focus between menu items for an arrow, Home or End key; false for any other key. */
|
||||
_moveMenuFocus (key) {
|
||||
const items = [...this.$$('.menu .menu-item')];
|
||||
const current = items.indexOf(this.shadowRoot.activeElement);
|
||||
const last = items.length - 1;
|
||||
const next = {
|
||||
ArrowDown: current < 0 || current === last ? 0 : current + 1,
|
||||
ArrowUp: current <= 0 ? last : current - 1,
|
||||
Home: 0,
|
||||
End: last,
|
||||
}[key];
|
||||
if ( next === undefined ) return false;
|
||||
items[next].focus({ preventScroll: true });
|
||||
return true;
|
||||
}
|
||||
|
||||
_openApps () {
|
||||
@@ -671,6 +780,11 @@ class PuterUserMenu extends PuterWebComponent {
|
||||
);
|
||||
}
|
||||
|
||||
_sendFeedback () {
|
||||
this._closePanels();
|
||||
void globalThis.puter.ui.showFeedbackDialog();
|
||||
}
|
||||
|
||||
async _signOut () {
|
||||
this.#focusSignIn = Boolean(this.shadowRoot.activeElement);
|
||||
this._closePanels();
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import PuterUserMenu from './PuterUserMenu.js';
|
||||
|
||||
const USER = { username: 'jelveh', uuid: '0b1c9a52-3f0e-4c7a-9a65-2f6d1e8b7c40' };
|
||||
const USER = { username: 'jelveh', uuid: '0b1c9a52-3f0e-4c7a-9a65-2f6d1e8b7c40', email: '[email protected]' };
|
||||
const OTHER = { username: 'sam', uuid: '7d2e4f10-8a3b-4c5d-9e6f-0a1b2c3d4e5f' };
|
||||
const TEMP = { username: 'quiet_otter_42', uuid: '5c4b3a29-1807-4f6e-9d5c-4b3a29180716', is_temp: true };
|
||||
const PICTURE = 'data:image/png;base64,iVBORw0KGgo=';
|
||||
@@ -24,6 +24,9 @@ const fakePuter = ({ token = null, user = null, picture = null } = {}) => {
|
||||
listeners.add(listener);
|
||||
return () => listeners.delete(listener);
|
||||
},
|
||||
ui: {
|
||||
showFeedbackDialog: vi.fn(() => new Promise(() => {})),
|
||||
},
|
||||
auth: {
|
||||
getProfilePicture: vi.fn(async () => currentPicture),
|
||||
signIn: vi.fn(() => new Promise(() => {})),
|
||||
@@ -164,7 +167,10 @@ describe('<puter-user-menu>', () => {
|
||||
part(el, '.avatar').click();
|
||||
expect(shown(el, '.menu')).toBe(true);
|
||||
expect(part(el, '.avatar').getAttribute('aria-expanded')).toBe('true');
|
||||
expect(part(el, '.menu-user').textContent).toBe('jelveh');
|
||||
expect(part(el, '.menu-username').textContent).toBe('jelveh');
|
||||
expect(part(el, '.menu-email').textContent).toBe(USER.email);
|
||||
expect(shown(el, '.menu-email')).toBe(true);
|
||||
expect(part(el, '.menu-avatar').textContent).toBe('J');
|
||||
|
||||
part(el, '.apps-button').click();
|
||||
expect(shown(el, '.menu')).toBe(false);
|
||||
@@ -173,6 +179,80 @@ describe('<puter-user-menu>', () => {
|
||||
expect(frameOf(el).src).toBe(`https://puter.com/embed/apps#user=${USER.uuid}`);
|
||||
});
|
||||
|
||||
it('links to the account, its usage and Puter, each in a new tab', async () => {
|
||||
vi.stubGlobal('puter', fakePuter({ token: 'app-token', user: USER }));
|
||||
const el = await mount();
|
||||
part(el, '.avatar').click();
|
||||
expect(el.shadowRoot.activeElement).toBe(part(el, '.manage-account'));
|
||||
const links = [...el.shadowRoot.querySelectorAll('.menu a.menu-item')].map(link => ({
|
||||
label: link.textContent,
|
||||
href: link.href,
|
||||
target: link.target,
|
||||
rel: link.rel,
|
||||
}));
|
||||
expect(links).toEqual([
|
||||
{ label: 'Manage account', href: 'https://puter.com/dashboard/#account', target: '_blank', rel: 'noopener' },
|
||||
{ label: 'Usage', href: 'https://puter.com/dashboard/#usage', target: '_blank', rel: 'noopener' },
|
||||
{ label: 'Open Puter', href: 'https://puter.com/', target: '_blank', rel: 'noopener' },
|
||||
]);
|
||||
|
||||
part(el, '.usage').dispatchEvent(new MouseEvent('click', { cancelable: true }));
|
||||
expect(shown(el, '.menu')).toBe(false);
|
||||
});
|
||||
|
||||
it('hides the email line when the account has none to show', async () => {
|
||||
vi.stubGlobal('puter', fakePuter({ token: 'app-token', user: OTHER }));
|
||||
const el = await mount();
|
||||
part(el, '.avatar').click();
|
||||
expect(part(el, '.menu-username').textContent).toBe('sam');
|
||||
expect(shown(el, '.menu-email')).toBe(false);
|
||||
});
|
||||
|
||||
it('shows the profile picture in the menu header too', async () => {
|
||||
vi.stubGlobal('puter', fakePuter({ token: 'app-token', user: USER, picture: PICTURE }));
|
||||
const el = await mount();
|
||||
expect(part(el, '.menu-avatar img').getAttribute('src')).toBe(PICTURE);
|
||||
expect(part(el, '.menu-avatar').textContent).toBe('J');
|
||||
});
|
||||
|
||||
it('opens the feedback dialog from the menu', async () => {
|
||||
const sdk = fakePuter({ token: 'app-token', user: USER });
|
||||
vi.stubGlobal('puter', sdk);
|
||||
const el = await mount();
|
||||
part(el, '.avatar').click();
|
||||
part(el, '.feedback').click();
|
||||
expect(sdk.ui.showFeedbackDialog).toHaveBeenCalledTimes(1);
|
||||
expect(shown(el, '.menu')).toBe(false);
|
||||
});
|
||||
|
||||
it('moves focus through the menu with the arrow, Home and End keys', async () => {
|
||||
vi.stubGlobal('puter', fakePuter({ token: 'app-token', user: USER }));
|
||||
const el = await mount();
|
||||
part(el, '.avatar').click();
|
||||
const press = (key) => {
|
||||
const event = new KeyboardEvent('keydown', { key, bubbles: true, composed: true, cancelable: true });
|
||||
el.shadowRoot.activeElement.dispatchEvent(event);
|
||||
return event;
|
||||
};
|
||||
const focused = () => el.shadowRoot.activeElement.classList[1];
|
||||
|
||||
expect(focused()).toBe('manage-account');
|
||||
expect(press('ArrowDown').defaultPrevented).toBe(true);
|
||||
expect(focused()).toBe('usage');
|
||||
press('End');
|
||||
expect(focused()).toBe('sign-out');
|
||||
press('ArrowDown');
|
||||
expect(focused()).toBe('manage-account');
|
||||
press('ArrowUp');
|
||||
expect(focused()).toBe('sign-out');
|
||||
press('ArrowUp');
|
||||
expect(focused()).toBe('feedback');
|
||||
press('Home');
|
||||
expect(focused()).toBe('manage-account');
|
||||
expect(press('a').defaultPrevented).toBe(false);
|
||||
expect(shown(el, '.menu')).toBe(true);
|
||||
});
|
||||
|
||||
it('hangs the panel under its button with the caret pointing at it', async () => {
|
||||
vi.stubGlobal('puter', fakePuter({ token: 'app-token', user: USER }));
|
||||
const el = await mount();
|
||||
@@ -381,12 +461,25 @@ describe('<puter-user-menu>', () => {
|
||||
|
||||
it('uses label attributes, including ones changed later', async () => {
|
||||
vi.stubGlobal('puter', fakePuter({ token: 'app-token', user: USER }));
|
||||
const el = await mount({ 'apps-label': 'Applis', 'sign-out-label': 'Déconnexion', 'account-label': 'Compte' });
|
||||
const el = await mount({
|
||||
'apps-label': 'Applis',
|
||||
'sign-out-label': 'Déconnexion',
|
||||
'account-label': 'Compte',
|
||||
'manage-account-label': 'Gérer le compte',
|
||||
'usage-label': 'Utilisation',
|
||||
'open-puter-label': 'Ouvrir Puter',
|
||||
'feedback-label': 'Envoyer un avis',
|
||||
});
|
||||
expect(part(el, '.apps-button').getAttribute('aria-label')).toBe('Applis');
|
||||
expect(part(el, '.apps-title').textContent).toBe('Applis');
|
||||
expect(part(el, '.apps-panel').getAttribute('aria-label')).toBe('Applis');
|
||||
expect(part(el, '.sign-out').textContent).toBe('Déconnexion');
|
||||
expect(part(el, '.avatar').getAttribute('aria-label')).toBe('Compte: jelveh');
|
||||
expect(part(el, '.manage-account').textContent).toBe('Gérer le compte');
|
||||
expect(part(el, '.usage').textContent).toBe('Utilisation');
|
||||
expect(part(el, '.open-puter').textContent).toBe('Ouvrir Puter');
|
||||
expect(part(el, '.feedback').textContent).toBe('Envoyer un avis');
|
||||
expect(part(el, '.manage-account .menu-external')).not.toBeNull();
|
||||
|
||||
el.setAttribute('close-label', 'Fermer');
|
||||
el.setAttribute('sign-in-label', 'Connexion');
|
||||
|
||||
+16
-16
@@ -6,7 +6,7 @@ import WebSocket, { WebSocketServer } from 'ws';
|
||||
import type { AddressInfo } from 'node:net';
|
||||
import { loadNodePuter, closeNodePuter } from '../harness/nodeSdkLoader';
|
||||
import type { EnvManifest, PuterSDK } from '../harness/types';
|
||||
import type { Conversation } from '../../../types/modules/ai/convo';
|
||||
import type { Conversation } from '../../../types/modules/ai/realtime';
|
||||
import type { RealtimeServerEvent } from '../../../types/modules/ai/types';
|
||||
import {
|
||||
setupPuterTestEnv,
|
||||
@@ -14,7 +14,7 @@ import {
|
||||
} from '@heyputer/backend/testUtil.ts';
|
||||
import type { IConfig } from '@heyputer/backend/types.ts';
|
||||
|
||||
describe('convo built SDK and browser media', () => {
|
||||
describe('realtime built SDK and browser media', () => {
|
||||
let env: PuterTestEnv;
|
||||
let upstream: WebSocketServer;
|
||||
let puter: PuterSDK;
|
||||
@@ -220,7 +220,7 @@ describe('convo built SDK and browser media', () => {
|
||||
it('resolves the original built SDK connect after an initial provider restart', async () => {
|
||||
upstreamRefusals = 1;
|
||||
const before = connections.length;
|
||||
const conversation = await puter.ai.convo({ output: ['text'] });
|
||||
const conversation = await puter.ai.realtime({ output: ['text'] });
|
||||
try {
|
||||
const ready = conversation.connect();
|
||||
expect(conversation.connect()).toBe(ready);
|
||||
@@ -234,7 +234,7 @@ describe('convo built SDK and browser media', () => {
|
||||
});
|
||||
|
||||
it('prepares, configures, sends, interrupts and truncates using the built SDK', async () => {
|
||||
const conversation = await puter.ai.convo({
|
||||
const conversation = await puter.ai.realtime({
|
||||
output: ['text'],
|
||||
turnDetection: false,
|
||||
});
|
||||
@@ -283,13 +283,13 @@ describe('convo built SDK and browser media', () => {
|
||||
new URL('../../../dist/puter.js', import.meta.url),
|
||||
'utf8',
|
||||
);
|
||||
await page.route('**/__convo_test__', (route) =>
|
||||
await page.route('**/__realtime_test__', (route) =>
|
||||
route.fulfill({
|
||||
contentType: 'text/html',
|
||||
body: '<!doctype html><title>Conversation media test</title>',
|
||||
}),
|
||||
);
|
||||
await page.goto(`${env.apiOrigin}/__convo_test__`);
|
||||
await page.goto(`${env.apiOrigin}/__realtime_test__`);
|
||||
await page.evaluate(
|
||||
({ origin, apiOrigin }) => {
|
||||
Object.assign(globalThis, {
|
||||
@@ -315,7 +315,7 @@ describe('convo built SDK and browser media', () => {
|
||||
};
|
||||
let c;
|
||||
try {
|
||||
c = await sdk.ai.convo();
|
||||
c = await sdk.ai.realtime();
|
||||
} catch (error) {
|
||||
throw new Error(`prepare: ${JSON.stringify(error)}`);
|
||||
}
|
||||
@@ -432,7 +432,7 @@ describe('convo built SDK and browser media', () => {
|
||||
{ ...manifest, apiOrigin: proxyOrigin },
|
||||
env.users.user.token,
|
||||
);
|
||||
const conversation = await dropping.ai.convo({ output: ['text'] });
|
||||
const conversation = await dropping.ai.realtime({ output: ['text'] });
|
||||
const log = watch(conversation);
|
||||
try {
|
||||
await conversation.connect();
|
||||
@@ -481,7 +481,7 @@ describe('convo built SDK and browser media', () => {
|
||||
{ ...manifest, apiOrigin: proxyOrigin },
|
||||
env.users.user.token,
|
||||
);
|
||||
const conversation = await dropping.ai.convo({ output: ['text'] });
|
||||
const conversation = await dropping.ai.realtime({ output: ['text'] });
|
||||
const log = watch(conversation);
|
||||
const auth = vi.spyOn(env.server.services.auth, 'authenticate');
|
||||
const warn = vi.spyOn(console, 'warn');
|
||||
@@ -524,7 +524,7 @@ describe('convo built SDK and browser media', () => {
|
||||
{ ...manifest, apiOrigin: proxyOrigin },
|
||||
env.users.user.token,
|
||||
);
|
||||
const conversation = await dropping.ai.convo({ output: ['text'] });
|
||||
const conversation = await dropping.ai.realtime({ output: ['text'] });
|
||||
const log = watch(conversation);
|
||||
try {
|
||||
await conversation.connect();
|
||||
@@ -564,7 +564,7 @@ describe('convo built SDK and browser media', () => {
|
||||
});
|
||||
|
||||
it('moves to a new session when a restart is announced, restoring context', async () => {
|
||||
const conversation = await puter.ai.convo({ output: ['text'] });
|
||||
const conversation = await puter.ai.realtime({ output: ['text'] });
|
||||
const log = watch(conversation);
|
||||
try {
|
||||
await conversation.connect();
|
||||
@@ -609,7 +609,7 @@ describe('convo built SDK and browser media', () => {
|
||||
});
|
||||
|
||||
it('closes with invalid_event when Puter refuses a native event', async () => {
|
||||
const conversation = await puter.ai.convo({ output: ['text'] });
|
||||
const conversation = await puter.ai.realtime({ output: ['text'] });
|
||||
const log = watch(conversation);
|
||||
try {
|
||||
await conversation.connect();
|
||||
@@ -646,13 +646,13 @@ describe('convo built SDK and browser media', () => {
|
||||
});
|
||||
const page = await browser.newPage();
|
||||
try {
|
||||
await page.route('**/__convo_mic_test__', (route) =>
|
||||
await page.route('**/__realtime_mic_test__', (route) =>
|
||||
route.fulfill({
|
||||
contentType: 'text/html',
|
||||
body: '<!doctype html><title>Microphone reconnect test</title>',
|
||||
}),
|
||||
);
|
||||
await page.goto(`${env.apiOrigin}/__convo_mic_test__`);
|
||||
await page.goto(`${env.apiOrigin}/__realtime_mic_test__`);
|
||||
// Through the proxy, so the test can drop the connection.
|
||||
await page.evaluate(
|
||||
({ origin, apiOrigin }) => {
|
||||
@@ -695,7 +695,7 @@ describe('convo built SDK and browser media', () => {
|
||||
tracks.push(...stream.getTracks());
|
||||
return stream;
|
||||
};
|
||||
const c = await g.puter.ai.convo({ output: ['text'] });
|
||||
const c = await g.puter.ai.realtime({ output: ['text'] });
|
||||
const events: string[] = [];
|
||||
c.on('*', (event) => events.push(event.type));
|
||||
await c.connect();
|
||||
@@ -800,7 +800,7 @@ describe('convo built SDK and browser media', () => {
|
||||
{ ...manifest, apiOrigin: proxyOrigin },
|
||||
env.users.user.token,
|
||||
);
|
||||
const conversation = await draining.ai.convo({ output: ['text'] });
|
||||
const conversation = await draining.ai.realtime({ output: ['text'] });
|
||||
const log = watch(conversation);
|
||||
try {
|
||||
await conversation.connect();
|
||||
+7
-7
@@ -2,7 +2,7 @@ import WebSocket from 'ws';
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
|
||||
import { closeNodePuter, loadNodePuter } from '../harness/nodeSdkLoader';
|
||||
import type { EnvManifest, PuterSDK } from '../harness/types';
|
||||
import type { Conversation } from '../../../types/modules/ai/convo';
|
||||
import type { Conversation } from '../../../types/modules/ai/realtime';
|
||||
import type { RealtimeServerEvent } from '../../../types/modules/ai/types';
|
||||
import { optionalEnv } from '@heyputer/backend/drivers/integrationTestUtil.ts';
|
||||
import {
|
||||
@@ -146,7 +146,7 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
|
||||
'batchIncrementUsages',
|
||||
);
|
||||
const startedAt = performance.now();
|
||||
const session = await puter.ai.convo({
|
||||
const session = await puter.ai.realtime({
|
||||
...(model ? { model } : {}),
|
||||
output: ['text'],
|
||||
instructions: 'Reply with exactly realtime-ok',
|
||||
@@ -188,7 +188,7 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
|
||||
.map((result) => result.value),
|
||||
);
|
||||
process.stdout.write(
|
||||
`[live convo] model=${model ?? 'default'} ready=${Math.round(readyAt - startedAt)}ms firstText=${Math.round(firstDeltaAt! - readyAt)}ms reply=${JSON.stringify(text)} metering=recorded\n`,
|
||||
`[live realtime] model=${model ?? 'default'} ready=${Math.round(readyAt - startedAt)}ms firstText=${Math.round(firstDeltaAt! - readyAt)}ms reply=${JSON.stringify(text)} metering=recorded\n`,
|
||||
);
|
||||
} finally {
|
||||
session.close();
|
||||
@@ -199,7 +199,7 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
|
||||
45_000,
|
||||
);
|
||||
it('restores tool and text context on a new provider session after a restart notice', async () => {
|
||||
const session = await puter.ai.convo({
|
||||
const session = await puter.ai.realtime({
|
||||
output: ['text'],
|
||||
instructions: 'Answer in one short sentence.',
|
||||
});
|
||||
@@ -272,7 +272,7 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
|
||||
expect(text.toLowerCase()).toContain('pineapple');
|
||||
expect(errors).toEqual([]);
|
||||
process.stdout.write(
|
||||
`[live convo] context restored after reconnect, reply=${JSON.stringify(text)}\n`,
|
||||
`[live realtime] context restored after reconnect, reply=${JSON.stringify(text)}\n`,
|
||||
);
|
||||
} finally {
|
||||
session.close();
|
||||
@@ -281,7 +281,7 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
|
||||
}, 60_000);
|
||||
|
||||
it('receives live PCM audio and interrupts the current response', async () => {
|
||||
const c = await puter.ai.convo({
|
||||
const c = await puter.ai.realtime({
|
||||
output: ['audio'],
|
||||
instructions: 'Count slowly from one to fifty.',
|
||||
});
|
||||
@@ -301,7 +301,7 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
|
||||
const response = await done;
|
||||
expect(response.response).toMatchObject({ status: 'cancelled' });
|
||||
process.stdout.write(
|
||||
'[live convo] PCM audio received, response canceled, playback history truncated\n',
|
||||
'[live realtime] PCM audio received, response canceled, playback history truncated\n',
|
||||
);
|
||||
} finally {
|
||||
c.close();
|
||||
@@ -1743,10 +1743,10 @@ export default suite('ai', {
|
||||
t.assert.equal(error.message, 'Input size cannot be larger than 25 MB');
|
||||
},
|
||||
|
||||
'convo streams a live text turn': {
|
||||
'realtime streams a live text turn': {
|
||||
requires: ['ai.openai'],
|
||||
fn: async (t) => {
|
||||
const conversation = await t.puter.ai.convo({ output: ['text'], instructions: 'Reply with exactly realtime-ok.' });
|
||||
const conversation = await t.puter.ai.realtime({ output: ['text'], instructions: 'Reply with exactly realtime-ok.' });
|
||||
let text = '';
|
||||
conversation.on('text', ({ delta }) => { text += delta; });
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
@@ -1767,8 +1767,8 @@ export default suite('ai', {
|
||||
},
|
||||
},
|
||||
|
||||
'convo prepares and closes without opening a provider session': async (t) => {
|
||||
const conversation = await t.puter.ai.convo({ output: ['text'] });
|
||||
'realtime prepares and closes without opening a provider session': async (t) => {
|
||||
const conversation = await t.puter.ai.realtime({ output: ['text'] });
|
||||
t.assert.equal(conversation.state, 'prepared');
|
||||
t.assert.equal(conversation.model, 'gpt-realtime-2.1-mini');
|
||||
let closed = 0;
|
||||
@@ -1779,9 +1779,9 @@ export default suite('ai', {
|
||||
t.assert.equal(closed, 1);
|
||||
},
|
||||
|
||||
'convo rejects invalid options and sending before connect': async (t) => {
|
||||
await t.assert.rejects(() => t.puter.ai.convo({ model: 'not-a-realtime-model' }));
|
||||
const conversation = await t.puter.ai.convo({ output: ['text'] });
|
||||
'realtime rejects invalid options and sending before connect': async (t) => {
|
||||
await t.assert.rejects(() => t.puter.ai.realtime({ model: 'not-a-realtime-model' }));
|
||||
const conversation = await t.puter.ai.realtime({ output: ['text'] });
|
||||
try {
|
||||
await t.assert.rejects(async () => conversation.sendText('Hello'));
|
||||
} finally {
|
||||
@@ -1789,8 +1789,8 @@ export default suite('ai', {
|
||||
}
|
||||
},
|
||||
|
||||
'convo rejects interruption readiness through its Promise': async (t) => {
|
||||
const conversation = await t.puter.ai.convo({ output: ['text'] });
|
||||
'realtime rejects interruption readiness through its Promise': async (t) => {
|
||||
const conversation = await t.puter.ai.realtime({ output: ['text'] });
|
||||
try {
|
||||
const prepared = conversation.interrupt();
|
||||
t.assert.equal(typeof prepared.then, 'function');
|
||||
@@ -1804,9 +1804,9 @@ export default suite('ai', {
|
||||
}
|
||||
},
|
||||
|
||||
'convo is the realtime SDK entry point': async (t) => {
|
||||
'realtime is the only realtime SDK entry point': async (t) => {
|
||||
t.assert.equal('assistant' in t.puter.ai, false);
|
||||
t.assert.equal('conversation' in t.puter.ai, false);
|
||||
t.assert.equal(typeof t.puter.ai.convo, 'function');
|
||||
t.assert.equal(typeof t.puter.ai.realtime, 'function');
|
||||
},
|
||||
});
|
||||
Reference in new issue
Block a user