Compare commits

...
Author SHA1 Message Date
jelveh 74ef0ac959 Drop the alert after a provider signup lands on an existing account
Signing up with a provider whose account already exists signs the user
in to it, then showed a blocking "You have been logged in to an existing
account" alert. In a sign-in popup it held back the token until the user
clicked OK, right before the popup closed. The sign-in now completes
without it, and the callback no longer adds oidc_switched to the return
URL.
2026-11-07 17:47:55 -07:00
Nariman Jelveh 8c14e7de30 Add account links and Send feedback to <puter-user-menu> (#4536)
The signed-in menu now shows the user's picture, username and email
(hidden when whoami leaves the email out), links to Manage account,
Usage and Open Puter on the GUI origin in a new tab, and Send feedback,
which calls puter.ui.showFeedbackDialog(). Each new item takes a label
attribute like the existing ones. The menu opens on its first item, and
the arrow, Home and End keys move between items.
2026-11-07 11:40:42 -07:00
Filip Kujundžić 84faf5c40c refactor(ai): rename convo API to realtime (#4524) 2026-11-07 10:55:35 -07:00
Rasad Regmi aba2666f26 fix(gui): improve dark mode input visibility (#4450) 2026-11-07 09:32:33 -07:00
tgarver-gdm abe6882de9 Import AGENTS.md from CLAUDE.md so Claude Code loads it (#4529) 2026-11-07 09:21:28 -07:00
jelveh 770af49528 Show a loading scrim while the feedback embed checks the session
The framed feedback dialog stayed hidden until the embed had checked the
GUI session, so on a slow network a click on Send feedback showed
nothing for seconds, and a second click silently resolved false. A
check that takes longer than 200ms now shows a scrim with a spinner over
the page, matching the embed's own scrim, which takes its place once
the dialog is ready. Escape or a click on it cancels, and a refused
session still falls back to the popup.
2026-11-07 02:12:59 -07:00
jelveh 47b4cc1158 Follow the OS theme in the feedback embed on Safari
WebKit gives a frame the color-scheme of its iframe element as its
preferred scheme, and the feedback embed's frame was set to normal, so
Safari in dark mode showed the light dialog. The frame and the embed's
root now both declare light dark: the embed follows the OS in every
engine, and the frame stays transparent because the two match.
2026-11-07 02:12:06 -07:00
Nariman Jelveh e253a7f862 Show the app feedback dialog in a frame on first-party pages (#4534)
* Show the app feedback dialog in a frame on first-party pages

puter.ui.showFeedbackDialog() on a page one label under the Puter origin
now frames /embed/feedback in a modal over the page, instead of opening a
puter.com popup. The embed uses the same framing rules, query redirect,
and host-user binding as /embed/apps, and sends the feedback for the
framing page's origin, which the server reads from the referrer and pins
with frame-ancestors. Without the named user's Puter session, or with an
email still to confirm, the embed reports itself unavailable and the SDK
falls back to the popup, which can sign in. A frame that never answers
(such as one the host's CSP blocks) falls back too.

The embed session check moves out of appsEmbed.js into a shared helper,
and the first-party page check out of <puter-user-menu> into puter.js's
lib, so both embeds use one copy of each.

* Keep the framing page from scrolling behind the feedback embed

A wheel or touch scroll over the feedback embed's scrim chained out of
the frame and scrolled the first-party page underneath the open dialog.
The embed's root now stops overscroll from leaving the frame.

* Name the feedback embed's user from the cached whoami

showFeedbackDialog() on a first-party page fetched /whoami before it
started loading the embed, adding a round trip to every call. It now
uses the user puter.js already cached for the session and asks the API
only when there is none.
2026-11-07 01:57:47 -07:00
53 changed files with 1892 additions and 650 deletions

No files matched your search

@@ -53,7 +53,7 @@ jobs:
# step above fails; the tests skip themselves without the key.
- name: Run live realtime conversation tests
if: ${{ !cancelled() }}
run: npm run setupExtensions && (cd src/puter-js && npm run build) && npx vitest run --config src/puter-js/tests/api/vitest.config.ts src/puter-js/tests/api/realtime/convoLive.test.ts
run: npm run setupExtensions && (cd src/puter-js && npm run build) && npx vitest run --config src/puter-js/tests/api/vitest.config.ts src/puter-js/tests/api/realtime/realtimeLive.test.ts
env:
CI: 'true'
PUTER_TEST_AI_OPENAI_API_KEY: ${{ secrets.PUTER_TEST_AI_OPENAI_API_KEY }}
+1 -1
View File
@@ -1 +1 @@
FOLLOW ./AGENTS.md
FOLLOW @AGENTS.md
+28 -3
View File
@@ -124,7 +124,7 @@ localization system.
</script>
```
## Implementation and checks
## Implementation
`createAppBrowser` in `src/gui/src/UI/Dashboard/AppBrowser.js` owns the shared UI
and data behavior; its `guestApps` option is the signed-out list. `TabApps.js`
@@ -134,14 +134,39 @@ injects desktop actions. `appsEmbed.js` supplies new-tab launching through
`launch_app`, and loads only the embed entry, its SDK, and shared styles. `embedHostUser` in
`src/gui/src/helpers/embedHostUser.js` reads the host's user from the fragment, and
`embedHostTheme.js` beside it applies the host's theme.
The normal GUI webpack build emits `bundle.min.js` and `apps-embed.min.js`.
The normal GUI webpack build emits `bundle.min.js`, `apps-embed.min.js`, and `feedback-embed.min.js`.
The `/embed/apps` route puts the recommended list in the page's config as
`guestApps`. No puter.js API or app-list endpoint changes are needed.
## Feedback dialog
`GET /embed/feedback` is the app feedback dialog (`puter.ui.showFeedbackDialog()`)
for a first-party page, with the same framing rules, query redirect, and
host-user binding as `/embed/apps`. It sends the feedback for the framing
page's origin, which the server reads from the referrer and pins with
`frame-ancestors`; opened top-level, it does nothing. Its page is transparent
so the dialog's own scrim covers the host.
puter.js uses it on a page one label under the GUI origin when the page is
signed in, framing it in a modal `<dialog>` that stays hidden until the embed
is ready. A session check that takes longer than 200ms shows a loading scrim
with a spinner in its place; Escape or a click on it cancels. The embed posts `{ msg: 'feedbackEmbed', state }` to its parent
(`loading`, then `ready` or `unavailable`) and finally
`{ msg: 'feedbackDialogClosed', sent }`. It reports `unavailable` without the
named user's session, or while that user's email still needs confirming, and
puter.js then uses the `/action/send-feedback` popup, which can sign in. So does
a frame that never answers, such as one the host's CSP blocks. `feedbackEmbed.js`
is the entry, sharing `embedSession` in `src/gui/src/helpers/embedSession.js`
with the app browser; `showFeedbackEmbed` in `src/puter-js/src/lib/guiEmbed.js`
is the host side.
## Checks
Run the GUI component/browser tests (with the SDK bundle built and Playwright
Chromium installed), and the backend route tests:
```sh
npx vitest run --config src/gui/vitest.config.js src/gui/src/UI/Dashboard/AppBrowser
npx vitest run --config src/gui/vitest.config.js src/gui/src/UI/Dashboard/AppBrowser src/feedbackEmbed
npx vitest run src/puter-js/src/lib/guiEmbed.test.js src/puter-js/src/modules/UI.showFeedbackDialog.test.js
npx vitest run --config src/backend/vitest.config.ts src/backend/server.test.ts
```
@@ -15,7 +15,6 @@ export interface OidcLoginContinuation {
embeddedInPopup: boolean;
openerOrigin: string | null;
messageId: string | null;
switchedToLogin: boolean;
}
interface OidcReturnDeps {
@@ -56,7 +55,6 @@ export function completeOidcLoginReturn(
data.userUuid !== user.uuid ||
typeof data.redirectUri !== 'string' ||
typeof data.embeddedInPopup !== 'boolean' ||
typeof data.switchedToLogin !== 'boolean' ||
(data.openerOrigin !== null && typeof data.openerOrigin !== 'string') ||
(data.messageId !== null && typeof data.messageId !== 'string')
) {
@@ -91,6 +89,5 @@ export function completeOidcLoginReturn(
),
);
}
if (data.switchedToLogin) append('oidc_switched', 'login');
return target;
}
@@ -27,7 +27,7 @@ import type { Actor } from '../../core/actor.js';
import { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { embedFrameAncestors } from './HomepageController.js';
import { embedFrameAncestors, embedHostOrigin } from './HomepageController.js';
// ── Test harness ────────────────────────────────────────────────────
//
@@ -550,3 +550,26 @@ describe('embedFrameAncestors', () => {
).toBe("'self'");
});
});
describe('embedHostOrigin', () => {
const config = {
origin: 'https://example.test',
domain: 'example.test',
api_base_url: 'https://backend.example.test',
static_hosting_domain: 'site.example.test',
};
it.each([
['https://sheets.example.test/doc/1', 'https://sheets.example.test'],
['https://backend.example.test/x', null],
['https://api.example.test/', null],
['https://someone.site.example.test/', null],
['https://a.sheets.example.test/', null],
['http://sheets.example.test/', null],
['https://example.test/', null],
['not a url', null],
[undefined, null],
])('%s frames as %s', (referer, expected) => {
expect(embedHostOrigin(config, referer)).toBe(expected);
});
});
@@ -61,32 +61,34 @@ export function appMetaDescription(text: string): string {
return `${cut.replace(/[\s,;:.!?…—-]+$/, '')}…`;
}
type EmbedFramingConfig = Pick<
IConfig,
| 'origin'
| 'domain'
| 'api_base_url'
| 'static_hosting_domain'
| 'static_hosting_domain_alt'
| 'private_app_hosting_domain'
| 'private_app_hosting_domain_alt'
>;
/**
* `frame-ancestors` for the app browser embed. A wildcard over `*.<domain>`
* The origin of a first-party page that may frame an `/embed/*` page, named
* exactly from the request's referrer, or null. A wildcard over `*.<domain>`
* would also admit user-hosted sites whenever the hosting domains sit under it,
* so the embedding page is named exactly, from the request's referrer. Only a
* single label under `<domain>` qualifies, and not one this server answers
* itself (the API, WebDAV, SDK and native-app hosts).
* so only a single label under `<domain>` qualifies, and not one this server
* answers itself (the API, WebDAV, SDK and native-app hosts).
*/
export function embedFrameAncestors(
config: Pick<
IConfig,
| 'origin'
| 'domain'
| 'api_base_url'
| 'static_hosting_domain'
| 'static_hosting_domain_alt'
| 'private_app_hosting_domain'
| 'private_app_hosting_domain_alt'
>,
export function embedHostOrigin(
config: EmbedFramingConfig,
referer: string | undefined,
): string {
): string | null {
const gui = new URL(config.origin ?? `https://${config.domain}`);
let parent: URL;
try {
parent = new URL(referer ?? '');
} catch {
return "'self'";
return null;
}
const host = parent.hostname.toLowerCase();
const label = subdomainLabelOf(host, config.domain);
@@ -105,7 +107,16 @@ export function embedFrameAncestors(
!isReservedSubdomain(label) &&
host !== new URL(config.api_base_url).hostname &&
!hostingDomains.some((d) => host === d || host.endsWith(`.${d}`));
return allowed ? `'self' ${parent.origin}` : "'self'";
return allowed ? parent.origin : null;
}
/** `frame-ancestors` for an `/embed/*` page: the GUI and {@link embedHostOrigin}. */
export function embedFrameAncestors(
config: EmbedFramingConfig,
referer: string | undefined,
): string {
const host = embedHostOrigin(config, referer);
return host ? `'self' ${host}` : "'self'";
}
/**
@@ -157,12 +168,20 @@ export class HomepageController extends PuterController {
router.get('/', {}, (req, res) => sendShell(req, res));
router.get('/embed/apps', {}, async (req, res) => {
// SDK bootstrap parameters must not change this page's account or API origin.
// Checked on the raw URL; `req.query` drops pairs past the parser's limit.
/**
* Headers shared by the `/embed/*` pages; false when the request was
* redirected instead. SDK bootstrap parameters must not change an
* embed's account or API origin, so a query string is dropped. Checked
* on the raw URL; `req.query` drops pairs past the parser's limit.
*/
const prepareEmbed = (
req: express.Request,
res: express.Response,
path: string,
): boolean => {
if (req.originalUrl.includes('?')) {
res.redirect('/embed/apps');
return;
res.redirect(path);
return false;
}
res.removeHeader('X-Frame-Options');
res.setHeader(
@@ -170,6 +189,11 @@ export class HomepageController extends PuterController {
`frame-ancestors ${embedFrameAncestors(this.config, req.get('referer'))}`,
);
res.setHeader('Cache-Control', 'no-store');
return true;
};
router.get('/embed/apps', {}, async (req, res) => {
if (!prepareEmbed(req, res, '/embed/apps')) return;
// Shown to visitors without a session; a signed-in embed fetches
// its own list, so a failure here only costs signed-out visitors.
const guestApps = await this.services.recommendedApps
@@ -178,6 +202,20 @@ export class HomepageController extends PuterController {
res.send(homepage.renderAppBrowserEmbed({ guestApps }));
});
router.get('/embed/feedback', {}, (req, res) => {
if (!prepareEmbed(req, res, '/embed/feedback')) return;
// The feedback is recorded against this origin, which
// frame-ancestors has just pinned to the actual framing page.
res.send(
homepage.renderFeedbackEmbed({
hostOrigin: embedHostOrigin(
this.config,
req.get('referer'),
),
}),
);
});
router.get('/settings', {}, (req, res) => sendShell(req, res));
router.get('/settings/*splat', {}, (req, res) => sendShell(req, res));
@@ -1620,7 +1620,7 @@ describe('OIDCController signup callback', () => {
expect(captured.redirectUrl).toContain('action=signup');
});
it('creates a fresh user without oidc_switched on first signup', async () => {
it('creates a fresh user on first signup', async () => {
const state = oidc().signState({
provider: 'custom',
redirect_uri: TEST_ORIGIN + '/',
@@ -1646,11 +1646,9 @@ describe('OIDCController signup callback', () => {
);
expect(captured.cookies).toHaveLength(1);
expect(captured.redirectUrl).toBe(TEST_ORIGIN + '/');
// Fresh creation — must NOT advertise "you've actually been signed in".
expect(captured.redirectUrl).not.toContain('oidc_switched=login');
});
it('adds oidc_switched=login when signup hits an already-linked account', async () => {
it('signs in to an already-linked account on signup, returning as a login would', async () => {
// Seed an account linked on (provider, sub) so the signup callback
// resolves to `linked-sub` rather than creating a new user.
const sub = `sub-${Math.random().toString(36).slice(2, 8)}`;
@@ -1686,7 +1684,7 @@ describe('OIDCController signup callback', () => {
res,
);
expect(captured.cookies).toHaveLength(1);
expect(captured.redirectUrl).toContain('oidc_switched=login');
expect(captured.redirectUrl).toBe(TEST_ORIGIN + '/');
});
it('redirects to action=signup with auth_error when user resolution fails (unconfirmed email)', async () => {
+5 -17
View File
@@ -688,14 +688,7 @@ export class OIDCController extends PuterController {
);
}
// If we landed on an existing account (either via the
// provider_sub or via email match), signal the GUI so it can
// render a "signed in" flow rather than "account created".
const extra =
resolved.origin === 'created'
? undefined
: { oidc_switched: 'login' };
await this.#finishLogin(req, res, user, stateDecoded, extra);
await this.#finishLogin(req, res, user, stateDecoded);
};
router.get('/auth/oidc/callback/signup', cbOpts, signupCb);
router.post('/auth/oidc/callback/signup', cbOpts, signupCb);
@@ -834,17 +827,14 @@ if (window.opener) {
attempt = 0,
): Promise<
| { error: string; code?: string; requestCode?: string }
| {
user: import('../../stores/user/UserStore.js').UserRow;
origin: 'linked-sub' | 'linked-email' | 'created';
}
| { user: import('../../stores/user/UserStore.js').UserRow }
> {
// 1. Existing provider/sub link.
const linked = await this.services.oidc.findUserByProviderSub(
provider,
userinfo.sub,
);
if (linked) return { user: linked, origin: 'linked-sub' };
if (linked) return { user: linked };
// 2/3. Email match branch.
const claimedEmail =
@@ -881,7 +871,7 @@ if (window.opener) {
error: outcome.error ?? 'Failed to link provider.',
};
}
return { user: byEmail, origin: 'linked-email' };
return { user: byEmail };
}
}
@@ -911,7 +901,7 @@ if (window.opener) {
requestCode: outcome.requestCode,
};
}
return { user: outcome.user, origin: 'created' };
return { user: outcome.user };
}
async #processCallback(
@@ -1000,7 +990,6 @@ if (window.opener) {
res: Response,
user: UserRow,
stateDecoded: Record<string, unknown>,
extraQueryParams?: Record<string, string>,
): Promise<void> {
const origin = (this.config.origin ?? '').replace(/\/$/, '');
const continuation: OidcLoginContinuation = {
@@ -1018,7 +1007,6 @@ if (window.opener) {
stateDecoded.msg_id == null
? null
: String(stateDecoded.msg_id),
switchedToLogin: extraQueryParams?.oidc_switched === 'login',
};
if (user.otp_enabled) {
const challenge = this.services.token.sign(
@@ -139,7 +139,7 @@ describe('OIDC account second-factor login', () => {
expect(await response.json()).toMatchObject({
proceed: true,
next_step: 'complete',
redirectUrl: `${env.origin}/dashboard${flow === 'signup' ? '?oidc_switched=login' : ''}`,
redirectUrl: `${env.origin}/dashboard`,
});
expect(
response.headers
+40
View File
@@ -368,6 +368,46 @@ describe('PuterServer host header validation', () => {
expect(api.status).toBe(404);
});
it('lets a deployment subdomain frame the feedback embed and names it', async () => {
const parent = `http://spreadsheet.puter.localhost:${port}`;
const res = await request('/embed/feedback', {
host: `puter.localhost:${port}`,
referer: `${parent}/sheet/1`,
});
expect(res.status).toBe(200);
expect(res.headers['x-frame-options']).toBeUndefined();
expect(res.headers['content-security-policy']).toBe(
`frame-ancestors 'self' ${parent}`,
);
expect(res.headers['cache-control']).toBe('no-store');
expect(res.body).toContain('/dist/feedback-embed.min.js');
expect(res.body).not.toContain('/dist/bundle.min.js');
const config = JSON.parse(
res.body.match(/id="feedback-embed-config">(.*?)<\/script>/)[1],
);
expect(config).toEqual({
apiOrigin: `http://api.puter.localhost:${port}`,
hostOrigin: parent,
});
const hosted = await request('/embed/feedback', {
host: `puter.localhost:${port}`,
referer: `http://someone.site.puter.localhost:${port}/`,
});
expect(hosted.headers['content-security-policy']).toBe("frame-ancestors 'self'");
expect(
JSON.parse(
hosted.body.match(/id="feedback-embed-config">(.*?)<\/script>/)[1],
).hostOrigin,
).toBeNull();
const redirected = await request('/embed/feedback?puter.api_origin=https://other.test', {
host: `puter.localhost:${port}`,
});
expect(redirected.status).toBe(302);
expect(redirected.headers.location).toBe('/embed/feedback');
});
it.each([
['a user-hosted site', 'http://someone.site.puter.localhost:PORT/'],
['a private app host', 'http://someone.app.puter.localhost:PORT/'],
@@ -107,6 +107,31 @@ describe('app browser embed shell', () => {
});
});
describe('feedback embed shell', () => {
it('carries the framing page and API origin, and loads only its own script', () => {
const html = makeService({
api_base_url: 'https://api.deployment.test',
gui_bundle: '/static/bundle.min.js?v=42',
}).renderFeedbackEmbed({ hostOrigin: 'https://notes.deployment.test' });
const configJson = html.match(/id="feedback-embed-config">(.*?)<\/script>/)![1];
expect(JSON.parse(configJson)).toEqual({
apiOrigin: 'https://api.deployment.test',
hostOrigin: 'https://notes.deployment.test',
});
expect(html).toContain('<script defer src="/static/feedback-embed.min.js?v=42"></script>');
expect(html).toContain('<link rel="stylesheet" href="/css/feedback-embed.css">');
expect(html).toContain('<main id="feedback-embed"></main>');
expect(html).not.toContain('bundle.min.js');
expect(html).not.toContain('apps-embed');
});
it('carries no framing page when none is allowed', () => {
const html = makeService({}).renderFeedbackEmbed({ hostOrigin: null });
const configJson = html.match(/id="feedback-embed-config">(.*?)<\/script>/)![1];
expect(JSON.parse(configJson).hostOrigin).toBeNull();
});
});
/** Capture what the service sends, and return the rendered HTML. */
const render = async (
service: PuterHomepageService,
@@ -171,15 +171,50 @@ export class PuterHomepageService extends PuterService {
renderAppBrowserEmbed({
guestApps = [],
}: { guestApps?: Array<Record<string, unknown>> } = {}): string {
const configJson = JSON.stringify({
apiOrigin: this.config.api_base_url,
sdkUrl:
this.config.env === 'dev'
? '/sdk/puter.dev.js'
: (this.config.gui_puterjs_bundle ??
'https://js.puter.com/v2/'),
guestApps,
}).replace(/</g, '\\u003c');
return this.#renderEmbed({
rootId: 'app-browser',
rootClass: 'dashboard',
stylesheet: 'apps-embed.css',
script: 'apps-embed.min.js',
config: {
apiOrigin: this.config.api_base_url,
sdkUrl:
this.config.env === 'dev'
? '/sdk/puter.dev.js'
: (this.config.gui_puterjs_bundle ??
'https://js.puter.com/v2/'),
guestApps,
},
});
}
/**
* `hostOrigin` is the first-party page framing the dialog, which the
* feedback is sent for; null when no allowed page frames it.
*/
renderFeedbackEmbed({ hostOrigin }: { hostOrigin: string | null }): string {
return this.#renderEmbed({
rootId: 'feedback-embed',
stylesheet: 'feedback-embed.css',
script: 'feedback-embed.min.js',
config: { apiOrigin: this.config.api_base_url, hostOrigin },
});
}
#renderEmbed({
rootId,
rootClass,
stylesheet,
script,
config,
}: {
rootId: string;
rootClass?: string;
stylesheet: string;
script: string;
config: Record<string, unknown>;
}): string {
const configJson = JSON.stringify(config).replace(/</g, '\\u003c');
return `<!DOCTYPE html>
<html lang="en">
@@ -190,27 +225,27 @@ export class PuterHomepageService extends PuterService {
<title>${encode(String(this.config.gui_params?.title ?? 'Puter'))}</title>
<link rel="stylesheet" href="/css/normalize.css">
<link rel="stylesheet" href="/css/dashboard.css">
<link rel="stylesheet" href="/css/apps-embed.css">
<link rel="stylesheet" href="/css/${stylesheet}">
</head>
<body>
<main id="app-browser" class="dashboard"></main>
<script type="application/json" id="app-browser-config">${configJson}</script>
<script defer src="${encode(this.#appBrowserEmbedScript())}"></script>
<main id="${rootId}"${rootClass ? ` class="${rootClass}"` : ''}></main>
<script type="application/json" id="${rootId}-config">${configJson}</script>
<script defer src="${encode(this.#embedScript(script))}"></script>
</body>
</html>`;
}
/**
* The embed's script is built beside the GUI bundle, so it is served from
* Embed scripts are built beside the GUI bundle, so they are served from
* wherever `gui_bundle` is (a CDN in production), with the same query.
*/
#appBrowserEmbedScript(): string {
#embedScript(name: string): string {
const base = 'http://relative.invalid';
const bundle = new URL(
this.config.gui_bundle ?? '/dist/bundle.min.js',
base,
);
const script = new URL('apps-embed.min.js', bundle);
const script = new URL(name, bundle);
script.search = bundle.search;
return script.origin === base
? `${script.pathname}${script.search}`
+3 -3
View File
@@ -1,11 +1,11 @@
# OpenAI Realtime relay
The relay serves `/puterai/openai/v1/realtime` on the API host. The SDK exposes
it through [`puter.ai.convo()`](../../../docs/src/AI/convo.md).
it through [`puter.ai.realtime()`](../../../docs/src/AI/realtime.md).
**Beta:** This API is incomplete. Its methods, options, events, and behavior may change abruptly, including breaking changes without notice.
Direct clients are documented in the [Realtime WebSocket API reference](../../../docs/src/AI/realtime.md).
Direct clients are documented in the [Realtime WebSocket API reference](../../../docs/src/AI/realtime-websocket.md).
## Transport
@@ -27,7 +27,7 @@ The socket opens before authentication completes. Authentication and model
failures arrive as OpenAI-shaped error events followed by a close frame.
```js
const conversation = await puter.ai.convo({ output: ['text'] });
const conversation = await puter.ai.realtime({ output: ['text'] });
conversation.on('error', (event) => console.error(event));
conversation.on('text', ({ delta }) => console.log(delta));
conversation.on('turn.done', () => conversation.close());
+3 -2
View File
@@ -1102,8 +1102,9 @@ interface IConfigOptional {
/** Force the bundled GUI even in dev. Default: false. */
use_bundled_gui: boolean;
/**
* Override the GUI bundle JS path. Default: `/dist/bundle.min.js`. The app
* browser embed loads `apps-embed.min.js` from the same directory.
* Override the GUI bundle JS path. Default: `/dist/bundle.min.js`. The
* `/embed/*` pages load `apps-embed.min.js` and `feedback-embed.min.js`
* from the same directory.
*/
gui_bundle: string;
/** Override the GUI CSS path when bundled. Default: `/dist/bundle.min.css`. */
+2 -2
View File
@@ -200,8 +200,8 @@ These AI features are supported out of the box when using Puter.js:
- **[`puter.ai.img2txt()`](/AI/img2txt/)** - Extract text from images (OCR)
- **[`puter.ai.txt2vid()`](/AI/txt2vid/)** - Generate short video clips from text or a reference image with Wan, Seedance, Veo, Kling, MiniMax H3 and other models, or edit a video you supply with Gemini Omni, FLUX 3 or Kling Omni
- **[`puter.ai.speech2speech()`](/AI/speech2speech/)** - Convert speech in one voice to another voice
- **[`puter.ai.convo()`](/AI/convo/)** - Prepare a beta realtime conversation with optional text, microphone, playback, and interruption helpers
- **[Realtime WebSocket API](/AI/realtime/)** - Connect directly without puter.js. This API is in beta and may change abruptly.
- **[`puter.ai.realtime()`](/AI/realtime/)** - Prepare a beta realtime conversation with optional text, microphone, playback, and interruption helpers
- **[Realtime WebSocket API](/AI/realtime-websocket/)** - Connect directly without puter.js. This API is in beta and may change abruptly.
- **[`puter.ai.listModels()`](/AI/listModels/)** - List available AI chat models (and providers) that Puter currently exposes.
- **[`puter.ai.listModelProviders()`](/AI/listModelProviders/)** - List the AI providers that Puter currently exposes.
- **[`puter.ai.txt2speech.listEngines()`](/AI/txt2speech.listEngines/)** - List available TTS engines/models
-151
View File
@@ -1,151 +0,0 @@
---
title: puter.ai.convo()
description: Prepare an OpenAI realtime conversation with optional audio and text helpers.
platforms: [websites, apps, nodejs, workers]
---
Prepares a realtime OpenAI conversation that your app can connect to when it's ready.
Preparing the conversation verifies Puter sign-in. Realtime conversations require a paid Puter plan, and usage is billed to the signed-in user. Native OpenAI events and convenience methods use the same session. Microphone capture and speaker playback start when your app requests them.
In Node.js, the SDK requires a global `WebSocket`, available in Node.js 22 or later.
**Beta:** This API is incomplete. Its methods, options, events, and behavior may change abruptly, including breaking changes without notice.
To connect without puter.js, see the [Realtime WebSocket API](/AI/realtime/).
## Syntax
```js
const conversation = await puter.ai.convo(options = {});
await conversation.connect();
```
## Parameters
#### `options` (Object) (Optional)
The options object accepts these fields.
| Option | Type | Description |
|--------|------|-------------|
| `model` | `String` | The realtime model. Defaults to `gpt-realtime-2.1-mini`. Also accepts `gpt-realtime-2.1`, `gpt-realtime-2`, `gpt-realtime-1.5`, `gpt-realtime`, and `gpt-realtime-mini`. |
| `instructions` | `String` | Session instructions, up to 8192 characters. Defaults to the provider's default. |
| `output` | `Array<String>` | Accepts `['text']`, `['audio']`, or both. Defaults to `['audio']`. Audio includes a transcript. Requesting both uses audio output and emits transcript deltas through the `text` event. |
| `voice` | `String` | OpenAI voice identifier, up to 128 characters. Defaults to the provider's default. |
| `turnDetection` | `Boolean` | Whether the provider detects the end of spoken turns automatically. Defaults to `true`. Set to `false` for manual audio commits. |
## Return value
A Promise resolves to a prepared conversation after options and sign-in are checked. In a browser, preparation opens the Puter sign-in prompt if needed. Preparation opens no realtime connection and requests no microphone access.
The conversation exposes these properties and methods.
| Member | Behavior |
| --- | --- |
| `state` | `prepared`, `connecting`, `connected`, or `closed`. |
| `model` | Resolved model name. |
| `connect()` | Opens the connection and applies settings. Resolves after the provider acknowledges session configuration. Concurrent calls share one connection Promise, including bounded retries for transient setup failures. Calling it while connected opens no additional connection. During a reconnection, it resolves when the reconnection succeeds and rejects if the conversation closes instead. |
| `configure(options)` | Updates the given session settings and resolves after the provider acknowledges this update. Settings it isn't given, including ones set through native events, are left alone. Accepts the factory options except `model`. One update can be pending at a time. Provider rules apply to changes such as switching a voice after audio has been generated. |
| `sendText(text)` | Adds a user message and requests a response. Text must be nonempty and at most 8192 characters. |
| `appendAudio(pcm)` | Appends an `Int16Array` containing PCM16, 24 kHz mono audio. A chunk contains at most 12000 samples. |
| `commitAudio()` | Commits buffered audio and requests a response. Requires turn detection to be off, through `turnDetection: false` or a native session update. The provider requires at least 100 ms of buffered audio. |
| `interrupt(position?)` | Stops built-in playback immediately and waits for cancellation and playback-history synchronization. A reply requested but not yet started is cancelled as soon as it starts. Custom players provide `{ itemId, contentIndex, playedMs }`. An idle call resolves immediately. Readiness failures reject the returned Promise with `session_not_ready` or `session_closed`. |
| `send(event)` | Sends a native OpenAI JSON event. |
| `on(type, handler)` | Registers a listener and returns an unsubscribe function. Use `'*'` to receive every native and convenience event. |
| `off(type, handler)` | Removes a listener. |
| `close()` | Closes the session, rejects pending operations, and releases SDK-owned capture and playback resources. Repeated calls have no additional effect. |
Attach listeners before connecting to receive initial session events. Closing during connection rejects the pending attempt.
If the connection drops because of network loss, a server restart or a provider error, the conversation reconnects on its own. Normal closes with code 1000 end the conversation. While it reconnects, `state` is `connecting`, a `reconnecting` event fires, and pending operations reject with `connection_lost`. The new session gets every setting the provider acknowledged, from the factory options, `configure()` and native `session.update` events. Then `ready` fires again with `reconnected: true`. Listeners, the microphone and the speaker carry over. Microphone audio captured while reconnecting is dropped.
The conversation also restores recent context. Before `ready` fires, it adds back up to 40 of the most recent conversation items, 48 KiB in total. Restored items are user text, user speech that has a transcript, assistant replies as text, and function calls with their outputs. User speech has a transcript only when input transcription is configured through native session settings. An assistant reply that was interrupted, or still playing when the connection dropped, is cut to what the user saw or heard. The cut is estimated from how much of the audio played, so it can be off by a few words. Images, speech without a transcript, deleted items, and replies still generating when the connection dropped are left out. Restored items get new item IDs, don't fire item events, and count as input tokens on the next response. Item IDs from before the reconnect don't work in later events. Restoration uses a contiguous suffix of restorable items. An oversized newest item leaves that suffix empty. Older items left out to fit the window fire a `context_trimmed` event with the count in `omittedItems`; this is routine and not an error. A provider refusal or a failure to send the restored items fires one `error` event with code `context_restore_failed` per restoration. In both cases the conversation stays connected. Items are preserved without truncating text, function arguments or function results.
When Puter knows ahead of time that the server a session runs on will restart, it sends a `puter.reconnect` event, which also reaches your listeners. Without that notice, a restart drops the connection and the conversation reconnects as above. Within a few seconds, the conversation opens a replacement session in the background while the current one keeps working. It switches once no reply is generating or playing, no user turn is being spoken or transcribed, no function call is waiting for its output, and nothing you sent is waiting on the provider. The switch goes through the same steps as a reconnect, and the `reconnecting` event has code 1012. If the replacement can't open yet, the conversation keeps the current session and tries again later. A refusal that retrying can't fix, such as `insufficient_funds`, stops the attempts. If the server closes the session first, the conversation switches to the replacement, or reconnects if none is open.
The conversation makes up to ten attempts over about a minute. In a browser that reports being offline, it first waits up to two minutes for the connection to come back. If every attempt fails, or the server refuses with an error such as `insufficient_funds`, the conversation closes. A closed conversation cannot reconnect. To start another session, prepare a new one.
Native event definitions are listed in the [OpenAI realtime reference](https://developers.openai.com/api/reference/resources/realtime/server-events). Native events retain their provider fields. Sending native events preserves direct control of the session. Puter refuses `conversation.item.retrieve` events and MCP tools (`"type": "mcp"`) in `session.update` and `response.create` events. Sending one fires an `error` event with code `invalid_event`, and the conversation closes with code 1008. Audio helpers expect the session to use PCM16 at 24 kHz.
## Errors and limits
Failures use objects containing `code` and `message`. Preparation can reject with `invalid_argument`, `model_not_found`, `auth_required`, `auth_canceled`, or `auth_failed`. Connection can reject with `subscription_required` or `insufficient_funds`, which also show the Puter upgrade prompt. Connection and asynchronous operations can reject with provider errors, `session_closed`, `connection_lost`, `operation_timeout`, or `operation_in_progress`. Sending before readiness, or while reconnecting, throws `session_not_ready`.
Invalid input throws `invalid_argument`, `invalid_event`, or `input_too_large`. A full SDK send queue throws `queue_full`. Media helpers report `unsupported_environment`, `permission_denied`, `media_error`, `media_timeout`, or `queue_full`. Runtime media errors and `context_restore_failed` arrive through the `error` event. Provider errors also arrive as native `error` events with a nested `error` object.
Each native event is capped at 64 KiB. The SDK bounds JSON traversal, outgoing queues, acknowledgment waits, microphone work, and playback. All numeric limits are listed on [rate limits and quotas](/rate-limits-and-quotas/).
## Convenience events
The conversation emits these events alongside native events.
| Event | Fields |
| --- | --- |
| `ready` | `model`, `provider`, and `reconnected: true` after a reconnection |
| `reconnecting` | `code`, `reason` of the dropped connection. After a `puter.reconnect` event, `code` is 1012. |
| `context_trimmed` | `omittedItems`, the number of restorable items left out of the restored context after a reconnection. Fires before `ready`. |
| `text` | `delta`, `responseId` |
| `transcript` | `role`, `text`, `final`. Assistant deltas use `final: false`. Completed assistant and user transcripts use `final: true`. User transcripts require transcription to be configured through native session settings. |
| `audio` | `pcm`, `sampleRate`, `responseId`, `itemId`, `contentIndex`. The PCM chunk is an `Int16Array`. |
| `interrupted` | `source` (`manual` or `speech`), `responseId` |
| `turn.done` | `responseId`, native provider `usage`, `status` |
| `close` | `code`, `reason` |
Each event includes its `type`. When the provider event leaves out an ID, `responseId` and `itemId` are empty strings and `contentIndex` is `0`. The `interrupted` event signals local interruption immediately. Its corresponding operation resolves after the provider acknowledges cancellation and any requested truncation. Late audio from a canceled response remains visible as native events and is excluded from convenience audio and built-in playback.
## Browser media
The optional microphone and speaker helpers require a browser with Web Audio support. Microphone capture also requires media permissions and a secure context. Start audio from a user action so the browser can allow playback.
| Member | Behavior |
| --- | --- |
| `microphone.start()` | Requests microphone access and streams PCM16, resampled to 24 kHz, in 20 ms chunks through an AudioWorklet. If the page or network falls behind, chunks are dropped rather than stopping capture. |
| `microphone.mute()` | Disables capture tracks and stops forwarding captured chunks. |
| `microphone.unmute()` | Enables capture tracks and forwarding. |
| `microphone.stop()` | Stops capture tracks and releases capture resources. A `start()` still waiting on the permission prompt rejects with `session_closed`. |
| `speaker.start({ bargeIn: true })` | Starts playback. Barge-in defaults to `true`. With barge-in enabled, provider speech-start events interrupt playback and synchronize played audio with conversation history. Starting the speaker requests no microphone access. |
| `speaker.stop()` | Clears playback and releases its audio context. |
Both start methods require a connected session. Capture and playback are independent, so custom audio input can use the built-in speaker, and built-in capture can use custom playback.
## Examples
### Have a text conversation
A text conversation uses convenience methods for input and streamed output.
```js
const conversation = await puter.ai.convo({
output: ['text'],
instructions: 'Keep replies short.',
});
conversation.on('text', ({ delta }) => console.log(delta));
conversation.on('turn.done', () => conversation.close());
conversation.on('error', (event) => console.error(event));
await conversation.connect();
conversation.sendText('Say hello in three words.');
```
### Have a voice conversation
A voice conversation enables the microphone and speaker after connection.
```js
const conversation = await puter.ai.convo();
conversation.on('transcript', ({ role, text }) => console.log(role, text));
conversation.on('error', (event) => console.error(event));
await conversation.connect();
await conversation.speaker.start();
await conversation.microphone.start();
// To interrupt a response and submit a replacement text turn:
await conversation.interrupt();
conversation.sendText('Actually, make it Wednesday.');
// When the user ends the conversation:
conversation.close();
```
+126
View File
@@ -0,0 +1,126 @@
---
title: Realtime WebSocket API (Beta)
description: Connect to OpenAI realtime conversations through Puter without puter.js.
platforms: [websites, apps, nodejs, workers]
---
The Realtime WebSocket API lets your client send and receive native OpenAI realtime JSON events through Puter.
Puter authenticates the user and charges their account for usage.
**Beta:** This API is incomplete. Its methods, options, events, and behavior may change abruptly, including breaking changes without notice.
For optional text, microphone, playback, and interruption helpers, use [`puter.ai.realtime()`](/AI/realtime/).
## Connection and authentication
Connect to this URL on the Puter API host:
```text
wss://api.puter.com/puterai/openai/v1/realtime?model=gpt-realtime-2.1-mini
```
The `model` query parameter defaults to `gpt-realtime-2.1-mini`. The model is fixed for the connection. A session update that changes it is rejected. It also accepts `gpt-realtime-2.1`, `gpt-realtime-2`, `gpt-realtime-1.5`, `gpt-realtime`, and `gpt-realtime-mini`.
Authenticate with a Puter session, app, or full-access API token. Puter supplies the provider credential. Server clients pass this header:
```text
Authorization: Bearer <PUTER_TOKEN>
```
Browser WebSocket clients pass `realtime` and `openai-insecure-api-key.<PUTER_TOKEN>` as subprotocols. The server echoes only `realtime`. The endpoint also accepts an `auth_token` query parameter. Prefer the authorization header where supported because URLs and subprotocols can appear in logs. Authentication takes precedence in this order, header, subprotocol, then query parameter.
The socket can open before authentication and provider setup finish. Listen for errors and closure immediately. Wait for `session.created`, send your session settings, then wait for `session.updated` before submitting a turn.
To open a replacement while another connection remains healthy, set the optional `purpose=handover` query parameter. These attempts share a smaller per-credential budget, checked before the general connection-attempt budget. Initial connections and dropped-connection recovery use the URL without this parameter. The limits are listed in [rate limits and quotas](/rate-limits-and-quotas/).
## Text example
Install the `ws` package and set `PUTER_TOKEN` to your Puter token. Run this example as a Node.js ES module:
```js
import WebSocket from 'ws';
const socket = new WebSocket(
'wss://api.puter.com/puterai/openai/v1/realtime?model=gpt-realtime-2.1-mini',
{ headers: { Authorization: `Bearer ${process.env.PUTER_TOKEN}` } },
);
const send = (event) => socket.send(JSON.stringify(event));
socket.on('message', (data) => {
const event = JSON.parse(data.toString());
if (event.type === 'session.created') {
send({
type: 'session.update',
session: {
type: 'realtime',
output_modalities: ['text'],
instructions: 'Keep replies short.',
},
});
}
if (event.type === 'session.updated') {
send({
type: 'conversation.item.create',
item: {
type: 'message',
role: 'user',
content: [{ type: 'input_text', text: 'Hello!' }],
},
});
send({ type: 'response.create' });
}
if (event.type === 'response.output_text.delta')
process.stdout.write(event.delta);
if (event.type === 'error')
console.error(event.error);
if (event.type === 'response.done')
socket.close();
});
socket.on('error', console.error);
socket.on('close', (code, reason) => console.log(code, reason.toString()));
```
## Events, audio, and interruption
Each message contains one JSON object with a nonempty string `type` of at most 128 characters. Binary messages are rejected. Native events retain their provider fields. The endpoint doesn't deliver `rate_limits.updated`, which describes Puter's provider account rather than yours, and it removes provider organization IDs from error messages. The endpoint exposes the WebSocket event protocol. Your client manages capture, playback, acknowledgments, and timeouts.
The endpoint refuses `conversation.item.retrieve` events and MCP tools (`"type": "mcp"`) in `session.update` and `response.create` events. A refused event isn't sent to the provider. The session gets an `invalid_event` error and closes with 1008.
PCM16 audio configured at 24 kHz mono is sent as base64 chunks in `input_audio_buffer.append` events. Output arrives in `response.output_audio.delta` events. With manual turn detection, commit buffered input with `input_audio_buffer.commit`, then request a reply with `response.create`.
To interrupt, stop local playback immediately and send `response.cancel` with the active `response_id`. If some generated audio was not played, send `conversation.item.truncate` with its `item_id`, `content_index`, and `audio_end_ms` set to the amount actually played. Wait for `response.done` and any requested `conversation.item.truncated` acknowledgment before submitting a replacement turn.
Connection loss ends the session. Reconnecting creates a new conversation. Tool execution and automatic reconnection belong to your client.
## Server restarts
When Puter knows ahead of time that the server a session runs on will restart, it sends each open session this event, at most once a minute:
```json
{
"type": "puter.reconnect",
"event_id": "event_puter_1767225600000",
"reason": "server_restarting",
"close_in_ms": 90000
}
```
The `reason` field is `server_restarting`. The `close_in_ms` field, when present, is about how long until the server closes the session itself. The session keeps working until then. Open a new connection after a short, randomized delay, and once it gets `session.created` and your client is idle, with no response generating and no user turn in progress, move to it and close the old one. A connection that reaches a server that is also restarting gets a `server_restarting` error and closes with 1012. In that case keep the old session and try again after a short, randomized delay. If the server closes the old session first, the close code is 1012, and your client reconnects the same way. A restart can also drop the connection without any notice, so treat an unexpected close the same way.
A new connection starts an empty conversation. To keep context, add earlier items back with `conversation.item.create` before you request a response. Event types that start with `puter.` come from Puter, not the provider.
## Errors, billing, and limits
API errors arrive as `{ "type": "error", "error": { "code": "...", "message": "..." } }`. Handle socket errors and close frames too. Invalid, missing, or revoked credentials use `invalid_api_key`. If Puter can't check a credential or the account's plan, the error is `internal_error` and the session closes with 1011, so try again after a short, randomized delay. Other relay codes include `permission_denied`, `model_not_found`, `insufficient_funds`, `subscription_required`, `rate_limit_exceeded`, `invalid_event`, `connection_timeout`, `provider_unavailable`, `upstream_error`, and `server_restarting`.
Policy failures generally close with 1008, oversized client messages with 1009, upstream or setup failures with 1011, server restarts with 1012, and temporary server overload with 1013. Reconnect after 1011, 1012, or 1013. Authentication and model failures can arrive after the WebSocket handshake succeeds.
The endpoint requires a paid Puter plan. Usage is charged after completed responses and input transcriptions. While a response generates, part of your balance is held for it, so a session opens only if your balance covers one response. If the provider connection closes before a response reports its usage, the response is charged the full amount held for it. Each account can open 20 sessions at once by default. Each JSON message you send is limited to 64 KiB, including reassembled fragments. Messages from the provider can be up to 4 MiB. Stream audio in small chunks. All message rates, queue caps, timeouts, and other bounds are published in [rate limits and quotas](/rate-limits-and-quotas/).
Input transcription holds a small amount of your balance per transcribed item, priced at the transcription model you select. Two items are reserved ahead of your audio and replenished as the provider commits them, so a session holds only its spare items plus the items still awaiting a transcript. Changing the transcription model reprices the spare items. If your balance can't cover another item, audio waits while earlier items settle; once none are left to settle, the session closes with `insufficient_funds`. Failed transcriptions, items without final usage, and audio whose commit the provider never acknowledges before the connection closes are each charged that model's fixed per-item amount. Unacknowledged audio is charged one item per started 160 seconds. With server turn detection, only detected speech counts, so silence is never charged. The amounts, timeout and audio limits are listed in [rate limits and quotas](/rate-limits-and-quotas/).
Provider-reported realtime models must have a known price. A missing model uses the last known model. An explicitly unpriced model closes the connection with `upstream_error`.
After recording usage, the connection can continue while settled credit is positive, including a fractional balance. Each new billable operation still needs its full reservation after accounting for other outstanding holds.
+128 -103
View File
@@ -1,126 +1,151 @@
---
title: Realtime WebSocket API (Beta)
description: Connect to OpenAI realtime conversations through Puter without puter.js.
title: puter.ai.realtime()
description: Prepare an OpenAI realtime conversation with optional audio and text helpers.
platforms: [websites, apps, nodejs, workers]
---
The Realtime WebSocket API lets your client send and receive native OpenAI realtime JSON events through Puter.
Prepares a realtime OpenAI conversation that your app can connect to when it's ready.
Puter authenticates the user and charges their account for usage.
Preparing the conversation verifies Puter sign-in. Realtime conversations require a paid Puter plan, and usage is billed to the signed-in user. Native OpenAI events and convenience methods use the same session. Microphone capture and speaker playback start when your app requests them.
In Node.js, the SDK requires a global `WebSocket`, available in Node.js 22 or later.
**Beta:** This API is incomplete. Its methods, options, events, and behavior may change abruptly, including breaking changes without notice.
For optional text, microphone, playback, and interruption helpers, use [`puter.ai.convo()`](/AI/convo/).
To connect without puter.js, see the [Realtime WebSocket API](/AI/realtime-websocket/).
## Connection and authentication
Connect to this URL on the Puter API host:
```text
wss://api.puter.com/puterai/openai/v1/realtime?model=gpt-realtime-2.1-mini
```
The `model` query parameter defaults to `gpt-realtime-2.1-mini`. The model is fixed for the connection. A session update that changes it is rejected. It also accepts `gpt-realtime-2.1`, `gpt-realtime-2`, `gpt-realtime-1.5`, `gpt-realtime`, and `gpt-realtime-mini`.
Authenticate with a Puter session, app, or full-access API token. Puter supplies the provider credential. Server clients pass this header:
```text
Authorization: Bearer <PUTER_TOKEN>
```
Browser WebSocket clients pass `realtime` and `openai-insecure-api-key.<PUTER_TOKEN>` as subprotocols. The server echoes only `realtime`. The endpoint also accepts an `auth_token` query parameter. Prefer the authorization header where supported because URLs and subprotocols can appear in logs. Authentication takes precedence in this order, header, subprotocol, then query parameter.
The socket can open before authentication and provider setup finish. Listen for errors and closure immediately. Wait for `session.created`, send your session settings, then wait for `session.updated` before submitting a turn.
To open a replacement while another connection remains healthy, set the optional `purpose=handover` query parameter. These attempts share a smaller per-credential budget, checked before the general connection-attempt budget. Initial connections and dropped-connection recovery use the URL without this parameter. The limits are listed in [rate limits and quotas](/rate-limits-and-quotas/).
## Text example
Install the `ws` package and set `PUTER_TOKEN` to your Puter token. Run this example as a Node.js ES module:
## Syntax
```js
import WebSocket from 'ws';
const conversation = await puter.ai.realtime(options = {});
await conversation.connect();
```
const socket = new WebSocket(
'wss://api.puter.com/puterai/openai/v1/realtime?model=gpt-realtime-2.1-mini',
{ headers: { Authorization: `Bearer ${process.env.PUTER_TOKEN}` } },
);
const send = (event) => socket.send(JSON.stringify(event));
## Parameters
socket.on('message', (data) => {
const event = JSON.parse(data.toString());
#### `options` (Object) (Optional)
if (event.type === 'session.created') {
send({
type: 'session.update',
session: {
type: 'realtime',
output_modalities: ['text'],
instructions: 'Keep replies short.',
},
});
}
if (event.type === 'session.updated') {
send({
type: 'conversation.item.create',
item: {
type: 'message',
role: 'user',
content: [{ type: 'input_text', text: 'Hello!' }],
},
});
send({ type: 'response.create' });
}
if (event.type === 'response.output_text.delta')
process.stdout.write(event.delta);
if (event.type === 'error')
console.error(event.error);
if (event.type === 'response.done')
socket.close();
The options object accepts these fields.
| Option | Type | Description |
|--------|------|-------------|
| `model` | `String` | The realtime model. Defaults to `gpt-realtime-2.1-mini`. Also accepts `gpt-realtime-2.1`, `gpt-realtime-2`, `gpt-realtime-1.5`, `gpt-realtime`, and `gpt-realtime-mini`. |
| `instructions` | `String` | Session instructions, up to 8192 characters. Defaults to the provider's default. |
| `output` | `Array<String>` | Accepts `['text']`, `['audio']`, or both. Defaults to `['audio']`. Audio includes a transcript. Requesting both uses audio output and emits transcript deltas through the `text` event. |
| `voice` | `String` | OpenAI voice identifier, up to 128 characters. Defaults to the provider's default. |
| `turnDetection` | `Boolean` | Whether the provider detects the end of spoken turns automatically. Defaults to `true`. Set to `false` for manual audio commits. |
## Return value
A Promise resolves to a prepared conversation after options and sign-in are checked. In a browser, preparation opens the Puter sign-in prompt if needed. Preparation opens no realtime connection and requests no microphone access.
The conversation exposes these properties and methods.
| Member | Behavior |
| --- | --- |
| `state` | `prepared`, `connecting`, `connected`, or `closed`. |
| `model` | Resolved model name. |
| `connect()` | Opens the connection and applies settings. Resolves after the provider acknowledges session configuration. Concurrent calls share one connection Promise, including bounded retries for transient setup failures. Calling it while connected opens no additional connection. During a reconnection, it resolves when the reconnection succeeds and rejects if the conversation closes instead. |
| `configure(options)` | Updates the given session settings and resolves after the provider acknowledges this update. Settings it isn't given, including ones set through native events, are left alone. Accepts the factory options except `model`. One update can be pending at a time. Provider rules apply to changes such as switching a voice after audio has been generated. |
| `sendText(text)` | Adds a user message and requests a response. Text must be nonempty and at most 8192 characters. |
| `appendAudio(pcm)` | Appends an `Int16Array` containing PCM16, 24 kHz mono audio. A chunk contains at most 12000 samples. |
| `commitAudio()` | Commits buffered audio and requests a response. Requires turn detection to be off, through `turnDetection: false` or a native session update. The provider requires at least 100 ms of buffered audio. |
| `interrupt(position?)` | Stops built-in playback immediately and waits for cancellation and playback-history synchronization. A reply requested but not yet started is cancelled as soon as it starts. Custom players provide `{ itemId, contentIndex, playedMs }`. An idle call resolves immediately. Readiness failures reject the returned Promise with `session_not_ready` or `session_closed`. |
| `send(event)` | Sends a native OpenAI JSON event. |
| `on(type, handler)` | Registers a listener and returns an unsubscribe function. Use `'*'` to receive every native and convenience event. |
| `off(type, handler)` | Removes a listener. |
| `close()` | Closes the session, rejects pending operations, and releases SDK-owned capture and playback resources. Repeated calls have no additional effect. |
Attach listeners before connecting to receive initial session events. Closing during connection rejects the pending attempt.
If the connection drops because of network loss, a server restart or a provider error, the conversation reconnects on its own. Normal closes with code 1000 end the conversation. While it reconnects, `state` is `connecting`, a `reconnecting` event fires, and pending operations reject with `connection_lost`. The new session gets every setting the provider acknowledged, from the factory options, `configure()` and native `session.update` events. Then `ready` fires again with `reconnected: true`. Listeners, the microphone and the speaker carry over. Microphone audio captured while reconnecting is dropped.
The conversation also restores recent context. Before `ready` fires, it adds back up to 40 of the most recent conversation items, 48 KiB in total. Restored items are user text, user speech that has a transcript, assistant replies as text, and function calls with their outputs. User speech has a transcript only when input transcription is configured through native session settings. An assistant reply that was interrupted, or still playing when the connection dropped, is cut to what the user saw or heard. The cut is estimated from how much of the audio played, so it can be off by a few words. Images, speech without a transcript, deleted items, and replies still generating when the connection dropped are left out. Restored items get new item IDs, don't fire item events, and count as input tokens on the next response. Item IDs from before the reconnect don't work in later events. Restoration uses a contiguous suffix of restorable items. An oversized newest item leaves that suffix empty. Older items left out to fit the window fire a `context_trimmed` event with the count in `omittedItems`; this is routine and not an error. A provider refusal or a failure to send the restored items fires one `error` event with code `context_restore_failed` per restoration. In both cases the conversation stays connected. Items are preserved without truncating text, function arguments or function results.
When Puter knows ahead of time that the server a session runs on will restart, it sends a `puter.reconnect` event, which also reaches your listeners. Without that notice, a restart drops the connection and the conversation reconnects as above. Within a few seconds, the conversation opens a replacement session in the background while the current one keeps working. It switches once no reply is generating or playing, no user turn is being spoken or transcribed, no function call is waiting for its output, and nothing you sent is waiting on the provider. The switch goes through the same steps as a reconnect, and the `reconnecting` event has code 1012. If the replacement can't open yet, the conversation keeps the current session and tries again later. A refusal that retrying can't fix, such as `insufficient_funds`, stops the attempts. If the server closes the session first, the conversation switches to the replacement, or reconnects if none is open.
The conversation makes up to ten attempts over about a minute. In a browser that reports being offline, it first waits up to two minutes for the connection to come back. If every attempt fails, or the server refuses with an error such as `insufficient_funds`, the conversation closes. A closed conversation cannot reconnect. To start another session, prepare a new one.
Native event definitions are listed in the [OpenAI realtime reference](https://developers.openai.com/api/reference/resources/realtime/server-events). Native events retain their provider fields. Sending native events preserves direct control of the session. Puter refuses `conversation.item.retrieve` events and MCP tools (`"type": "mcp"`) in `session.update` and `response.create` events. Sending one fires an `error` event with code `invalid_event`, and the conversation closes with code 1008. Audio helpers expect the session to use PCM16 at 24 kHz.
## Errors and limits
Failures use objects containing `code` and `message`. Preparation can reject with `invalid_argument`, `model_not_found`, `auth_required`, `auth_canceled`, or `auth_failed`. Connection can reject with `subscription_required` or `insufficient_funds`, which also show the Puter upgrade prompt. Connection and asynchronous operations can reject with provider errors, `session_closed`, `connection_lost`, `operation_timeout`, or `operation_in_progress`. Sending before readiness, or while reconnecting, throws `session_not_ready`.
Invalid input throws `invalid_argument`, `invalid_event`, or `input_too_large`. A full SDK send queue throws `queue_full`. Media helpers report `unsupported_environment`, `permission_denied`, `media_error`, `media_timeout`, or `queue_full`. Runtime media errors and `context_restore_failed` arrive through the `error` event. Provider errors also arrive as native `error` events with a nested `error` object.
Each native event is capped at 64 KiB. The SDK bounds JSON traversal, outgoing queues, acknowledgment waits, microphone work, and playback. All numeric limits are listed on [rate limits and quotas](/rate-limits-and-quotas/).
## Convenience events
The conversation emits these events alongside native events.
| Event | Fields |
| --- | --- |
| `ready` | `model`, `provider`, and `reconnected: true` after a reconnection |
| `reconnecting` | `code`, `reason` of the dropped connection. After a `puter.reconnect` event, `code` is 1012. |
| `context_trimmed` | `omittedItems`, the number of restorable items left out of the restored context after a reconnection. Fires before `ready`. |
| `text` | `delta`, `responseId` |
| `transcript` | `role`, `text`, `final`. Assistant deltas use `final: false`. Completed assistant and user transcripts use `final: true`. User transcripts require transcription to be configured through native session settings. |
| `audio` | `pcm`, `sampleRate`, `responseId`, `itemId`, `contentIndex`. The PCM chunk is an `Int16Array`. |
| `interrupted` | `source` (`manual` or `speech`), `responseId` |
| `turn.done` | `responseId`, native provider `usage`, `status` |
| `close` | `code`, `reason` |
Each event includes its `type`. When the provider event leaves out an ID, `responseId` and `itemId` are empty strings and `contentIndex` is `0`. The `interrupted` event signals local interruption immediately. Its corresponding operation resolves after the provider acknowledges cancellation and any requested truncation. Late audio from a canceled response remains visible as native events and is excluded from convenience audio and built-in playback.
## Browser media
The optional microphone and speaker helpers require a browser with Web Audio support. Microphone capture also requires media permissions and a secure context. Start audio from a user action so the browser can allow playback.
| Member | Behavior |
| --- | --- |
| `microphone.start()` | Requests microphone access and streams PCM16, resampled to 24 kHz, in 20 ms chunks through an AudioWorklet. If the page or network falls behind, chunks are dropped rather than stopping capture. |
| `microphone.mute()` | Disables capture tracks and stops forwarding captured chunks. |
| `microphone.unmute()` | Enables capture tracks and forwarding. |
| `microphone.stop()` | Stops capture tracks and releases capture resources. A `start()` still waiting on the permission prompt rejects with `session_closed`. |
| `speaker.start({ bargeIn: true })` | Starts playback. Barge-in defaults to `true`. With barge-in enabled, provider speech-start events interrupt playback and synchronize played audio with conversation history. Starting the speaker requests no microphone access. |
| `speaker.stop()` | Clears playback and releases its audio context. |
Both start methods require a connected session. Capture and playback are independent, so custom audio input can use the built-in speaker, and built-in capture can use custom playback.
## Examples
### Have a text conversation
A text conversation uses convenience methods for input and streamed output.
```js
const conversation = await puter.ai.realtime({
output: ['text'],
instructions: 'Keep replies short.',
});
socket.on('error', console.error);
socket.on('close', (code, reason) => console.log(code, reason.toString()));
conversation.on('text', ({ delta }) => console.log(delta));
conversation.on('turn.done', () => conversation.close());
conversation.on('error', (event) => console.error(event));
await conversation.connect();
conversation.sendText('Say hello in three words.');
```
## Events, audio, and interruption
### Have a voice conversation
Each message contains one JSON object with a nonempty string `type` of at most 128 characters. Binary messages are rejected. Native events retain their provider fields. The endpoint doesn't deliver `rate_limits.updated`, which describes Puter's provider account rather than yours, and it removes provider organization IDs from error messages. The endpoint exposes the WebSocket event protocol. Your client manages capture, playback, acknowledgments, and timeouts.
A voice conversation enables the microphone and speaker after connection.
The endpoint refuses `conversation.item.retrieve` events and MCP tools (`"type": "mcp"`) in `session.update` and `response.create` events. A refused event isn't sent to the provider. The session gets an `invalid_event` error and closes with 1008.
```js
const conversation = await puter.ai.realtime();
conversation.on('transcript', ({ role, text }) => console.log(role, text));
conversation.on('error', (event) => console.error(event));
PCM16 audio configured at 24 kHz mono is sent as base64 chunks in `input_audio_buffer.append` events. Output arrives in `response.output_audio.delta` events. With manual turn detection, commit buffered input with `input_audio_buffer.commit`, then request a reply with `response.create`.
await conversation.connect();
await conversation.speaker.start();
await conversation.microphone.start();
To interrupt, stop local playback immediately and send `response.cancel` with the active `response_id`. If some generated audio was not played, send `conversation.item.truncate` with its `item_id`, `content_index`, and `audio_end_ms` set to the amount actually played. Wait for `response.done` and any requested `conversation.item.truncated` acknowledgment before submitting a replacement turn.
// To interrupt a response and submit a replacement text turn:
await conversation.interrupt();
conversation.sendText('Actually, make it Wednesday.');
Connection loss ends the session. Reconnecting creates a new conversation. Tool execution and automatic reconnection belong to your client.
## Server restarts
When Puter knows ahead of time that the server a session runs on will restart, it sends each open session this event, at most once a minute:
```json
{
"type": "puter.reconnect",
"event_id": "event_puter_1767225600000",
"reason": "server_restarting",
"close_in_ms": 90000
}
// When the user ends the conversation:
conversation.close();
```
The `reason` field is `server_restarting`. The `close_in_ms` field, when present, is about how long until the server closes the session itself. The session keeps working until then. Open a new connection after a short, randomized delay, and once it gets `session.created` and your client is idle, with no response generating and no user turn in progress, move to it and close the old one. A connection that reaches a server that is also restarting gets a `server_restarting` error and closes with 1012. In that case keep the old session and try again after a short, randomized delay. If the server closes the old session first, the close code is 1012, and your client reconnects the same way. A restart can also drop the connection without any notice, so treat an unexpected close the same way.
A new connection starts an empty conversation. To keep context, add earlier items back with `conversation.item.create` before you request a response. Event types that start with `puter.` come from Puter, not the provider.
## Errors, billing, and limits
API errors arrive as `{ "type": "error", "error": { "code": "...", "message": "..." } }`. Handle socket errors and close frames too. Invalid, missing, or revoked credentials use `invalid_api_key`. If Puter can't check a credential or the account's plan, the error is `internal_error` and the session closes with 1011, so try again after a short, randomized delay. Other relay codes include `permission_denied`, `model_not_found`, `insufficient_funds`, `subscription_required`, `rate_limit_exceeded`, `invalid_event`, `connection_timeout`, `provider_unavailable`, `upstream_error`, and `server_restarting`.
Policy failures generally close with 1008, oversized client messages with 1009, upstream or setup failures with 1011, server restarts with 1012, and temporary server overload with 1013. Reconnect after 1011, 1012, or 1013. Authentication and model failures can arrive after the WebSocket handshake succeeds.
The endpoint requires a paid Puter plan. Usage is charged after completed responses and input transcriptions. While a response generates, part of your balance is held for it, so a session opens only if your balance covers one response. If the provider connection closes before a response reports its usage, the response is charged the full amount held for it. Each account can open 20 sessions at once by default. Each JSON message you send is limited to 64 KiB, including reassembled fragments. Messages from the provider can be up to 4 MiB. Stream audio in small chunks. All message rates, queue caps, timeouts, and other bounds are published in [rate limits and quotas](/rate-limits-and-quotas/).
Input transcription holds a small amount of your balance per transcribed item, priced at the transcription model you select. Two items are reserved ahead of your audio and replenished as the provider commits them, so a session holds only its spare items plus the items still awaiting a transcript. Changing the transcription model reprices the spare items. If your balance can't cover another item, audio waits while earlier items settle; once none are left to settle, the session closes with `insufficient_funds`. Failed transcriptions, items without final usage, and audio whose commit the provider never acknowledges before the connection closes are each charged that model's fixed per-item amount. Unacknowledged audio is charged one item per started 160 seconds. With server turn detection, only detected speech counts, so silence is never charged. The amounts, timeout and audio limits are listed in [rate limits and quotas](/rate-limits-and-quotas/).
Provider-reported realtime models must have a known price. A missing model uses the last known model. An explicitly unpriced model closes the connection with `upstream_error`.
After recording usage, the connection can continue while settled credit is positive, including a fractional balance. Each new billable operation still needs its full reservation after accounting for other outstanding holds.
+1 -1
View File
@@ -8,7 +8,7 @@ Opens a dialog where the user can send you, the app's developer, feedback about
The message is delivered by Puter: it is stored and emailed to the email address on your Puter account. The feedback never passes through your app's code, and the dialog tells the user what is shared with you: their username, plus their email address (as the email's reply-to, so you can respond) when their email is verified.
Inside Puter, the dialog is rendered by the desktop environment. On a website, a puter.com popup hosts the dialog (signing the user in first if needed).
Inside Puter, the dialog is rendered by the desktop environment. On a website, a puter.com popup hosts the dialog (signing the user in first if needed). Puter's own apps on puter.com subdomains show it in a frame over the page instead, when the user is signed in to both.
**Feedback is opt-in.** Users can only send feedback if it's enabled for your app. Apps created in the Dev Center have feedback enabled at creation — you can turn it off with the "User Feedback" toggle in the app's settings. Apps created through `puter.apps.create` default to off; enable it by setting `feedbackEnabled`:
+2 -2
View File
@@ -91,7 +91,7 @@ Gemini speech-to-text first counts the audio's tokens, which times out after 30
The OpenAI- and Anthropic-compatible endpoints (`/puterai/openai/v1/*`, `/puterai/anthropic/v1/messages`) require a paid plan; a free account gets `402 subscription_required`. The same models are available to every account through `puter.ai.*` and `/drivers/call`, and the model catalogue endpoints are open to everyone.
Realtime sessions ([`puter.ai.convo()`](/AI/convo/), `/puterai/openai/v1/realtime`) require a paid plan, like the other OpenAI-compatible endpoints, and are capped at 20 open at once per account. An account without a plan gets an `error` event with code `subscription_required`. A session over the cap gets `rate_limit_exceeded`. Both close with 1008.
Realtime sessions ([`puter.ai.realtime()`](/AI/realtime/), `/puterai/openai/v1/realtime`) require a paid plan, like the other OpenAI-compatible endpoints, and are capped at 20 open at once per account. An account without a plan gets an `error` event with code `subscription_required`. A session over the cap gets `rate_limit_exceeded`. Both close with 1008.
| Realtime transport limit | Value | Behavior at the limit |
| --- | --- | --- |
@@ -131,7 +131,7 @@ Realtime sessions ([`puter.ai.convo()`](/AI/convo/), `/puterai/openai/v1/realtim
Send audio as a stream of small chunks. The message cap applies to text, configuration, tool output, and audio alike.
The [`puter.ai.convo()`](/AI/convo/) SDK also applies these limits before sending or retaining data.
The [`puter.ai.realtime()`](/AI/realtime/) SDK also applies these limits before sending or retaining data.
| Conversation SDK limit | Value | Behavior at the limit |
| --- | --- | --- |
+7 -7
View File
@@ -124,17 +124,17 @@ let sidebar = [
path: '/AI/speech2speech',
},
{
title: '<code>convo()</code>',
page_title: '<code>puter.ai.convo()</code>',
title_tag: 'puter.ai.convo()',
title: '<code>realtime()</code>',
page_title: '<code>puter.ai.realtime()</code>',
title_tag: 'puter.ai.realtime()',
icon: '/assets/img/function.svg',
source: '/AI/convo.md',
path: '/AI/convo',
source: '/AI/realtime.md',
path: '/AI/realtime',
},
{
title: 'Realtime WebSocket API (Beta)',
source: '/AI/realtime.md',
path: '/AI/realtime',
source: '/AI/realtime-websocket.md',
path: '/AI/realtime-websocket',
},
{
title: '<code>listModels()</code>',
+2 -1
View File
@@ -54,7 +54,8 @@ async function UIWindowAppFeedback (options) {
let sending = false;
let closed = false;
const authToken = puter.authToken ?? window.auth_token;
// The feedback embed runs without puter.js.
const authToken = globalThis.puter?.authToken ?? window.auth_token;
const target_params = options.app
? { app: options.app }
: { origin: options.origin };
+2 -46
View File
@@ -25,7 +25,7 @@ import { appTileLink } from './UI/Dashboard/appLink.js';
import { installAppIconFallback } from './helpers/appIcon.js';
import { guestAppLaunchUrl, launchAppInTab } from './helpers/appTabLaunch.js';
import { embedHostTheme, forceColorScheme } from './helpers/embedHostTheme.js';
import { embedHostUser } from './helpers/embedHostUser.js';
import { embedSession } from './helpers/embedSession.js';
window.$ = window.jQuery = jQuery;
window.html_encode = value => encode(String(value ?? ''), { mode: 'nonAsciiPrintable' });
@@ -104,56 +104,12 @@ function launchTile ({ appName, targetLink }) {
.finally(() => launching.delete(appName));
}
/**
* The GUI session, from local storage or else the session cookie. Safari keys a
* frame's local storage by the top-level origin, so under a sibling subdomain
* the GUI's stored token is out of reach while its cookie is still sent.
*/
async function readSession () {
let stored = null;
try {
stored = localStorage.getItem('auth_token_v2');
} catch { /* Storage the browser refuses holds no session; the cookie may. */ }
if ( stored ) return stored;
const response = await fetch('/get-gui-token', { credentials: 'include' });
// Only a 401 means there is no session; a rate limit or server error
// must not pass a signed-in user off as signed out.
if ( response.status === 401 ) return null;
if ( ! response.ok ) throw new Error(`Session lookup responded ${response.status}`);
return (await response.json())?.token || null;
}
/**
* The uuid of the user a GUI token belongs to, or null when the API refuses it.
*/
async function sessionUser (token) {
const response = await fetch(new URL('/whoami', config.apiOrigin), {
headers: { Authorization: `Bearer ${token}` },
});
if ( response.status === 401 ) return null;
if ( ! response.ok ) throw new Error(`Session check responded ${response.status}`);
const uuid = (await response.json())?.uuid;
return typeof uuid === 'string' ? uuid.toLowerCase() : null;
}
/**
* The GUI session, only when it belongs to the user the host page names. A
* host signed out, or signed in as someone else, gets the signed-out view.
*/
async function hostSession () {
const hostUser = embedHostUser(window.location.hash);
if ( ! hostUser ) return null;
const token = await readSession();
if ( ! token ) return null;
return await sessionUser(token) === hostUser ? token : null;
}
/**
* Signed in, the user's own apps through the SDK. Without a session, the
* recommended apps the page was rendered with, and no SDK.
*/
async function mount () {
window.auth_token = await hostSession();
window.auth_token = (await embedSession(config.apiOrigin, window.location.hash))?.token ?? null;
const guestApps = window.auth_token ? undefined : config.guestApps;
if ( !window.auth_token && !guestApps?.length ) throw new Error('No apps to show without a session');
+17
View File
@@ -0,0 +1,17 @@
/* Transparent, so the dialog's own scrim covers the framing page. */
html,
body {
margin: 0;
width: 100%;
height: 100%;
overflow: hidden;
background: transparent;
font-family: "Inter", "Helvetica Neue", Helvetica, Arial, sans-serif;
}
/* A scroll over the scrim must not scroll the framing page behind it. */
html {
overscroll-behavior: none;
/* Matches the frame's, so it stays transparent and follows the OS theme. */
color-scheme: light dark;
}
+14
View File
@@ -181,6 +181,20 @@ input[type=text], input[type=password], input[type=email], select {
font-size: 14px;
}
@media (prefers-color-scheme: dark) {
input[type=text], input[type=password], input[type=email], select {
background-color: #2d2d2d;
border: 1px solid #555;
color: #e6e6e6;
}
input[type=text]::placeholder,
input[type=password]::placeholder,
input[type=email]::placeholder {
color: #999;
}
}
/* to prevent auto-zoom on input focus in mobile */
.device-phone input[type=text], .device-phone input[type=password], .device-phone input[type=email], .device-phone select {
font-size: 17px;
+252
View File
@@ -0,0 +1,252 @@
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { chromium } from '@playwright/test';
import express from 'express';
import webpack from 'webpack';
import { mkdtemp, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { inflateSync } from 'node:zlib';
let browser;
let server;
let directory;
let origin;
let hostOrigin;
let port;
const requests = [];
const guiRoot = fileURLToPath(new URL('../', import.meta.url));
const sessionToken = 'feedback-embed-test-session';
const unconfirmedToken = 'feedback-embed-test-unconfirmed';
const sessionUser = '0b1c9a52-3f0e-4c7a-9a65-2f6d1e8b7c40';
const unconfirmedUser = '1c2d3e4f-5a6b-4c7d-8e9f-0a1b2c3d4e5f';
const otherUser = '7d2e4f10-8a3b-4c5d-9e6f-0a1b2c3d4e5f';
beforeAll(async () => {
directory = await mkdtemp(path.join(tmpdir(), 'puter-feedback-embed-'));
await new Promise((resolve, reject) => {
const compiler = webpack({
mode: 'development',
entry: { 'feedback-embed': path.join(guiRoot, 'src/feedbackEmbed.js') },
output: { path: directory, filename: '[name].min.js' },
module: { rules: [{ test: /jquery-3\.6\.1\.min\.js$/, type: 'javascript/auto' }] },
});
compiler.run((error, stats) => compiler.close(closeError => {
if ( error || closeError ) return reject(error || closeError);
if ( stats.hasErrors() ) return reject(new Error(stats.toString()));
resolve();
}));
});
const app = express();
app.use(express.json());
app.use((req, res, next) => {
requests.push({ path: req.path, query: req.query, body: req.body, auth: req.headers.authorization });
next();
});
app.use('/dist', express.static(directory));
app.use('/css', express.static(path.join(guiRoot, 'src/css')));
app.get('/get-gui-token', (_req, res) => res.sendStatus(401));
const whoamiByToken = {
[sessionToken]: { uuid: sessionUser, username: 'session-user', email: '[email protected]', email_confirmed: true },
[unconfirmedToken]: {
uuid: unconfirmedUser, username: 'new-user', email_confirmed: false, requires_email_confirmation: true,
},
};
app.get('/whoami', (req, res) => {
const user = whoamiByToken[req.headers.authorization?.replace(/^Bearer /, '')];
if ( ! user ) return res.sendStatus(401);
res.json(user);
});
app.get('/app-feedback/target', (req, res) => {
if ( req.headers.authorization !== `Bearer ${sessionToken}` ) return res.sendStatus(401);
if ( req.query.origin !== hostOrigin ) return res.json({ enabled: false });
res.json({ enabled: true, app: { name: 'notes', title: 'Notes' } });
});
app.post('/app-feedback', (req, res) => {
if ( req.headers.authorization !== `Bearer ${sessionToken}` ) return res.sendStatus(401);
res.json({});
});
// A first-party page that frames the dialog as puter.js does, recording what the embed tells it.
app.get('/host', (req, res) => res.send(`<!DOCTYPE html><html><body style="margin:0;background:rgb(255,0,0)">
<script>
window.received = [];
addEventListener('message', (e) => {
if ( e.origin === '${origin}' ) window.received.push(e.data);
});
</script>
<iframe referrerpolicy="origin" title="Send feedback" src="${origin}/embed/feedback#user=${req.query.user ?? ''}"
style="display:block;border:0;width:100vw;height:100vh;color-scheme:light dark"></iframe>
<div style="height:3000px"></div>
</body></html>`));
app.get('/embed/feedback', (_req, res) => {
res.setHeader('Content-Security-Policy', `frame-ancestors 'self' ${hostOrigin}`);
res.send(`<!DOCTYPE html><html><head><meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="/css/normalize.css"><link rel="stylesheet" href="/css/dashboard.css">
<link rel="stylesheet" href="/css/feedback-embed.css"></head><body>
<main id="feedback-embed"></main>
<script type="application/json" id="feedback-embed-config">${JSON.stringify({ apiOrigin: origin, hostOrigin })}</script>
<script defer src="/dist/feedback-embed.min.js"></script></body></html>`);
});
server = await new Promise(resolve => {
const listener = app.listen(0, '127.0.0.1', () => resolve(listener));
});
port = server.address().port;
origin = `http://puter.localhost:${port}`;
hostOrigin = `http://notes.puter.localhost:${port}`;
browser = await chromium.launch();
}, 60000);
afterAll(async () => {
await browser?.close();
if ( server ) await new Promise(resolve => server.close(resolve));
if ( directory ) await rm(directory, { recursive: true, force: true });
});
/** A browser whose GUI origin is signed in with `token`, if any. */
const signedIn = (token, options = {}) => browser.newContext({ ...options, storageState: {
cookies: [],
origins: token ? [{ origin, localStorage: [{ name: 'auth_token_v2', value: token }] }] : [],
} });
const received = page => page.evaluate(() => window.received);
/** The [r, g, b] of one pixel of the page, from a screenshot. */
const pixel = async (page, x, y) => {
const png = await page.screenshot({ clip: { x, y, width: 1, height: 1 }, animations: 'disabled' });
const data = [];
for ( let at = 8; at < png.length; ) {
const length = png.readUInt32BE(at);
if ( png.toString('ascii', at + 4, at + 8) === 'IDAT' ) data.push(png.subarray(at + 8, at + 8 + length));
at += 12 + length;
}
// One pixel: a filter byte, then its channels unchanged.
return [...inflateSync(Buffer.concat(data)).subarray(1, 4)];
};
describe('feedback embed', () => {
it('sends feedback for the framing page as the user it names, then tells it', async () => {
const context = await signedIn(sessionToken);
try {
const page = await context.newPage();
page.setDefaultTimeout(6000);
const errors = [];
page.on('pageerror', error => errors.push(error.message));
requests.length = 0;
await page.goto(`${hostOrigin}/host?user=${sessionUser}`);
const frame = page.frameLocator('iframe');
await frame.locator('.app-feedback-form').waitFor();
expect(await frame.locator('.app-feedback-target-title').textContent()).toBe('Notes');
expect(await received(page)).toEqual([
{ msg: 'feedbackEmbed', state: 'loading' },
{ msg: 'feedbackEmbed', state: 'ready' },
]);
// The page behind shows through around the dialog, under its scrim.
expect(await frame.locator('body').evaluate(el => getComputedStyle(el).backgroundColor)).toBe('rgba(0, 0, 0, 0)');
await page.screenshot({ path: path.join(tmpdir(), 'puter-feedback-embed.png'), animations: 'disabled' });
await frame.locator('.app-feedback-message').fill('Love the new editor');
await frame.locator('.app-feedback-send-btn').click();
await frame.locator('.app-feedback-success').waitFor();
await page.waitForFunction(() => window.received.some(m => m.msg === 'feedbackDialogClosed'));
expect((await received(page)).at(-1)).toEqual({ msg: 'feedbackDialogClosed', sent: true });
expect(await frame.locator('.app-feedback-overlay').count()).toBe(0);
const submit = requests.find(req => req.path === '/app-feedback');
expect(submit.body).toEqual({ origin: hostOrigin, message: 'Love the new editor', context: 'web' });
expect(submit.auth).toBe(`Bearer ${sessionToken}`);
expect(errors).toEqual([]);
} finally {
await context.close();
}
}, 30000);
it('answers not sent when the user dismisses it', async () => {
const context = await signedIn(sessionToken);
try {
const page = await context.newPage();
page.setDefaultTimeout(6000);
await page.goto(`${hostOrigin}/host?user=${sessionUser}`);
const frame = page.frameLocator('iframe');
await frame.locator('.app-feedback-form').waitFor();
await frame.locator('.app-feedback-message').press('Escape');
await page.waitForFunction(() => window.received.some(m => m.msg === 'feedbackDialogClosed'));
expect((await received(page)).at(-1)).toEqual({ msg: 'feedbackDialogClosed', sent: false });
} finally {
await context.close();
}
}, 30000);
it.each(['light', 'dark'])('lets the page show through under the scrim in %s mode', async (colorScheme) => {
const context = await signedIn(sessionToken, { colorScheme });
try {
const page = await context.newPage();
page.setDefaultTimeout(6000);
await page.goto(`${hostOrigin}/host?user=${sessionUser}`);
const frame = page.frameLocator('iframe');
await frame.locator('.app-feedback-form').waitFor();
expect(await frame.locator('body').evaluate(() => matchMedia('(prefers-color-scheme: dark)').matches))
.toBe(colorScheme === 'dark');
await page.waitForTimeout(300);
// The red page, dimmed by the scrim, rather than an opaque frame.
const [red, green, blue] = await pixel(page, 10, 10);
expect(red).toBeGreaterThan(green + 40);
expect(green).toBe(blue);
} finally {
await context.close();
}
}, 30000);
it('keeps the page behind from scrolling under the scrim', async () => {
const context = await signedIn(sessionToken);
try {
const page = await context.newPage();
page.setDefaultTimeout(6000);
await page.goto(`${hostOrigin}/host?user=${sessionUser}`);
await page.frameLocator('iframe').locator('.app-feedback-form').waitFor();
await page.mouse.move(10, 10);
await page.mouse.wheel(0, 600);
await page.waitForTimeout(300);
expect(await page.evaluate(() => window.scrollY)).toBe(0);
} finally {
await context.close();
}
}, 30000);
it.each([
['no Puter session', null, sessionUser],
['a session for another user', sessionToken, otherUser],
['a host that names no user', sessionToken, ''],
['an email still to confirm', unconfirmedToken, unconfirmedUser],
])('tells the host it is unavailable with %s', async (_label, token, user) => {
const context = await signedIn(token);
try {
const page = await context.newPage();
page.setDefaultTimeout(6000);
requests.length = 0;
await page.goto(`${hostOrigin}/host?user=${user}`);
await page.waitForFunction(() => window.received.length >= 2);
expect(await received(page)).toEqual([
{ msg: 'feedbackEmbed', state: 'loading' },
{ msg: 'feedbackEmbed', state: 'unavailable' },
]);
expect(await page.frameLocator('iframe').locator('.app-feedback-overlay').count()).toBe(0);
expect(requests.some(req => req.path.startsWith('/app-feedback'))).toBe(false);
} finally {
await context.close();
}
}, 30000);
it('does nothing when opened on its own', async () => {
const context = await signedIn(sessionToken);
try {
const page = await context.newPage();
requests.length = 0;
await page.goto(`${origin}/embed/feedback#user=${sessionUser}`);
await page.waitForLoadState('networkidle');
expect(await page.locator('.app-feedback-overlay').count()).toBe(0);
expect(requests.some(req => req.path === '/whoami')).toBe(false);
} finally {
await context.close();
}
}, 30000);
});
+86
View File
@@ -0,0 +1,86 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import jQuery from './lib/jquery-3.6.1/jquery-3.6.1.min.js';
import { encode } from 'html-entities';
import { loadLocale } from './i18n/embedI18n.js';
import UIWindowAppFeedback from './UI/UIWindowAppFeedback.js';
import { embedSession } from './helpers/embedSession.js';
window.$ = window.jQuery = jQuery;
window.html_encode = value => encode(String(value ?? ''), { mode: 'nonAsciiPrintable' });
const config = JSON.parse(document.querySelector('#feedback-embed-config').textContent);
window.api_origin = config.apiOrigin;
window.locale = 'en';
try {
window.locale = JSON.parse(localStorage.getItem('user_preferences') || '{}')?.language || 'en';
} catch { /* Corrupt preferences must not prevent sending feedback. */ }
const localeReady = loadLocale(window.locale);
/** Answers only the page frame-ancestors pinned, which the feedback is for. */
const tellHost = (data) => window.parent.postMessage(data, config.hostOrigin);
/** Resolves once the dialog's overlay has left the page, after its success pane and exit. */
const overlayGone = () => new Promise((resolve) => {
const gone = () => !document.querySelector('.app-feedback-overlay');
if ( gone() ) return resolve();
const observer = new MutationObserver(() => {
if ( ! gone() ) return;
observer.disconnect();
resolve();
});
observer.observe(document.body, { childList: true });
});
/**
* Shows the app feedback dialog for the framing page, as the signed-in user it
* names. Without that user's session, or with an email still to confirm, the
* host is told the embed is unavailable and uses the popup, which can sign in.
*/
async function run () {
tellHost({ msg: 'feedbackEmbed', state: 'loading' });
let session = null;
try {
session = await embedSession(config.apiOrigin, window.location.hash);
} catch (e) {
console.error('feedback embed: session check failed', e);
}
const user = session?.user;
if ( !user || (user.requires_email_confirmation && !user.email_confirmed) ) {
tellHost({ msg: 'feedbackEmbed', state: 'unavailable' });
return;
}
window.auth_token = session.token;
window.user = user;
await localeReady;
tellHost({ msg: 'feedbackEmbed', state: 'ready' });
let sent = false;
try {
sent = await UIWindowAppFeedback({ origin: config.hostOrigin, source: 'web' });
await overlayGone();
} catch (e) {
console.error('feedback embed: dialog failed', e);
}
tellHost({ msg: 'feedbackDialogClosed', sent: sent === true });
}
// Opened on its own, there is no page to send feedback for or answer.
if ( config.hostOrigin && window.parent !== window ) run();
+83
View File
@@ -0,0 +1,83 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { embedHostUser } from './embedHostUser.js';
/**
* @typedef {Object} EmbedSession
* @property {string} token - the GUI session's token
* @property {Record<string, unknown>} user - `/whoami` for that token
*/
/**
* The GUI session, from local storage or else the session cookie. Safari keys a
* frame's local storage by the top-level origin, so under a sibling subdomain
* the GUI's stored token is out of reach while its cookie is still sent.
*
* @returns {Promise<string | null>}
*/
async function readSession () {
let stored = null;
try {
stored = localStorage.getItem('auth_token_v2');
} catch { /* Storage the browser refuses holds no session; the cookie may. */ }
if ( stored ) return stored;
const response = await fetch('/get-gui-token', { credentials: 'include' });
// Only a 401 means there is no session; a rate limit or server error
// must not pass a signed-in user off as signed out.
if ( response.status === 401 ) return null;
if ( ! response.ok ) throw new Error(`Session lookup responded ${response.status}`);
return (await response.json())?.token || null;
}
/**
* `/whoami` for a GUI token, or null when the API refuses it.
*
* @param {string} token
* @param {string} apiOrigin
* @returns {Promise<Record<string, unknown> | null>}
*/
async function whoami (token, apiOrigin) {
const response = await fetch(new URL('/whoami', apiOrigin), {
headers: { Authorization: `Bearer ${token}` },
});
if ( response.status === 401 ) return null;
if ( ! response.ok ) throw new Error(`Session check responded ${response.status}`);
const user = await response.json();
return user && typeof user === 'object' ? user : null;
}
/**
* The GUI session, only when it belongs to the user the embedding page names
* in its `#user=<uuid>` fragment. A host signed out, or signed in as someone
* else, gets null. Rejects when the session can't be checked.
*
* @param {string} apiOrigin
* @param {string} hash - `location.hash`
* @returns {Promise<EmbedSession | null>}
*/
export async function embedSession (apiOrigin, hash) {
const hostUser = embedHostUser(hash);
if ( ! hostUser ) return null;
const token = await readSession();
if ( ! token ) return null;
const user = await whoami(token, apiOrigin);
const uuid = typeof user?.uuid === 'string' ? user.uuid.toLowerCase() : null;
return uuid === hostUser ? { token, user } : null;
}
-1
View File
@@ -623,7 +623,6 @@ const cs = {
"window_title_open": "Otevřít",
"window_title_change_password": "Změnit heslo",
"login_email_username_required": "E-mail nebo uživatelské jméno je povinné",
"oidc_switched_to_login_message": "Byli jste přihlášeni ke stávajícímu účtu.",
"sign_in_with_provider": "Přihlásit se pomocí %%",
"sign_up_with_provider": "Zaregistrujte se pomocí %%",
"welcome_description": "Ukládejte soubory, hrajte hry, najděte úžasné aplikace a mnoho dalšího! Vše na jednom místě, dostupné odkudkoli a kdykoli.",
-1
View File
@@ -1013,7 +1013,6 @@ const en = {
sign_up_with_email: 'Sign up using email',
popup_opener_uses_puter: '%strong% is powered by Puter',
popup_opener_origin_unsupported: 'Puter cannot sign you in to this page because the browser gives it no identifiable address. Pages opened straight from a file (file://) and iframes sandboxed without allow-same-origin are not supported — serve the page over http://localhost or a real domain and try again.',
oidc_switched_to_login_message: 'You have been logged in to an existing account.',
// Login Window
'login_email_username_required': 'Email or username is required',
-19
View File
@@ -1826,25 +1826,6 @@ window.initgui = async function (options) {
});
}
//--------------------------------------------------------------------------------------
// Inform the user if they chose "signup" but were logged into an existing account
//--------------------------------------------------------------------------------------
if (
window.url_query_params.get('oidc_switched') === 'login' &&
window.is_auth()
) {
await UIAlert({
message: i18n('oidc_switched_to_login_message'),
});
const params = new URLSearchParams(window.location.search);
params.delete('oidc_switched');
const cleanSearch = params.toString();
const cleanUrl = cleanSearch
? `${window.location.pathname}?${cleanSearch}`
: window.location.pathname || '/';
window.history.replaceState(null, document.title, cleanUrl);
}
//--------------------------------------------------------------------------------------
// Early check for fullpage mode from app metadata
// If the user navigated to /app/<app_name> and the app has fullpage_on_landing,
+1
View File
@@ -93,6 +93,7 @@ module.exports = async (options = {}) => {
...entries,
],
'apps-embed': './src/appsEmbed.js',
'feedback-embed': './src/feedbackEmbed.js',
};
config.output = {
path: path.resolve(__dirname, '../dist'),
+13 -13
View File
@@ -44,17 +44,17 @@ export type {
ListTTSEnginesOptions,
ListTTSVoicesOptions,
OcrAnnotationFormat,
ConvoOptions,
ConvoTextEvent,
ConvoAudioEvent,
ConvoTranscriptEvent,
ConvoInterruptedEvent,
ConvoTurnDoneEvent,
ConvoErrorEvent,
ConvoEventMap,
ConvoSessionOptions,
ConvoPlaybackPosition,
ConvoSpeakerOptions,
RealtimeOptions,
RealtimeTextEvent,
RealtimeAudioEvent,
RealtimeTranscriptEvent,
RealtimeInterruptedEvent,
RealtimeTurnDoneEvent,
RealtimeErrorEvent,
RealtimeEventMap,
RealtimeSessionOptions,
RealtimePlaybackPosition,
RealtimeSpeakerOptions,
RealtimeServerEvent,
RealtimeClientEvent,
Speech2SpeechOptions,
@@ -391,5 +391,5 @@ export type Teams = InstanceType<import('./types/modules/teams/index.js').TeamsC
export type UI = InstanceType<import('./types/modules/UI.js').UIConstructor>;
export type WorkersHandler = InstanceType<import('./types/modules/Workers.js').WorkersConstructor>;
export type { Conversation, ConvoError } from './types/modules/ai/convo.js';
export type { ConvoMicrophone, ConvoSpeaker } from './types/modules/ai/convoMedia.js';
export type { Conversation, RealtimeError } from './types/modules/ai/realtime.js';
export type { RealtimeMicrophone, RealtimeSpeaker } from './types/modules/ai/realtimeMedia.js';
+168
View File
@@ -0,0 +1,168 @@
/**
* Puter GUI pages a first-party site can frame, under `/embed/*` on the GUI
* origin. The server lets only a page one label under that origin frame them.
*/
// After the frame loads, how long to wait for the embed's first message
// before treating it as refused (a CSP that blocks it still fires `load`).
const LOAD_GRACE_MS = 1000;
// How long the embed may take to check the session before the popup is used.
const READY_TIMEOUT_MS = 15_000;
// A session check slower than this shows a loading scrim over the page.
const LOADING_DELAY_MS = 200;
// The loading scrim matches the embed's own, which fades in over it once ready.
const STYLE = `
dialog.puter-feedback-embed::backdrop{background:transparent}
dialog.puter-feedback-embed:focus{outline:none}
.puter-feedback-embed-loading{position:absolute;inset:0;display:flex;align-items:center;justify-content:center;background:rgba(0,0,0,.5);opacity:0;pointer-events:none;transition:opacity .16s ease-in-out}
.puter-feedback-embed.is-loading .puter-feedback-embed-loading{opacity:1;pointer-events:auto}
.puter-feedback-embed.is-loading iframe{visibility:hidden}
.puter-feedback-embed-loading span{width:28px;height:28px;box-sizing:border-box;border:3px solid rgba(255,255,255,.3);border-top-color:#fff;border-radius:50%;animation:puter-feedback-embed-spin .7s linear infinite}
@keyframes puter-feedback-embed-spin{to{transform:rotate(360deg)}}
@media (prefers-color-scheme:dark){.puter-feedback-embed-loading{background:rgba(0,0,0,.66)}}
@media (prefers-reduced-motion:reduce){.puter-feedback-embed-loading{transition:none}.puter-feedback-embed-loading span{animation-duration:1.6s}}
`;
/**
* Whether a page at `pageUrl` may frame an embed served by `guiOrigin`:
* exactly one label under its host, on the same scheme and port.
*
* @param {string | undefined} pageUrl
* @param {string} guiOrigin
* @returns {boolean}
*/
export const canFrameGuiEmbed = (pageUrl, guiOrigin) => {
let page;
let gui;
try {
page = new URL(pageUrl);
gui = new URL(guiOrigin);
} catch {
return false;
}
const host = page.hostname.replace(/\.$/, '').toLowerCase();
const suffix = `.${gui.hostname.toLowerCase()}`;
if ( page.protocol !== gui.protocol || page.port !== gui.port || !host.endsWith(suffix) ) return false;
const label = host.slice(0, -suffix.length);
return label.length > 0 && !label.includes('.');
};
/**
* Shows the app feedback dialog from `/embed/feedback` in a modal frame over
* the page, as the user named, behind a loading scrim while a slow session
* check runs. Resolves `{ sent }` once the dialog closes or the user cancels
* the loading scrim, or null when the embed can't serve this user (no
* matching Puter session, an email to confirm, or a refused frame), so the
* caller can use the popup instead. Never rejects.
*
* @param {Object} opts
* @param {string} opts.guiOrigin
* @param {string} opts.userUuid - the page's signed-in user
* @returns {Promise<{ sent: boolean } | null>}
*/
export const showFeedbackEmbed = ({ guiOrigin, userUuid }) => new Promise((resolve) => {
let origin;
try {
origin = new URL(guiOrigin).origin;
} catch {
resolve(null);
return;
}
const url = new URL('/embed/feedback', origin);
url.hash = new URLSearchParams({ user: userUuid }).toString();
const dialog = document.createElement('dialog');
dialog.className = 'puter-feedback-embed';
dialog.style.cssText = 'position:fixed;inset:0;width:100%;height:100%;max-width:none;max-height:none;margin:0;padding:0;border:0;background:transparent;overflow:hidden;';
const style = document.createElement('style');
style.textContent = STYLE;
const frame = document.createElement('iframe');
frame.title = 'Send feedback';
// The embed allows a framing page it can name from the referrer.
frame.setAttribute('referrerpolicy', 'origin');
// A frame whose color-scheme differs from its document's is painted
// opaque, so this matches the embed's. WebKit also gives the frame this
// scheme as its preferred one, so it has to follow the OS.
frame.style.cssText = 'display:block;width:100%;height:100%;border:0;background:transparent;color-scheme:light dark;';
const loading = document.createElement('div');
loading.className = 'puter-feedback-embed-loading';
loading.setAttribute('role', 'status');
loading.setAttribute('aria-label', 'Loading');
loading.append(document.createElement('span'));
dialog.append(style, frame, loading);
let settled = false;
let heard = false;
let shown = false;
let ready = false;
let timer = null;
let loadingTimer = null;
const restoreFocus = document.activeElement;
const wait = (ms) => {
clearTimeout(timer);
timer = setTimeout(() => finish(null), ms);
};
const finish = (result) => {
if ( settled ) return;
settled = true;
clearTimeout(timer);
clearTimeout(loadingTimer);
window.removeEventListener('message', onMessage);
dialog.remove();
if ( shown ) restoreFocus?.focus?.({ preventScroll: true });
resolve(result);
};
const show = () => {
if ( dialog.open ) return;
shown = true;
dialog.showModal();
};
const onMessage = (event) => {
if ( event.origin !== origin || event.source !== frame.contentWindow ) return;
const { msg, state, sent } = event.data ?? {};
if ( msg === 'feedbackDialogClosed' ) {
finish({ sent: sent === true });
} else if ( msg === 'feedbackEmbed' && state === 'loading' ) {
heard = true;
wait(READY_TIMEOUT_MS);
} else if ( msg === 'feedbackEmbed' && state === 'ready' ) {
heard = true;
ready = true;
clearTimeout(timer);
clearTimeout(loadingTimer);
dialog.classList.remove('is-loading');
show();
frame.focus();
} else if ( msg === 'feedbackEmbed' && state === 'unavailable' ) {
finish(null);
}
};
window.addEventListener('message', onMessage);
frame.addEventListener('load', () => {
if ( ! heard ) wait(LOAD_GRACE_MS);
});
// Once ready, Escape outside the frame would close the dialog with the
// embed's dialog still open inside it; the embed owns dismissal.
dialog.addEventListener('cancel', (event) => {
event.preventDefault();
if ( ready ) frame.focus();
else finish({ sent: false });
});
loading.addEventListener('click', () => {
if ( ! ready ) finish({ sent: false });
});
try {
frame.src = url.href;
// Hidden at first, so a quickly refused session shows nothing.
document.body.append(dialog);
wait(READY_TIMEOUT_MS);
loadingTimer = setTimeout(() => {
dialog.classList.add('is-loading');
show();
}, LOADING_DELAY_MS);
} catch {
finish(null);
}
});
+194
View File
@@ -0,0 +1,194 @@
// @vitest-environment jsdom
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { canFrameGuiEmbed, showFeedbackEmbed } from './guiEmbed.js';
const GUI_ORIGIN = 'https://puter.test';
const USER = '0b1c9a52-3f0e-4c7a-9a65-2f6d1e8b7c40';
// jsdom has no top-layer dialogs.
HTMLDialogElement.prototype.showModal = function () {
this.open = true;
};
const frame = () => document.querySelector('dialog.puter-feedback-embed iframe');
/** A message from the embed, by default from its frame on the GUI origin. */
const fromEmbed = (data, { origin = GUI_ORIGIN, source = frame().contentWindow } = {}) =>
window.dispatchEvent(new MessageEvent('message', { data, origin, source }));
beforeEach(() => {
document.body.innerHTML = '';
});
afterEach(() => {
vi.useRealTimers();
});
describe('canFrameGuiEmbed', () => {
it.each([
['https://notes.puter.com/a', 'https://puter.com', true],
['https://NOTES.puter.com./a', 'https://puter.com', true],
['http://notes.puter.localhost:4100/', 'http://puter.localhost:4100', true],
['https://puter.com/', 'https://puter.com', false],
['https://a.b.puter.com/', 'https://puter.com', false],
['https://notputer.com/', 'https://puter.com', false],
['https://notes.puter.com.evil.com/', 'https://puter.com', false],
['http://notes.puter.com/', 'https://puter.com', false],
['https://notes.puter.com:8443/', 'https://puter.com', false],
['https://notes.puter.site/', 'https://puter.com', false],
[undefined, 'https://puter.com', false],
])('%s under %s is %s', (page, gui, expected) => {
expect(canFrameGuiEmbed(page, gui)).toBe(expected);
});
});
describe('showFeedbackEmbed', () => {
it('loads the embed for the named user, hidden until it is ready', () => {
showFeedbackEmbed({ guiOrigin: `${GUI_ORIGIN}/`, userUuid: USER });
const url = new URL(frame().src);
expect(url.origin + url.pathname).toBe(`${GUI_ORIGIN}/embed/feedback`);
expect(url.search).toBe('');
expect(new URLSearchParams(url.hash.slice(1)).get('user')).toBe(USER);
expect(frame().getAttribute('referrerpolicy')).toBe('origin');
// WebKit hands the frame this as its preferred scheme; it must follow the OS.
expect(frame().getAttribute('style')).toContain('color-scheme: light dark');
expect(frame().closest('dialog').open).toBe(false);
});
it('shows the dialog once ready and resolves what it reports, restoring focus', async () => {
const trigger = document.createElement('button');
document.body.append(trigger);
trigger.focus();
const result = showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
const dialog = frame().closest('dialog');
fromEmbed({ msg: 'feedbackEmbed', state: 'loading' });
expect(dialog.open).toBe(false);
fromEmbed({ msg: 'feedbackEmbed', state: 'ready' });
expect(dialog.open).toBe(true);
expect(document.activeElement).toBe(frame());
fromEmbed({ msg: 'feedbackDialogClosed', sent: true });
await expect(result).resolves.toEqual({ sent: true });
expect(dialog.isConnected).toBe(false);
expect(document.activeElement).toBe(trigger);
});
it('shows a loading scrim over the page while a slow session check runs, then the dialog', async () => {
vi.useFakeTimers();
const trigger = document.createElement('button');
document.body.append(trigger);
trigger.focus();
const result = showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
const dialog = frame().closest('dialog');
fromEmbed({ msg: 'feedbackEmbed', state: 'loading' });
await vi.advanceTimersByTimeAsync(199);
expect(dialog.open).toBe(false);
await vi.advanceTimersByTimeAsync(1);
expect(dialog.open).toBe(true);
expect(dialog.classList.contains('is-loading')).toBe(true);
expect(dialog.querySelector('[role="status"]')).not.toBeNull();
fromEmbed({ msg: 'feedbackEmbed', state: 'ready' });
expect(dialog.classList.contains('is-loading')).toBe(false);
expect(document.activeElement).toBe(frame());
fromEmbed({ msg: 'feedbackDialogClosed', sent: true });
await expect(result).resolves.toEqual({ sent: true });
expect(document.activeElement).toBe(trigger);
});
it('never shows the loading scrim when the embed is ready quickly', async () => {
vi.useFakeTimers();
showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
const dialog = frame().closest('dialog');
fromEmbed({ msg: 'feedbackEmbed', state: 'ready' });
await vi.advanceTimersByTimeAsync(1000);
expect(dialog.open).toBe(true);
expect(dialog.classList.contains('is-loading')).toBe(false);
});
it('takes the loading scrim down for the popup when the embed turns out unavailable', async () => {
vi.useFakeTimers();
const trigger = document.createElement('button');
document.body.append(trigger);
trigger.focus();
const result = showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
await vi.advanceTimersByTimeAsync(200);
fromEmbed({ msg: 'feedbackEmbed', state: 'unavailable' });
await expect(result).resolves.toBeNull();
expect(document.querySelector('dialog')).toBeNull();
expect(document.activeElement).toBe(trigger);
});
it.each([
['Escape', (dialog) => dialog.dispatchEvent(new Event('cancel', { cancelable: true }))],
['a click on the scrim', (dialog) => dialog.querySelector('[role="status"]').click()],
])('lets %s cancel while loading, without falling back to the popup', async (_label, cancel) => {
vi.useFakeTimers();
const result = showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
await vi.advanceTimersByTimeAsync(200);
cancel(frame().closest('dialog'));
await expect(result).resolves.toEqual({ sent: false });
expect(document.querySelector('dialog')).toBeNull();
});
it('resolves null without showing anything when the embed is unavailable', async () => {
const result = showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
const dialog = frame().closest('dialog');
fromEmbed({ msg: 'feedbackEmbed', state: 'loading' });
fromEmbed({ msg: 'feedbackEmbed', state: 'unavailable' });
await expect(result).resolves.toBeNull();
expect(dialog.open).toBe(false);
expect(dialog.isConnected).toBe(false);
});
it('ignores messages from another origin or another window', async () => {
const result = showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
fromEmbed({ msg: 'feedbackDialogClosed', sent: true }, { origin: 'https://other.test' });
fromEmbed({ msg: 'feedbackDialogClosed', sent: true }, { source: window });
fromEmbed({ msg: 'feedbackEmbed', state: 'ready' }, { origin: 'https://notes.puter.test' });
expect(frame().closest('dialog').open).toBe(false);
fromEmbed({ msg: 'feedbackDialogClosed', sent: false });
await expect(result).resolves.toEqual({ sent: false });
});
it('resolves null when the frame loads without the embed answering', async () => {
vi.useFakeTimers();
const result = showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
frame().dispatchEvent(new Event('load'));
await vi.advanceTimersByTimeAsync(1000);
await expect(result).resolves.toBeNull();
expect(document.querySelector('dialog')).toBeNull();
});
it('gives a loading embed time to check the session, then gives up', async () => {
vi.useFakeTimers();
const result = showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
fromEmbed({ msg: 'feedbackEmbed', state: 'loading' });
frame().dispatchEvent(new Event('load'));
let settled = false;
result.then(() => {
settled = true;
});
await vi.advanceTimersByTimeAsync(5000);
expect(settled).toBe(false);
await vi.advanceTimersByTimeAsync(10_000);
await expect(result).resolves.toBeNull();
});
it('keeps the dialog open on Escape outside the frame, since the embed owns dismissal', () => {
showFeedbackEmbed({ guiOrigin: GUI_ORIGIN, userUuid: USER });
fromEmbed({ msg: 'feedbackEmbed', state: 'ready' });
const dialog = frame().closest('dialog');
const cancel = new Event('cancel', { cancelable: true });
dialog.dispatchEvent(cancel);
expect(cancel.defaultPrevented).toBe(true);
expect(dialog.isConnected).toBe(true);
});
it('resolves null for an unusable GUI origin', async () => {
await expect(showFeedbackEmbed({ guiOrigin: 'not a url', userUuid: USER })).resolves.toBeNull();
expect(document.querySelector('dialog')).toBeNull();
});
});
+37 -3
View File
@@ -2,6 +2,7 @@ import EventListener from '../lib/EventListener.js';
import { PuterJSError } from '../lib/PuterJSError.js';
import FSItem from './FSItem.js';
import PuterDialog from './PuterDialog.js';
import { canFrameGuiEmbed, showFeedbackEmbed } from '../lib/guiEmbed.js';
import { checkPermissions } from './perms/lib/holds.js';
@@ -528,6 +529,9 @@ export class UIModule extends EventListener {
#guiFeatures = new URLSearchParams(globalThis.location?.search ?? '')
.get('puter.gui_features')?.split(',') ?? [];
// Set while the framed feedback dialog is open on a first-party page.
#feedbackEmbedOpen = false;
// Holds the callback functions for the various events
// that are triggered when a watched item has changed.
itemWatchCallbackFunctions = [];
@@ -2249,9 +2253,12 @@ export class UIModule extends EventListener {
* have opted in by setting `feedbackEnabled` on the app — otherwise the
* dialog tells the user feedback is unavailable.
*
* In the `app` environment the Puter desktop renders the dialog; in the
* `web` environment a puter.com popup hosts it (signing the user in first
* if needed). Every other environment resolves `false`.
* In the `app` environment the Puter desktop renders the dialog. In the
* `web` environment a first-party page one label under the Puter origin
* shows it in a frame over the page when the user's Puter session is the
* one the page is signed in as; otherwise a puter.com popup hosts it
* (signing the user in first if needed). Every other environment
* resolves `false`.
*
* @returns {Promise<boolean>} `true` when the user submitted feedback,
* `false` when the dialog was dismissed or feedback is unavailable.
@@ -2276,6 +2283,10 @@ export class UIModule extends EventListener {
if ( this.env !== 'web' ) {
return false;
}
const embedded = await this.#showFeedbackEmbed();
if ( embedded ) {
return embedded.sent;
}
// The app's identity is deliberately NOT in the popup URL: the GUI
// derives it from the browser-attested opener origin, so a link
// can't open a feedback dialog in another app's name.
@@ -2287,6 +2298,29 @@ export class UIModule extends EventListener {
return result?.sent === true;
};
/**
* The framed feedback dialog for a first-party page that is signed in, or
* null when the popup has to host it. One at a time: a call while one is
* open resolves `{ sent: false }`, as the desktop does.
*
* @returns {Promise<{ sent: boolean } | null>}
*/
async #showFeedbackEmbed () {
if ( ! globalThis.document?.body ) return null;
if ( ! canFrameGuiEmbed(globalThis.location?.href, puter.defaultGUIOrigin) ) return null;
if ( ! puter.auth?.isSignedIn?.() ) return null;
if ( this.#feedbackEmbedOpen ) return { sent: false };
this.#feedbackEmbedOpen = true;
try {
const known = await Promise.resolve(puter.whoami ?? puter.whoamiCache_).catch(() => null);
const user = known?.uuid ? known : await puter.auth.getUser().catch(() => null);
if ( typeof user?.uuid !== 'string' || ! user.uuid ) return null;
return await showFeedbackEmbed({ guiOrigin: puter.defaultGUIOrigin, userUuid: user.uuid });
} finally {
this.#feedbackEmbedOpen = false;
}
}
/**
* Opens `/action/<action>` on the GUI origin in a popup and waits for
* its single result message. Resolves the message's data, or `null` when
@@ -0,0 +1,111 @@
// @vitest-environment jsdom
// @vitest-environment-options {"url": "https://notes.puter.test/doc"}
import { beforeEach, describe, expect, it, vi } from 'vitest';
const { UIModule } = await import('./UI.js');
// jsdom has no top-layer dialogs.
HTMLDialogElement.prototype.showModal = function () {
this.open = true;
};
const GUI_ORIGIN = 'https://puter.test';
const USER = '0b1c9a52-3f0e-4c7a-9a65-2f6d1e8b7c40';
const makeUI = () => new UIModule(
{ env: 'web', authToken: 'token-1', APIOrigin: 'https://api.puter.test', util: {} },
{},
);
const embedFrame = () => document.querySelector('dialog.puter-feedback-embed iframe');
const fromEmbed = (data) => window.dispatchEvent(new MessageEvent('message', {
data, origin: GUI_ORIGIN, source: embedFrame().contentWindow,
}));
let popup;
const fakePuter = ({ guiOrigin = GUI_ORIGIN, signedIn = true } = {}) => {
globalThis.puter = {
defaultGUIOrigin: guiOrigin,
auth: {
isSignedIn: () => signedIn,
getUser: vi.fn(async () => ({ uuid: USER })),
},
};
};
beforeEach(() => {
document.body.innerHTML = '';
popup = { closed: false, close: vi.fn() };
window.open = vi.fn(() => popup);
Object.defineProperty(navigator, 'userActivation', {
configurable: true,
value: { hasBeenActive: true, isActive: true },
});
fakePuter();
});
describe('ui.showFeedbackDialog on a first-party page', () => {
it('shows the dialog in a frame instead of a popup', async () => {
const result = makeUI().showFeedbackDialog();
await vi.waitFor(() => expect(embedFrame()).not.toBeNull());
fromEmbed({ msg: 'feedbackEmbed', state: 'ready' });
fromEmbed({ msg: 'feedbackDialogClosed', sent: true });
await expect(result).resolves.toBe(true);
expect(window.open).not.toHaveBeenCalled();
});
it('names the user puter.js already knows, without asking the API again', async () => {
globalThis.puter.whoamiCache_ = Promise.resolve({ uuid: USER });
const result = makeUI().showFeedbackDialog();
await vi.waitFor(() => expect(embedFrame()).not.toBeNull());
expect(new URLSearchParams(new URL(embedFrame().src).hash.slice(1)).get('user')).toBe(USER);
expect(globalThis.puter.auth.getUser).not.toHaveBeenCalled();
fromEmbed({ msg: 'feedbackDialogClosed', sent: false });
await expect(result).resolves.toBe(false);
});
it('uses the popup when the embed cannot serve the user', async () => {
const result = makeUI().showFeedbackDialog();
await vi.waitFor(() => expect(embedFrame()).not.toBeNull());
fromEmbed({ msg: 'feedbackEmbed', state: 'unavailable' });
await vi.waitFor(() => expect(window.open).toHaveBeenCalledTimes(1));
const url = new URL(window.open.mock.calls[0][0]);
expect(url.origin + url.pathname).toBe(`${GUI_ORIGIN}/action/send-feedback`);
window.dispatchEvent(new MessageEvent('message', {
data: { msg: 'feedbackDialogClosed', sent: true, original_msg_id: url.searchParams.get('msg_id') },
origin: GUI_ORIGIN,
source: popup,
}));
await expect(result).resolves.toBe(true);
});
it('answers false to a second call while the framed dialog is open', async () => {
const ui = makeUI();
const first = ui.showFeedbackDialog();
await vi.waitFor(() => expect(embedFrame()).not.toBeNull());
await expect(ui.showFeedbackDialog()).resolves.toBe(false);
expect(document.querySelectorAll('dialog.puter-feedback-embed')).toHaveLength(1);
fromEmbed({ msg: 'feedbackDialogClosed', sent: false });
await expect(first).resolves.toBe(false);
});
it('goes straight to the popup, which can sign in, when the page is signed out', async () => {
fakePuter({ signedIn: false });
makeUI().showFeedbackDialog();
await vi.waitFor(() => expect(window.open).toHaveBeenCalledTimes(1));
expect(embedFrame()).toBeNull();
});
});
describe('ui.showFeedbackDialog on a third-party page', () => {
it('uses the popup', async () => {
fakePuter({ guiOrigin: 'https://other-puter.test' });
makeUI().showFeedbackDialog();
await vi.waitFor(() => expect(window.open).toHaveBeenCalledTimes(1));
expect(embedFrame()).toBeNull();
expect(globalThis.puter.auth.getUser).not.toHaveBeenCalled();
});
});
+3 -3
View File
@@ -1,5 +1,5 @@
import { PuterModule } from '../../lib/PuterModule.js';
import { convo } from './convo.js';
import { realtime } from './realtime.js';
import { chat } from './chat.js';
import { txt2img } from './image.js';
import { listModelProviders, listModels } from './models.js';
@@ -49,7 +49,7 @@ export class AIModule extends PuterModule {
// runtime so destructured calls (`const { chat } = puter.ai`) keep
// working like the old arrow fields did.
chat = chat;
convo = convo;
realtime = realtime;
img2txt = img2txt;
speech2txt = speech2txt;
speech2speech = speech2speech;
@@ -66,7 +66,7 @@ export class AIModule extends PuterModule {
/** @type {unknown} */ (this)
);
for ( const name of [
'chat', 'convo', 'img2txt', 'speech2txt', 'speech2speech',
'chat', 'realtime', 'img2txt', 'speech2txt', 'speech2speech',
'txt2img', 'txt2vid', 'listModels', 'listModelProviders',
] ) {
methods[name] = methods[name].bind(this);
@@ -4,17 +4,17 @@ import {
CLOSE_MESSAGE_TOO_LARGE,
RealtimeSession,
createRealtimeSocket,
} from './convoTransport.js';
} from './realtimeTransport.js';
import { RealtimeEvents } from './realtimeEvents.js';
import { ConvoMicrophone, ConvoSpeaker } from './convoMedia.js';
import { ConvoHistory } from './convoHistory.js';
import { RealtimeMicrophone, RealtimeSpeaker } from './realtimeMedia.js';
import { RealtimeHistory } from './realtimeHistory.js';
/** @typedef {import('./types.js').ConvoOptions} ConvoOptions */
/** @typedef {import('./types.js').ConvoSessionOptions} ConvoSessionOptions */
/** @typedef {import('./types.js').ConvoPlaybackPosition} ConvoPlaybackPosition */
/** @typedef {import('./types.js').RealtimeOptions} RealtimeOptions */
/** @typedef {import('./types.js').RealtimeSessionOptions} RealtimeSessionOptions */
/** @typedef {import('./types.js').RealtimePlaybackPosition} RealtimePlaybackPosition */
/** @typedef {import('./types.js').RealtimeServerEvent} RealtimeServerEvent */
/** @typedef {import('./types.js').RealtimeClientEvent} RealtimeClientEvent */
/** @typedef {{ code: string; message: string }} ConvoError */
/** @typedef {{ code: string; message: string }} RealtimeError */
/**
* A pending reconnect notice. A replacement session opens right away and takes
* over once the conversation is idle.
@@ -29,7 +29,7 @@ import { ConvoHistory } from './convoHistory.js';
* probe: RealtimeSession | null;
* created: RealtimeServerEvent | null;
* stopProbe: (() => void) | null;
* }} ConvoHandover
* }} RealtimeHandover
* @internal
*/
@@ -106,7 +106,7 @@ const encoder = new TextEncoder();
/**
* @param {string} code
* @param {string} message
* @returns {ConvoError}
* @returns {RealtimeError}
*/
const failure = (code, message) => ({ code, message });
@@ -138,7 +138,7 @@ function mergeSession(base, patch) {
}
/**
* @param {ConvoOptions} options
* @param {RealtimeOptions} options
* @param {boolean} [allowModel]
* @returns {void}
*/
@@ -261,12 +261,12 @@ function encodeEvent(event) {
/**
* A prepared realtime conversation with optional media helpers.
*
* @extends {RealtimeEvents<import('./types.js').ConvoEventMap>}
* @extends {RealtimeEvents<import('./types.js').RealtimeEventMap>}
*/
export class Conversation extends RealtimeEvents {
/** @type {import('../../index.js').Puter} */
#puter;
/** @type {ConvoOptions} */
/** @type {RealtimeOptions} */
#options;
/** @type {'prepared' | 'connecting' | 'connected' | 'closed'} */
#state = 'prepared';
@@ -285,7 +285,7 @@ export class Conversation extends RealtimeEvents {
* type: string;
* matches: (event: RealtimeServerEvent) => boolean;
* resolve: (event: RealtimeServerEvent) => void;
* reject: (error: ConvoError) => void;
* reject: (error: RealtimeError) => void;
* timer: ReturnType<typeof setTimeout>;
* eventId?: string;
* anyError?: boolean;
@@ -313,7 +313,7 @@ export class Conversation extends RealtimeEvents {
/**
* @type {{
* resolve: () => void;
* reject: (error: ConvoError) => void;
* reject: (error: RealtimeError) => void;
* }[]}
*/
#reconnectWaiters = [];
@@ -330,8 +330,8 @@ export class Conversation extends RealtimeEvents {
*/
#interruption = null;
#cancelled = new Set();
#history = new ConvoHistory((id) => this.#cancelled.has(id));
/** @type {ConvoHandover | null} */
#history = new RealtimeHistory((id) => this.#cancelled.has(id));
/** @type {RealtimeHandover | null} */
#handover = null;
// Speech started and its audio isn't committed or cleared yet.
#speaking = false;
@@ -358,17 +358,17 @@ export class Conversation extends RealtimeEvents {
/** @type {Set<string>} */
#committedItems = new Set();
// Replayed `conversation.item.create` event IDs on the current session.
/** @type {Map<string, import('./convoHistory.js').ReplayEntry>} */
/** @type {Map<string, import('./realtimeHistory.js').ReplayEntry>} */
#replayEvents = new Map();
#replayReported = false;
/** @type {ConvoMicrophone} */
/** @type {RealtimeMicrophone} */
microphone;
/** @type {ConvoSpeaker} */
/** @type {RealtimeSpeaker} */
speaker;
/**
* @param {import('../../index.js').Puter} puter
* @param {ConvoOptions} options
* @param {RealtimeOptions} options
* @internal
*/
constructor(puter, options) {
@@ -380,8 +380,8 @@ export class Conversation extends RealtimeEvents {
model: options.model ?? DEFAULT_MODEL,
};
this.#turnDetection = this.#options.turnDetection !== false;
this.microphone = new ConvoMicrophone(this);
this.speaker = new ConvoSpeaker(this);
this.microphone = new RealtimeMicrophone(this);
this.speaker = new RealtimeSpeaker(this);
}
/** @returns {'prepared' | 'connecting' | 'connected' | 'closed'} */
@@ -807,7 +807,7 @@ export class Conversation extends RealtimeEvents {
* restarting, is retried with backoff while the current session keeps
* working; any other refusal ends the attempts.
*
* @param {ConvoHandover} handover
* @param {RealtimeHandover} handover
* @returns {Promise<void>}
*/
async #openReplacement(handover) {
@@ -887,7 +887,7 @@ export class Conversation extends RealtimeEvents {
}
/**
* @param {ConvoHandover} handover
* @param {RealtimeHandover} handover
* @param {string} [code]
* @returns {void}
*/
@@ -1011,7 +1011,7 @@ export class Conversation extends RealtimeEvents {
}
/**
* @param {ConvoSessionOptions} options
* @param {RealtimeSessionOptions} options
* @returns {Promise<void>}
*/
async configure(options) {
@@ -1025,7 +1025,7 @@ export class Conversation extends RealtimeEvents {
* new session needs. A partial update leaves settings made through native
* events alone.
*
* @param {ConvoSessionOptions} options
* @param {RealtimeSessionOptions} options
* @param {boolean} [full]
* @param {number} [ackMs]
* @returns {Promise<void>}
@@ -1168,7 +1168,7 @@ export class Conversation extends RealtimeEvents {
* Stops local playback immediately and waits for cancellation and
* truncation.
*
* @param {ConvoPlaybackPosition} [position]
* @param {RealtimePlaybackPosition} [position]
* @returns {Promise<void>}
*/
interrupt(position) {
@@ -1181,7 +1181,7 @@ export class Conversation extends RealtimeEvents {
}
/**
* @param {ConvoPlaybackPosition | undefined} position
* @param {RealtimePlaybackPosition | undefined} position
* @param {'manual' | 'speech'} source
* @returns {Promise<void>}
*/
@@ -1323,7 +1323,7 @@ export class Conversation extends RealtimeEvents {
}
/**
* @param {ConvoError} error
* @param {RealtimeError} error
* @returns {void}
* @internal
*/
@@ -1347,7 +1347,7 @@ export class Conversation extends RealtimeEvents {
promptIfUpgradeRequired(
error,
{
method: 'puter.ai.convo',
method: 'puter.ai.realtime',
subscriptionMessage:
'Realtime conversations require a subscription.',
},
@@ -1404,7 +1404,7 @@ export class Conversation extends RealtimeEvents {
}
}
if (event.type === RECONNECT_NOTICE && !this.#handover) {
/** @type {ConvoHandover} */
/** @type {RealtimeHandover} */
const handover = {
reason:
typeof event.reason === 'string' && event.reason
@@ -1707,7 +1707,7 @@ export class Conversation extends RealtimeEvents {
/** @returns {string} */
#eventId() {
return `puter_convo_${++this.#sequence}`;
return `puter_realtime_${++this.#sequence}`;
}
/**
* Sends events together or not at all. `limit` caps the socket's send
@@ -1830,7 +1830,7 @@ export class Conversation extends RealtimeEvents {
return this.#wait(type, matches, event.event_id, false, timeoutMs);
}
/**
* @param {ConvoError} error
* @param {RealtimeError} error
* @returns {void}
*/
#rejectPending(error) {
@@ -1847,10 +1847,10 @@ export class Conversation extends RealtimeEvents {
* connection.
*
* @this {import('./index.js').AIModule}
* @param {ConvoOptions} [options]
* @param {RealtimeOptions} [options]
* @returns {Promise<Conversation>}
*/
export async function convo(options = {}) {
export async function realtime(options = {}) {
validateOptions(options, true);
const settings = {
...options,
@@ -1,5 +1,5 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { Conversation, convo } from './convo.js';
import { Conversation, realtime } from './realtime.js';
class FakeWebSocket extends EventTarget {
static instances = [];
@@ -283,14 +283,14 @@ describe('conversation preparation', () => {
'rejects invalid options before opening a socket (%j)',
async (options) => {
await expect(
convo.call({ puter: owner() }, options),
realtime.call({ puter: owner() }, options),
).rejects.toMatchObject({ code: expect.any(String) });
expect(FakeWebSocket.instances).toHaveLength(0);
},
);
it('requires authentication outside a browser', async () => {
await expect(
convo.call({ puter: { ...owner(), authToken: null } }),
realtime.call({ puter: { ...owner(), authToken: null } }),
).rejects.toMatchObject({ code: 'auth_required' });
});
it('reports canceled browser authentication', async () => {
@@ -304,7 +304,7 @@ describe('conversation preparation', () => {
},
},
};
await expect(convo.call({ puter })).rejects.toMatchObject({
await expect(realtime.call({ puter })).rejects.toMatchObject({
code: 'auth_canceled',
});
});
@@ -988,7 +988,7 @@ it('preflights both events of a text turn before sending either', async () => {
socket().bufferedAmount =
192 * 1024 -
new TextEncoder().encode(
JSON.stringify({ ...item, event_id: 'puter_convo_99' }),
JSON.stringify({ ...item, event_id: 'puter_realtime_99' }),
).byteLength;
expect(() => c.sendText(text)).toThrow(
expect.objectContaining({ code: 'queue_full' }),
@@ -4,7 +4,7 @@
* An item `conversation.item.create` accepts: a text message, a function call
* or a function call's output.
*
* @typedef {Record<string, unknown> & { type: string }} ConvoReplayItem
* @typedef {Record<string, unknown> & { type: string }} RealtimeReplayItem
*/
/**
@@ -29,7 +29,7 @@
* @property {string} [output]
*/
/** @typedef {{ record: HistoryRecord; item: ConvoReplayItem }} ReplayEntry */
/** @typedef {{ record: HistoryRecord; item: RealtimeReplayItem }} ReplayEntry */
const ITEM_EVENTS = new Set([
'conversation.item.added',
@@ -99,7 +99,7 @@ function partText(part) {
*
* @internal
*/
export class ConvoHistory {
export class RealtimeHistory {
/** @type {HistoryRecord[]} */
#records = [];
/** @type {Map<string, HistoryRecord>} */
@@ -426,7 +426,7 @@ export class ConvoHistory {
* The `conversation.item.create` item that recreates `record`, or `null`.
*
* @param {HistoryRecord} record
* @returns {ConvoReplayItem | null}
* @returns {RealtimeReplayItem | null}
*/
#replayItem(record) {
if (!record.final) return null;
@@ -1,5 +1,5 @@
import { describe, expect, it } from 'vitest';
import { ConvoHistory } from './convoHistory.js';
import { RealtimeHistory } from './realtimeHistory.js';
const user = (id, content) => ({
type: 'conversation.item.added',
@@ -10,9 +10,9 @@ const texts = (history) =>
.replay(40, 48 * 1024)
.entries.map(({ item }) => item.content[0].text);
describe('ConvoHistory', () => {
describe('RealtimeHistory', () => {
it('cuts text without word breaks at the played share', () => {
const history = new ConvoHistory(() => false);
const history = new RealtimeHistory(() => false);
history.observe({
type: 'response.output_item.added',
response_id: 'r',
@@ -40,7 +40,7 @@ describe('ConvoHistory', () => {
});
it('keeps a known transcript when a later copy of the item lacks it', () => {
const history = new ConvoHistory(() => false);
const history = new RealtimeHistory(() => false);
history.observe(user('u', [{ type: 'input_audio', transcript: null }]));
expect(texts(history)).toEqual([]);
history.observe({
@@ -57,7 +57,7 @@ describe('ConvoHistory', () => {
});
it('keeps a bounded number of the newest items', () => {
const history = new ConvoHistory(() => false);
const history = new RealtimeHistory(() => false);
for (let i = 0; i < 100; i++)
history.observe(
user(`u${i}`, [{ type: 'input_text', text: `${i}` }]),
@@ -67,7 +67,7 @@ describe('ConvoHistory', () => {
});
it('stops at the first item over the byte budget instead of leaving a gap', () => {
const history = new ConvoHistory(() => false);
const history = new RealtimeHistory(() => false);
history.observe(user('u1', [{ type: 'input_text', text: 'old' }]));
history.observe(
user('u2', [{ type: 'input_text', text: 'x'.repeat(50_000) }]),
@@ -79,7 +79,7 @@ describe('ConvoHistory', () => {
it.each(['x'.repeat(49 * 1024), '😀'.repeat(13 * 1024)])(
'reports an oversized newest item without truncating it',
(text) => {
const history = new ConvoHistory(() => false);
const history = new RealtimeHistory(() => false);
history.observe(
user('old', [{ type: 'input_text', text: 'tiny' }]),
);
@@ -96,7 +96,7 @@ describe('ConvoHistory', () => {
);
it('waits on every unanswered call since the last user message', () => {
const history = new ConvoHistory(() => false);
const history = new RealtimeHistory(() => false);
const call = (id, status = 'completed') =>
history.observe({
type: 'response.output_item.done',
@@ -133,7 +133,7 @@ describe('ConvoHistory', () => {
});
it('reports the new session copies of replayed items until they are done', () => {
const history = new ConvoHistory(() => false);
const history = new RealtimeHistory(() => false);
history.observe(user('old', [{ type: 'input_text', text: 'hi' }]));
history.replayed(history.replay(40, 48 * 1024).entries);
const copy = user('new', [{ type: 'input_text', text: 'hi' }]);
@@ -1,6 +1,6 @@
/** @typedef {import('./convo.js').Conversation} Conversation */
/** @typedef {import('./types.js').ConvoPlaybackPosition} ConvoPlaybackPosition */
/** @typedef {import('./types.js').ConvoSpeakerOptions} ConvoSpeakerOptions */
/** @typedef {import('./realtime.js').Conversation} Conversation */
/** @typedef {import('./types.js').RealtimePlaybackPosition} RealtimePlaybackPosition */
/** @typedef {import('./types.js').RealtimeSpeakerOptions} RealtimeSpeakerOptions */
/** @typedef {import('./types.js').RealtimeServerEvent} RealtimeServerEvent */
const SAMPLE_RATE = 24000;
@@ -69,7 +69,7 @@ function audioContext(sampleRate) {
// The worklet owns resampling. Chunks the main thread hasn't taken within
// about half a second are dropped rather than queued.
const CAPTURE_WORKLET = `
class PuterConvoCapture extends AudioWorkletProcessor {
class PuterRealtimeCapture extends AudioWorkletProcessor {
constructor() {
super();
this.samples = new Int16Array(480);
@@ -108,13 +108,13 @@ class PuterConvoCapture extends AudioWorkletProcessor {
return true;
}
}
registerProcessor('puter-convo-capture', PuterConvoCapture);
registerProcessor('puter-realtime-capture', PuterRealtimeCapture);
`;
/** Optional browser microphone capture. */
export class ConvoMicrophone {
export class RealtimeMicrophone {
/** @type {Conversation} */
#convo;
#conversation;
/** @type {MediaStream | null} */
#stream = null;
/** @type {AudioContext | null} */
@@ -131,11 +131,11 @@ export class ConvoMicrophone {
#cancelStart = null;
/**
* @param {Conversation} convo
* @param {Conversation} conversation
* @internal
*/
constructor(convo) {
this.#convo = convo;
constructor(conversation) {
this.#conversation = conversation;
}
/**
@@ -145,7 +145,7 @@ export class ConvoMicrophone {
*/
start() {
try {
this.#convo.assertConnected();
this.#conversation.assertConnected();
} catch (error) {
return Promise.reject(error);
}
@@ -184,7 +184,8 @@ export class ConvoMicrophone {
this.#context = context;
// A reconnect keeps capture running; chunks are dropped meanwhile.
const current = () =>
generation === this.#generation && this.#convo.state !== 'closed';
generation === this.#generation &&
this.#conversation.state !== 'closed';
const permission = navigator.mediaDevices
.getUserMedia({
audio: {
@@ -228,27 +229,27 @@ export class ConvoMicrophone {
}
if (!current())
throw { code: 'session_closed', message: 'Capture was stopped' };
this.#node = new AudioWorkletNode(context, 'puter-convo-capture');
this.#node = new AudioWorkletNode(context, 'puter-realtime-capture');
this.#source = context.createMediaStreamSource(stream);
const node = this.#node;
node.port.onmessage = ({ data }) => {
if (!current()) return;
node.port.postMessage('ack');
if (this.#muted || this.#convo.state !== 'connected') return;
if (this.#muted || this.#conversation.state !== 'connected') return;
try {
this.#convo.appendAudio(data.pcm);
this.#conversation.appendAudio(data.pcm);
} catch (error) {
// A congested socket, or one closing before the conversation
// knows it, drops the chunk instead of ending capture: a
// reconnect resumes it and a final close stops it.
if (DROPPED_CHUNK_ERRORS.has(error?.code)) return;
this.stop();
this.#convo.reportMediaError(error);
this.#conversation.reportMediaError(error);
}
};
node.onprocessorerror = () => {
this.stop();
this.#convo.reportMediaError({
this.#conversation.reportMediaError({
code: 'media_error',
message: 'Microphone processor failed',
});
@@ -298,9 +299,9 @@ export class ConvoMicrophone {
}
/** Optional bounded browser PCM playback. */
export class ConvoSpeaker {
export class RealtimeSpeaker {
/** @type {Conversation} */
#convo;
#conversation;
/** @type {AudioContext | null} */
#context = null;
/** @type {Promise<void> | null} */
@@ -317,7 +318,7 @@ export class ConvoSpeaker {
* }>}
*/
#sources = new Set();
/** @type {(ConvoPlaybackPosition & { responseId: string }) | null} */
/** @type {(RealtimePlaybackPosition & { responseId: string }) | null} */
#last = null;
#nextAt = 0;
#bargeIn = true;
@@ -325,11 +326,11 @@ export class ConvoSpeaker {
#overflowed = '';
/**
* @param {Conversation} convo
* @param {Conversation} conversation
* @internal
*/
constructor(convo) {
this.#convo = convo;
constructor(conversation) {
this.#conversation = conversation;
}
/**
* @returns {boolean}
@@ -365,12 +366,12 @@ export class ConvoSpeaker {
/**
* Starts playback without requesting microphone access.
*
* @param {ConvoSpeakerOptions} [options]
* @param {RealtimeSpeakerOptions} [options]
* @returns {Promise<void>}
*/
start(options = {}) {
try {
this.#convo.assertConnected();
this.#conversation.assertConnected();
} catch (error) {
return Promise.reject(error);
}
@@ -424,7 +425,7 @@ export class ConvoSpeaker {
const responseId = String(event.response_id ?? '');
if (this.#overflowed !== responseId) {
this.#overflowed = responseId;
this.#convo.reportMediaError({
this.#conversation.reportMediaError({
code: 'queue_full',
message: `Playback exceeds ${MAX_SCHEDULED_SECONDS} seconds`,
});
@@ -471,7 +472,7 @@ export class ConvoSpeaker {
this.#last.playedMs,
(offset + duration) * 1000,
);
if (!this.#sources.size) this.#convo.playbackEnded();
if (!this.#sources.size) this.#conversation.playbackEnded();
};
this.#nextAt = source.end;
node.start(start);
@@ -483,7 +484,7 @@ export class ConvoSpeaker {
* is scheduled, it is the end of what played for `responseId`.
*
* @param {string} [responseId] The response still generating, if any.
* @returns {ConvoPlaybackPosition | undefined}
* @returns {RealtimePlaybackPosition | undefined}
* @internal
*/
playbackPosition(responseId) {
@@ -503,7 +504,7 @@ export class ConvoSpeaker {
* Each item with scheduled audio the listener hasn't fully heard, in play
* order, with how much of it was heard.
*
* @returns {ConvoPlaybackPosition[]}
* @returns {RealtimePlaybackPosition[]}
* @internal
*/
unheard() {
@@ -511,7 +512,7 @@ export class ConvoSpeaker {
const heard =
(context?.currentTime ?? 0) -
(context?.outputLatency || context?.baseLatency || 0);
/** @type {Map<string, ConvoPlaybackPosition>} */
/** @type {Map<string, RealtimePlaybackPosition>} */
const items = new Map();
for (const source of this.#sources) {
if (source.end <= heard || items.has(source.itemId)) continue;
@@ -540,7 +541,7 @@ export class ConvoSpeaker {
this.#sources.clear();
this.#last = null;
this.#nextAt = 0;
if (playing) this.#convo.playbackEnded();
if (playing) this.#conversation.playbackEnded();
}
/**
@@ -1,5 +1,5 @@
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
import { ConvoMicrophone, ConvoSpeaker } from './convoMedia.js';
import { RealtimeMicrophone, RealtimeSpeaker } from './realtimeMedia.js';
class FakeContext {
static instances = [];
@@ -43,7 +43,7 @@ class FakeWorklet {
connect = vi.fn();
disconnect = vi.fn();
}
const makeConvo = () => ({
const makeConversation = () => ({
state: 'connected',
assertConnected() {
if (this.state !== 'connected') throw { code: 'session_closed' };
@@ -73,7 +73,7 @@ afterEach(async () => {
});
it('captures PCM, acknowledges bounded worklet chunks, mutes and releases tracks', async () => {
const c = makeConvo();
const c = makeConversation();
const audioTrack = track();
vi.stubGlobal('navigator', {
mediaDevices: {
@@ -83,7 +83,7 @@ it('captures PCM, acknowledges bounded worklet chunks, mutes and releases tracks
})),
},
});
const mic = new ConvoMicrophone(c);
const mic = new RealtimeMicrophone(c);
helpers.push(mic);
const pending = mic.start();
expect(mic.start()).toBe(pending);
@@ -115,7 +115,7 @@ it('releases a stream when permission resolves after stop', async () => {
}),
},
});
const mic = new ConvoMicrophone(makeConvo());
const mic = new RealtimeMicrophone(makeConversation());
helpers.push(mic);
const pending = mic.start();
mic.stop();
@@ -134,7 +134,7 @@ it('settles start() when stopped during the permission prompt', async () => {
}),
},
});
const mic = new ConvoMicrophone(makeConvo());
const mic = new RealtimeMicrophone(makeConversation());
helpers.push(mic);
const pending = mic.start();
mic.stop();
@@ -154,7 +154,7 @@ it('reports browser playback failures with SDK error codes', async () => {
}
},
);
const speaker = new ConvoSpeaker(makeConvo());
const speaker = new RealtimeSpeaker(makeConversation());
helpers.push(speaker);
await expect(speaker.start()).rejects.toEqual({
code: 'media_error',
@@ -162,14 +162,14 @@ it('reports browser playback failures with SDK error codes', async () => {
});
});
/** A started microphone over fake browser media. */
const startedMic = async (c = makeConvo()) => {
const startedMic = async (c = makeConversation()) => {
const audioTrack = track();
const getUserMedia = vi.fn(async () => ({
getTracks: () => [audioTrack],
getAudioTracks: () => [audioTrack],
}));
vi.stubGlobal('navigator', { mediaDevices: { getUserMedia } });
const mic = new ConvoMicrophone(c);
const mic = new RealtimeMicrophone(c);
helpers.push(mic);
await mic.start();
const port = FakeWorklet.instances.at(-1).port;
@@ -269,7 +269,7 @@ it('captures at the device rate', async () => {
return new FakeContext();
});
vi.stubGlobal('AudioContext', ContextSpy);
const mic = new ConvoMicrophone(makeConvo());
const mic = new RealtimeMicrophone(makeConversation());
helpers.push(mic);
await mic.start();
expect(ContextSpy).toHaveBeenCalledWith(undefined);
@@ -284,7 +284,7 @@ it('reports a denied permission', async () => {
},
},
});
const mic = new ConvoMicrophone(makeConvo());
const mic = new RealtimeMicrophone(makeConversation());
helpers.push(mic);
await expect(mic.start()).rejects.toMatchObject({
code: 'permission_denied',
@@ -292,7 +292,7 @@ it('reports a denied permission', async () => {
});
it('rejects unsupported capture environments', async () => {
vi.stubGlobal('navigator', {});
const mic = new ConvoMicrophone(makeConvo());
const mic = new RealtimeMicrophone(makeConversation());
helpers.push(mic);
await expect(mic.start()).rejects.toMatchObject({
code: 'unsupported_environment',
@@ -310,7 +310,7 @@ it('waits on the permission prompt but times out a stuck audio context', async (
}),
},
});
const mic = new ConvoMicrophone(makeConvo());
const mic = new RealtimeMicrophone(makeConversation());
helpers.push(mic);
const result = expect(mic.start()).rejects.toMatchObject({
code: 'media_timeout',
@@ -323,7 +323,7 @@ it('waits on the permission prompt but times out a stuck audio context', async (
expect(audioTrack.stop).toHaveBeenCalled();
});
it('tracks played position and clears playback immediately', async () => {
const speaker = new ConvoSpeaker(makeConvo());
const speaker = new RealtimeSpeaker(makeConversation());
helpers.push(speaker);
await speaker.start();
speaker.enqueue(new Int16Array(2400), {
@@ -348,8 +348,8 @@ it('tracks played position and clears playback immediately', async () => {
expect(context.close).toHaveBeenCalled();
});
it('bounds scheduled playback without interrupting the reply', async () => {
const c = makeConvo();
const speaker = new ConvoSpeaker(c);
const c = makeConversation();
const speaker = new RealtimeSpeaker(c);
helpers.push(speaker);
await speaker.start({ bargeIn: false });
const delta = {
@@ -371,7 +371,7 @@ it('bounds scheduled playback without interrupting the reply', async () => {
expect(speaker.bargeIn).toBe(false);
});
it('subtracts output latency and covers gaps between chunks', async () => {
const speaker = new ConvoSpeaker(makeConvo());
const speaker = new RealtimeSpeaker(makeConversation());
helpers.push(speaker);
await speaker.start();
const context = FakeContext.instances[0];
@@ -396,8 +396,8 @@ it('subtracts output latency and covers gaps between chunks', async () => {
});
it('reports playing until the last scheduled chunk ends', async () => {
const c = makeConvo();
const speaker = new ConvoSpeaker(c);
const c = makeConversation();
const speaker = new RealtimeSpeaker(c);
helpers.push(speaker);
await speaker.start();
const context = FakeContext.instances[0];
@@ -429,7 +429,7 @@ it('reports playing until the last scheduled chunk ends', async () => {
});
it('lists every item not fully heard, in play order', async () => {
const speaker = new ConvoSpeaker(makeConvo());
const speaker = new RealtimeSpeaker(makeConversation());
helpers.push(speaker);
await speaker.start();
const context = FakeContext.instances[0];
@@ -469,7 +469,7 @@ it('a late canceled permission request does not stop a newer capture', async ()
: Promise.resolve({ getTracks: () => [newTrack] }),
},
});
const mic = new ConvoMicrophone(makeConvo());
const mic = new RealtimeMicrophone(makeConversation());
helpers.push(mic);
const oldStart = mic.start();
mic.stop();
@@ -1,5 +1,5 @@
import { describe, expect, it, vi } from 'vitest';
import { RealtimeSession } from './convoTransport.js';
import { RealtimeSession } from './realtimeTransport.js';
/** Just enough of a WebSocket to drive the session's listeners. */
const fakeSocket = () => {
+11 -11
View File
@@ -602,7 +602,7 @@ export {};
/**
* Session settings for a live conversation.
*
* @typedef {Object} ConvoSessionOptions
* @typedef {Object} RealtimeSessionOptions
* @property {string} [instructions] Instructions, up to 8192 characters.
* @property {('text' | 'audio')[]} [output] Defaults to audio. Audio includes transcript events.
* @property {string} [voice] OpenAI voice identifier, up to 128 characters.
@@ -611,12 +611,12 @@ export {};
/**
* Options for preparing a conversation.
*
* @typedef {ConvoSessionOptions & { model?: string }} ConvoOptions
* @typedef {RealtimeSessionOptions & { model?: string }} RealtimeOptions
*/
/**
* How much of an assistant audio item a custom player has played.
*
* @typedef {Object} ConvoPlaybackPosition
* @typedef {Object} RealtimePlaybackPosition
* @property {string} itemId Assistant message item ID.
* @property {number} contentIndex Audio content index.
* @property {number} playedMs Played duration in milliseconds.
@@ -624,24 +624,24 @@ export {};
/**
* Built-in speaker options.
*
* @typedef {Object} ConvoSpeakerOptions
* @typedef {Object} RealtimeSpeakerOptions
* @property {boolean} [bargeIn] Stop replies when user speech is detected. Defaults to true.
*/
/**
* @typedef {{ type: 'text'; delta: string; responseId: string }} ConvoTextEvent */
* @typedef {{ type: 'text'; delta: string; responseId: string }} RealtimeTextEvent */
/**
* @typedef {{ type: 'audio'; pcm: Int16Array; sampleRate: number; responseId: string; itemId: string; contentIndex: number }} ConvoAudioEvent */
* @typedef {{ type: 'audio'; pcm: Int16Array; sampleRate: number; responseId: string; itemId: string; contentIndex: number }} RealtimeAudioEvent */
/**
* @typedef {{ type: 'transcript'; role: 'user' | 'assistant'; text: string; final: boolean }} ConvoTranscriptEvent */
* @typedef {{ type: 'transcript'; role: 'user' | 'assistant'; text: string; final: boolean }} RealtimeTranscriptEvent */
/**
* @typedef {{ type: 'interrupted'; source: 'manual' | 'speech'; responseId: string }} ConvoInterruptedEvent */
* @typedef {{ type: 'interrupted'; source: 'manual' | 'speech'; responseId: string }} RealtimeInterruptedEvent */
/**
* A finished reply. `usage` and `status` are OpenAI's, unchanged.
* @typedef {{ type: 'turn.done'; responseId: string | undefined; usage: unknown; status: unknown }} ConvoTurnDoneEvent */
* @typedef {{ type: 'turn.done'; responseId: string | undefined; usage: unknown; status: unknown }} RealtimeTurnDoneEvent */
/**
* An OpenAI `error` event, or an SDK media failure, which also carries `code` and `message` at the
* top level.
* @typedef {{ type: 'error'; error: { code?: string; message?: string } & Record<string, unknown>; code?: string; message?: string }} ConvoErrorEvent */
* @typedef {{ type: 'error'; error: { code?: string; message?: string } & Record<string, unknown>; code?: string; message?: string }} RealtimeErrorEvent */
/** Native provider events retain their provider-owned fields.
* @typedef {Record<string, RealtimeServerEvent> & { text: ConvoTextEvent; audio: ConvoAudioEvent; transcript: ConvoTranscriptEvent; interrupted: ConvoInterruptedEvent; 'turn.done': ConvoTurnDoneEvent; error: ConvoErrorEvent; ready: { type: 'ready'; model: string; provider: 'openai'; reconnected?: boolean }; reconnecting: { type: 'reconnecting'; code: number; reason: string }; context_trimmed: { type: 'context_trimmed'; omittedItems: number }; close: { type: 'close'; code: number; reason: string } }} ConvoEventMap */
* @typedef {Record<string, RealtimeServerEvent> & { text: RealtimeTextEvent; audio: RealtimeAudioEvent; transcript: RealtimeTranscriptEvent; interrupted: RealtimeInterruptedEvent; 'turn.done': RealtimeTurnDoneEvent; error: RealtimeErrorEvent; ready: { type: 'ready'; model: string; provider: 'openai'; reconnected?: boolean }; reconnecting: { type: 'reconnecting'; code: number; reason: string }; context_trimmed: { type: 'context_trimmed'; omittedItems: number }; close: { type: 'close'; code: number; reason: string } }} RealtimeEventMap */
+10 -1
View File
@@ -340,7 +340,12 @@ el.open();
The account button and Puter's app browser for a first-party app on a Puter subdomain (`notes.puter.com`). It is internal and left out of the public docs, because the app browser only works one subdomain under the Puter origin. It follows the page's own puter.js sign-in, never the Puter session's.
- Signed out, it shows **Sign in**, which calls `puter.auth.signIn()`. A temporary account counts as signed out.
- Signed in, it shows the user's profile picture, or the username's initial when there is none. Clicking it opens a menu with the username and **Sign out**, which calls `puter.auth.signOut()`.
- Signed in, it shows the user's profile picture, or the username's initial when there is none. Clicking it opens a menu with:
- The user's picture, username and email. The email line is hidden when whoami leaves it out, such as for an app that can't read it.
- **Manage account**, **Usage** and **Open Puter**, links that open the GUI origin's `/dashboard/#account`, `/dashboard/#usage` and `/` in a new tab.
- **Send feedback**, which calls `puter.ui.showFeedbackDialog()`.
- **Sign out**, which calls `puter.auth.signOut()`.
- The menu opens with focus on its first item. ArrowUp and ArrowDown move between items and wrap around, and Home and End jump to the first and last.
- On a page exactly one label under the Puter origin, with the same scheme and port, an **Apps** button opens the app browser (`/embed/apps`) in a panel under the button. It names the signed-in user as `#user=<uuid>`, so the app browser shows the Puter session's apps only to that user (see `doc/embedding-app-browser.md`), and passes a forced `theme` on as `theme=`. Elsewhere the button is hidden, because the server refuses the framing.
- The menu and the Apps panel hang under their button with a caret pointing at it, and close on Escape, a press outside them, or their button. They're in the browser's top layer where it has one, so a host's `transform`, `overflow` or `z-index` can't clip or cover them. On screens up to 600px wide, the Apps panel spans the width below the button.
- Closing the Apps panel keeps the app browser, so reopening it keeps its scroll and search. Any change to the page's sign-in discards it, so it never stays loaded for the previous user.
@@ -355,6 +360,10 @@ The account button and Puter's app browser for a first-party app on a Puter subd
| `sign-out-label` | `Sign out` | Sign-out menu item text |
| `close-label` | `Close` | The Apps panel's close button label |
| `account-label` | `Account` | Account button label, read as `Account: <username>` |
| `manage-account-label` | `Manage account` | Menu link to the account settings |
| `usage-label` | `Usage` | Menu link to the usage page |
| `open-puter-label` | `Open Puter` | Menu link to Puter |
| `feedback-label` | `Send feedback` | Menu item that opens the feedback dialog |
| `theme` | follows the OS | `light` or `dark`, for the component and the app browser in it |
Changing an attribute updates the component in place.
+138 -47
View File
@@ -4,7 +4,8 @@
*
* Signed out (a temporary account counts as signed out), it shows a Sign in
* button. Signed in, it shows the user's profile picture (or initial), which
* opens a menu with the username and Sign out. On a page one subdomain under
* opens a menu with the account, links to manage it, see its usage and open
* Puter, Send feedback, and Sign out. On a page one subdomain under
* the Puter origin, an Apps button opens Puter's app browser in a panel under
* it, for the same user: signed out, or signed in as an account other than
* the Puter session's, the browser shows only recommended apps. The browser
@@ -12,17 +13,20 @@
* Renders nothing inside the Puter desktop.
*
* Attributes: theme, apps-label, sign-in-label, sign-out-label, close-label,
* account-label
* account-label, manage-account-label, usage-label, open-puter-label,
* feedback-label
* Parts: apps-button, avatar, sign-in, menu, apps-panel
* Events: signin (detail.user), signout
*/
import PuterWebComponent from '../PuterWebComponent.js';
import { defaultFontFamily } from '../PuterDefaultStyles.js';
import { canFrameGuiEmbed } from '../../lib/guiEmbed.js';
const APPS_ICON = `<svg viewBox="0 0 24 24" width="20" height="20" aria-hidden="true" focusable="false">${
[5, 12, 19].flatMap(y => [5, 12, 19].map(x => `<circle cx="${x}" cy="${y}" r="2"/>`)).join('')
}</svg>`;
const EXTERNAL_ICON = '<svg class="menu-external" viewBox="0 0 32 32" width="11" height="11" aria-hidden="true" focusable="false"><path d="m26 28h-20a2 2 0 0 1 -2-2v-20a2 2 0 0 1 2-2h10v2h-10v20h20v-10h2v10a2 2 0 0 1 -2 2z"/><path d="m20 2v2h6.586l-8.586 8.586 1.414 1.414 8.586-8.586v6.586h2v-10z"/></svg>';
const CLOSE_ICON = '<svg viewBox="0 0 24 24" width="16" height="16" aria-hidden="true" focusable="false" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><path d="M6 6l12 12M18 6L6 18"/></svg>';
const LABELS = {
@@ -31,36 +35,23 @@ const LABELS = {
'sign-out-label': 'Sign out',
'close-label': 'Close',
'account-label': 'Account',
'manage-account-label': 'Manage account',
'usage-label': 'Usage',
'open-puter-label': 'Open Puter',
'feedback-label': 'Send feedback',
};
// Menu links, each opening a page of the Puter GUI in a new tab.
const MENU_LINKS = [
{ className: 'manage-account', label: 'manage-account-label', path: '/dashboard/#account' },
{ className: 'usage', label: 'usage-label', path: '/dashboard/#usage' },
{ className: 'open-puter', label: 'open-puter-label', path: '/' },
];
// Gap between a panel and the viewport edge, and between its caret and corners.
const EDGE = 8;
const CARET_INSET = 20;
/**
* Whether a page at `pageUrl` may frame the app browser served by
* `guiOrigin`: exactly one label under its host, on the same scheme and port.
*
* @param {string | undefined} pageUrl
* @param {string} guiOrigin
* @returns {boolean}
*/
export const canFrameAppBrowser = (pageUrl, guiOrigin) => {
let page;
let gui;
try {
page = new URL(pageUrl);
gui = new URL(guiOrigin);
} catch {
return false;
}
const host = page.hostname.replace(/\.$/, '').toLowerCase();
const suffix = `.${gui.hostname.toLowerCase()}`;
if ( page.protocol !== gui.protocol || page.port !== gui.port || !host.endsWith(suffix) ) return false;
const label = host.slice(0, -suffix.length);
return label.length > 0 && !label.includes('.');
};
class PuterUserMenu extends PuterWebComponent {
static get observedAttributes () {
return Object.keys(LABELS);
@@ -133,7 +124,7 @@ class PuterUserMenu extends PuterWebComponent {
}
_appsAllowed () {
return canFrameAppBrowser(globalThis.location?.href, globalThis.puter.defaultGUIOrigin);
return canFrameGuiEmbed(globalThis.location?.href, globalThis.puter.defaultGUIOrigin);
}
/** A temporary account has nothing to sign out of, so it gets Sign in. */
@@ -239,7 +230,8 @@ class PuterUserMenu extends PuterWebComponent {
text-align: center;
}
.avatar[aria-expanded="true"] { box-shadow: 0 0 0 3px rgba(8, 142, 240, .25); }
.avatar img {
.avatar img,
.menu-avatar img {
position: absolute;
inset: 0;
width: 100%;
@@ -308,31 +300,78 @@ class PuterUserMenu extends PuterWebComponent {
}
.menu {
min-width: 200px;
min-width: 220px;
max-width: min(300px, calc(100vw - ${EDGE * 2}px));
padding: 4px 0;
text-align: left;
}
.menu-user {
display: flex;
align-items: center;
gap: 10px;
padding: 8px 14px 10px;
margin-bottom: 4px;
border-bottom: 1px solid #eee;
color: #414650;
font-size: 14px;
}
.menu-avatar {
position: relative;
flex: none;
width: 40px;
height: 40px;
overflow: hidden;
border-radius: 50%;
background: linear-gradient(#34a5f8, #088ef0);
color: #fff;
font-size: 18px;
font-weight: 600;
line-height: 40px;
text-align: center;
}
.menu-identity {
display: flex;
flex-direction: column;
min-width: 0;
}
.menu-username,
.menu-email {
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.menu-username {
color: #414650;
font-size: 14px;
font-weight: 600;
}
.menu-email {
color: #6b7280;
font-size: 12px;
}
.menu-item {
display: block;
display: flex;
align-items: center;
gap: 4px;
box-sizing: border-box;
width: 100%;
padding: 8px 14px;
border: none;
background: none;
color: #333;
font-size: 14px;
line-height: normal;
text-align: left;
text-decoration: none;
cursor: pointer;
}
.menu-external {
flex: none;
fill: currentColor;
opacity: .7;
}
.menu-separator {
height: 1px;
margin: 4px 0;
background-color: #eee;
}
.menu-item:focus-visible {
outline: none;
@@ -434,7 +473,9 @@ class PuterUserMenu extends PuterWebComponent {
}
:host(.puter-theme-dark) .menu-user,
:host(.puter-theme-dark) .apps-header { border-color: #3a3a3a; }
:host(.puter-theme-dark) .menu-user,
:host(.puter-theme-dark) .menu-separator { background-color: #3a3a3a; }
:host(.puter-theme-dark) .menu-email { color: #a0a0a0; }
:host(.puter-theme-dark) .menu-username,
:host(.puter-theme-dark) .apps-title,
:host(.puter-theme-dark) .menu-item { color: #e6e6e6; }
:host(.puter-theme-dark) .menu-item:focus-visible { background-color: #3a3a3a; color: #fff; }
@@ -465,7 +506,16 @@ class PuterUserMenu extends PuterWebComponent {
<button type="button" class="avatar" part="avatar" aria-haspopup="menu" aria-expanded="false" hidden></button>
</div>
<div class="panel menu" part="menu" role="menu" tabindex="-1" hidden>
<div class="menu-user" hidden></div>
<div class="menu-user" hidden>
<span class="menu-avatar" aria-hidden="true"></span>
<span class="menu-identity">
<span class="menu-username"></span>
<span class="menu-email" hidden></span>
</span>
</div>
${MENU_LINKS.map(link => `<a class="menu-item ${link.className}" role="menuitem" target="_blank" rel="noopener"><span class="menu-label"></span>${EXTERNAL_ICON}</a>`).join('')}
<button type="button" class="menu-item feedback" role="menuitem"></button>
<div class="menu-separator" role="separator"></div>
<button type="button" class="menu-item sign-out" role="menuitem"></button>
</div>
<div class="panel apps-panel" part="apps-panel" role="dialog" tabindex="-1" hidden>
@@ -494,6 +544,10 @@ class PuterUserMenu extends PuterWebComponent {
if ( this.$('.menu').hidden ) this._openMenu();
else this._closePanels();
});
for ( const link of MENU_LINKS ) {
this.$(`.${link.className}`).addEventListener('click', () => this._closePanels());
}
this.$('.feedback').addEventListener('click', () => this._sendFeedback());
this.$('.sign-out').addEventListener('click', () => this._signOut());
this.$('.apps-close').addEventListener('click', () => this._closePanels({ returnFocus: true }));
this.shadowRoot.addEventListener('keydown', (event) => {
@@ -505,6 +559,8 @@ class PuterUserMenu extends PuterWebComponent {
} else if ( event.key === 'Tab' && open.classList.contains('menu') ) {
// Tab leaves the menu: it closes, and focus moves on from its button.
this._closePanels({ returnFocus: true });
} else if ( open.classList.contains('menu') && this._moveMenuFocus(event.key) ) {
event.preventDefault();
}
});
}
@@ -524,6 +580,10 @@ class PuterUserMenu extends PuterWebComponent {
close.title = this._label('close-label');
this.$('.sign-in').textContent = this._label('sign-in-label');
this.$('.sign-out').textContent = this._label('sign-out-label');
this.$('.feedback').textContent = this._label('feedback-label');
for ( const link of MENU_LINKS ) {
this.$(`.${link.className} .menu-label`).textContent = this._label(link.label);
}
this._renderAvatarLabel();
}
@@ -545,19 +605,17 @@ class PuterUserMenu extends PuterWebComponent {
const avatar = this.$('.avatar');
avatar.hidden = !this.#ready || !signedIn;
avatar.textContent = username ? username[0].toUpperCase() : '?';
if ( this.#picture ) {
const img = document.createElement('img');
img.alt = '';
// The initial stays underneath for a picture the page's CSP refuses.
img.addEventListener('error', () => img.remove());
img.src = this.#picture;
avatar.append(img);
}
this._fillAvatar(avatar, username);
this._renderAvatarLabel();
const menuUser = this.$('.menu-user');
menuUser.textContent = username;
menuUser.hidden = !username;
this.$('.menu-user').hidden = !username;
this.$('.menu-username').textContent = username;
this._fillAvatar(this.$('.menu-avatar'), username);
const email = typeof this.#user?.email === 'string' ? this.#user.email : '';
this.$('.menu-email').textContent = email;
this.$('.menu-email').hidden = !email;
for ( const link of MENU_LINKS ) {
this.$(`.${link.className}`).href = new URL(link.path, globalThis.puter.defaultGUIOrigin).href;
}
if ( avatar.hidden && !this.$('.menu').hidden ) this._closePanels();
if ( this.#ready && this.#focusSignIn ) {
this.#focusSignIn = false;
@@ -565,6 +623,18 @@ class PuterUserMenu extends PuterWebComponent {
}
}
/** The username's initial, with the profile picture over it once there is one. */
_fillAvatar (el, username) {
el.textContent = username ? username[0].toUpperCase() : '?';
if ( ! this.#picture ) return;
const img = document.createElement('img');
img.alt = '';
// The initial stays underneath for a picture the page's CSP refuses.
img.addEventListener('error', () => img.remove());
img.src = this.#picture;
el.append(img);
}
// -- Panels --
_openPanel () {
@@ -630,7 +700,23 @@ class PuterUserMenu extends PuterWebComponent {
_openMenu () {
const menu = this.$('.menu');
this._showPanel(menu);
this.$('.sign-out').focus({ preventScroll: true });
this.$('.menu-item').focus({ preventScroll: true });
}
/** Moves focus between menu items for an arrow, Home or End key; false for any other key. */
_moveMenuFocus (key) {
const items = [...this.$$('.menu .menu-item')];
const current = items.indexOf(this.shadowRoot.activeElement);
const last = items.length - 1;
const next = {
ArrowDown: current < 0 || current === last ? 0 : current + 1,
ArrowUp: current <= 0 ? last : current - 1,
Home: 0,
End: last,
}[key];
if ( next === undefined ) return false;
items[next].focus({ preventScroll: true });
return true;
}
_openApps () {
@@ -694,6 +780,11 @@ class PuterUserMenu extends PuterWebComponent {
);
}
_sendFeedback () {
this._closePanels();
void globalThis.puter.ui.showFeedbackDialog();
}
async _signOut () {
this.#focusSignIn = Boolean(this.shadowRoot.activeElement);
this._closePanels();
@@ -1,9 +1,9 @@
// @vitest-environment jsdom
// @vitest-environment-options {"url": "https://notes.puter.com/sheet"}
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
import PuterUserMenu, { canFrameAppBrowser } from './PuterUserMenu.js';
import PuterUserMenu from './PuterUserMenu.js';
const USER = { username: 'jelveh', uuid: '0b1c9a52-3f0e-4c7a-9a65-2f6d1e8b7c40' };
const USER = { username: 'jelveh', uuid: '0b1c9a52-3f0e-4c7a-9a65-2f6d1e8b7c40', email: '[email protected]' };
const OTHER = { username: 'sam', uuid: '7d2e4f10-8a3b-4c5d-9e6f-0a1b2c3d4e5f' };
const TEMP = { username: 'quiet_otter_42', uuid: '5c4b3a29-1807-4f6e-9d5c-4b3a29180716', is_temp: true };
const PICTURE = 'data:image/png;base64,iVBORw0KGgo=';
@@ -24,6 +24,9 @@ const fakePuter = ({ token = null, user = null, picture = null } = {}) => {
listeners.add(listener);
return () => listeners.delete(listener);
},
ui: {
showFeedbackDialog: vi.fn(() => new Promise(() => {})),
},
auth: {
getProfilePicture: vi.fn(async () => currentPicture),
signIn: vi.fn(() => new Promise(() => {})),
@@ -76,24 +79,6 @@ afterEach(() => {
vi.unstubAllGlobals();
});
describe('canFrameAppBrowser', () => {
it.each([
['https://notes.puter.com/a', 'https://puter.com', true],
['https://NOTES.puter.com./a', 'https://puter.com', true],
['http://notes.puter.localhost:4100/', 'http://puter.localhost:4100', true],
['https://puter.com/', 'https://puter.com', false],
['https://a.b.puter.com/', 'https://puter.com', false],
['https://notputer.com/', 'https://puter.com', false],
['https://notes.puter.com.evil.com/', 'https://puter.com', false],
['http://notes.puter.com/', 'https://puter.com', false],
['https://notes.puter.com:8443/', 'https://puter.com', false],
['https://notes.puter.site/', 'https://puter.com', false],
[undefined, 'https://puter.com', false],
])('%s under %s is %s', (page, gui, expected) => {
expect(canFrameAppBrowser(page, gui)).toBe(expected);
});
});
describe('<puter-user-menu>', () => {
it.each(['app', 'gui'])('renders nothing in the %s environment', async (env) => {
vi.stubGlobal('puter', { ...fakePuter({ token: 't', user: USER }), env });
@@ -182,7 +167,10 @@ describe('<puter-user-menu>', () => {
part(el, '.avatar').click();
expect(shown(el, '.menu')).toBe(true);
expect(part(el, '.avatar').getAttribute('aria-expanded')).toBe('true');
expect(part(el, '.menu-user').textContent).toBe('jelveh');
expect(part(el, '.menu-username').textContent).toBe('jelveh');
expect(part(el, '.menu-email').textContent).toBe(USER.email);
expect(shown(el, '.menu-email')).toBe(true);
expect(part(el, '.menu-avatar').textContent).toBe('J');
part(el, '.apps-button').click();
expect(shown(el, '.menu')).toBe(false);
@@ -191,6 +179,80 @@ describe('<puter-user-menu>', () => {
expect(frameOf(el).src).toBe(`https://puter.com/embed/apps#user=${USER.uuid}`);
});
it('links to the account, its usage and Puter, each in a new tab', async () => {
vi.stubGlobal('puter', fakePuter({ token: 'app-token', user: USER }));
const el = await mount();
part(el, '.avatar').click();
expect(el.shadowRoot.activeElement).toBe(part(el, '.manage-account'));
const links = [...el.shadowRoot.querySelectorAll('.menu a.menu-item')].map(link => ({
label: link.textContent,
href: link.href,
target: link.target,
rel: link.rel,
}));
expect(links).toEqual([
{ label: 'Manage account', href: 'https://puter.com/dashboard/#account', target: '_blank', rel: 'noopener' },
{ label: 'Usage', href: 'https://puter.com/dashboard/#usage', target: '_blank', rel: 'noopener' },
{ label: 'Open Puter', href: 'https://puter.com/', target: '_blank', rel: 'noopener' },
]);
part(el, '.usage').dispatchEvent(new MouseEvent('click', { cancelable: true }));
expect(shown(el, '.menu')).toBe(false);
});
it('hides the email line when the account has none to show', async () => {
vi.stubGlobal('puter', fakePuter({ token: 'app-token', user: OTHER }));
const el = await mount();
part(el, '.avatar').click();
expect(part(el, '.menu-username').textContent).toBe('sam');
expect(shown(el, '.menu-email')).toBe(false);
});
it('shows the profile picture in the menu header too', async () => {
vi.stubGlobal('puter', fakePuter({ token: 'app-token', user: USER, picture: PICTURE }));
const el = await mount();
expect(part(el, '.menu-avatar img').getAttribute('src')).toBe(PICTURE);
expect(part(el, '.menu-avatar').textContent).toBe('J');
});
it('opens the feedback dialog from the menu', async () => {
const sdk = fakePuter({ token: 'app-token', user: USER });
vi.stubGlobal('puter', sdk);
const el = await mount();
part(el, '.avatar').click();
part(el, '.feedback').click();
expect(sdk.ui.showFeedbackDialog).toHaveBeenCalledTimes(1);
expect(shown(el, '.menu')).toBe(false);
});
it('moves focus through the menu with the arrow, Home and End keys', async () => {
vi.stubGlobal('puter', fakePuter({ token: 'app-token', user: USER }));
const el = await mount();
part(el, '.avatar').click();
const press = (key) => {
const event = new KeyboardEvent('keydown', { key, bubbles: true, composed: true, cancelable: true });
el.shadowRoot.activeElement.dispatchEvent(event);
return event;
};
const focused = () => el.shadowRoot.activeElement.classList[1];
expect(focused()).toBe('manage-account');
expect(press('ArrowDown').defaultPrevented).toBe(true);
expect(focused()).toBe('usage');
press('End');
expect(focused()).toBe('sign-out');
press('ArrowDown');
expect(focused()).toBe('manage-account');
press('ArrowUp');
expect(focused()).toBe('sign-out');
press('ArrowUp');
expect(focused()).toBe('feedback');
press('Home');
expect(focused()).toBe('manage-account');
expect(press('a').defaultPrevented).toBe(false);
expect(shown(el, '.menu')).toBe(true);
});
it('hangs the panel under its button with the caret pointing at it', async () => {
vi.stubGlobal('puter', fakePuter({ token: 'app-token', user: USER }));
const el = await mount();
@@ -399,12 +461,25 @@ describe('<puter-user-menu>', () => {
it('uses label attributes, including ones changed later', async () => {
vi.stubGlobal('puter', fakePuter({ token: 'app-token', user: USER }));
const el = await mount({ 'apps-label': 'Applis', 'sign-out-label': 'Déconnexion', 'account-label': 'Compte' });
const el = await mount({
'apps-label': 'Applis',
'sign-out-label': 'Déconnexion',
'account-label': 'Compte',
'manage-account-label': 'Gérer le compte',
'usage-label': 'Utilisation',
'open-puter-label': 'Ouvrir Puter',
'feedback-label': 'Envoyer un avis',
});
expect(part(el, '.apps-button').getAttribute('aria-label')).toBe('Applis');
expect(part(el, '.apps-title').textContent).toBe('Applis');
expect(part(el, '.apps-panel').getAttribute('aria-label')).toBe('Applis');
expect(part(el, '.sign-out').textContent).toBe('Déconnexion');
expect(part(el, '.avatar').getAttribute('aria-label')).toBe('Compte: jelveh');
expect(part(el, '.manage-account').textContent).toBe('Gérer le compte');
expect(part(el, '.usage').textContent).toBe('Utilisation');
expect(part(el, '.open-puter').textContent).toBe('Ouvrir Puter');
expect(part(el, '.feedback').textContent).toBe('Envoyer un avis');
expect(part(el, '.manage-account .menu-external')).not.toBeNull();
el.setAttribute('close-label', 'Fermer');
el.setAttribute('sign-in-label', 'Connexion');
@@ -6,7 +6,7 @@ import WebSocket, { WebSocketServer } from 'ws';
import type { AddressInfo } from 'node:net';
import { loadNodePuter, closeNodePuter } from '../harness/nodeSdkLoader';
import type { EnvManifest, PuterSDK } from '../harness/types';
import type { Conversation } from '../../../types/modules/ai/convo';
import type { Conversation } from '../../../types/modules/ai/realtime';
import type { RealtimeServerEvent } from '../../../types/modules/ai/types';
import {
setupPuterTestEnv,
@@ -14,7 +14,7 @@ import {
} from '@heyputer/backend/testUtil.ts';
import type { IConfig } from '@heyputer/backend/types.ts';
describe('convo built SDK and browser media', () => {
describe('realtime built SDK and browser media', () => {
let env: PuterTestEnv;
let upstream: WebSocketServer;
let puter: PuterSDK;
@@ -220,7 +220,7 @@ describe('convo built SDK and browser media', () => {
it('resolves the original built SDK connect after an initial provider restart', async () => {
upstreamRefusals = 1;
const before = connections.length;
const conversation = await puter.ai.convo({ output: ['text'] });
const conversation = await puter.ai.realtime({ output: ['text'] });
try {
const ready = conversation.connect();
expect(conversation.connect()).toBe(ready);
@@ -234,7 +234,7 @@ describe('convo built SDK and browser media', () => {
});
it('prepares, configures, sends, interrupts and truncates using the built SDK', async () => {
const conversation = await puter.ai.convo({
const conversation = await puter.ai.realtime({
output: ['text'],
turnDetection: false,
});
@@ -283,13 +283,13 @@ describe('convo built SDK and browser media', () => {
new URL('../../../dist/puter.js', import.meta.url),
'utf8',
);
await page.route('**/__convo_test__', (route) =>
await page.route('**/__realtime_test__', (route) =>
route.fulfill({
contentType: 'text/html',
body: '<!doctype html><title>Conversation media test</title>',
}),
);
await page.goto(`${env.apiOrigin}/__convo_test__`);
await page.goto(`${env.apiOrigin}/__realtime_test__`);
await page.evaluate(
({ origin, apiOrigin }) => {
Object.assign(globalThis, {
@@ -315,7 +315,7 @@ describe('convo built SDK and browser media', () => {
};
let c;
try {
c = await sdk.ai.convo();
c = await sdk.ai.realtime();
} catch (error) {
throw new Error(`prepare: ${JSON.stringify(error)}`);
}
@@ -432,7 +432,7 @@ describe('convo built SDK and browser media', () => {
{ ...manifest, apiOrigin: proxyOrigin },
env.users.user.token,
);
const conversation = await dropping.ai.convo({ output: ['text'] });
const conversation = await dropping.ai.realtime({ output: ['text'] });
const log = watch(conversation);
try {
await conversation.connect();
@@ -481,7 +481,7 @@ describe('convo built SDK and browser media', () => {
{ ...manifest, apiOrigin: proxyOrigin },
env.users.user.token,
);
const conversation = await dropping.ai.convo({ output: ['text'] });
const conversation = await dropping.ai.realtime({ output: ['text'] });
const log = watch(conversation);
const auth = vi.spyOn(env.server.services.auth, 'authenticate');
const warn = vi.spyOn(console, 'warn');
@@ -524,7 +524,7 @@ describe('convo built SDK and browser media', () => {
{ ...manifest, apiOrigin: proxyOrigin },
env.users.user.token,
);
const conversation = await dropping.ai.convo({ output: ['text'] });
const conversation = await dropping.ai.realtime({ output: ['text'] });
const log = watch(conversation);
try {
await conversation.connect();
@@ -564,7 +564,7 @@ describe('convo built SDK and browser media', () => {
});
it('moves to a new session when a restart is announced, restoring context', async () => {
const conversation = await puter.ai.convo({ output: ['text'] });
const conversation = await puter.ai.realtime({ output: ['text'] });
const log = watch(conversation);
try {
await conversation.connect();
@@ -609,7 +609,7 @@ describe('convo built SDK and browser media', () => {
});
it('closes with invalid_event when Puter refuses a native event', async () => {
const conversation = await puter.ai.convo({ output: ['text'] });
const conversation = await puter.ai.realtime({ output: ['text'] });
const log = watch(conversation);
try {
await conversation.connect();
@@ -646,13 +646,13 @@ describe('convo built SDK and browser media', () => {
});
const page = await browser.newPage();
try {
await page.route('**/__convo_mic_test__', (route) =>
await page.route('**/__realtime_mic_test__', (route) =>
route.fulfill({
contentType: 'text/html',
body: '<!doctype html><title>Microphone reconnect test</title>',
}),
);
await page.goto(`${env.apiOrigin}/__convo_mic_test__`);
await page.goto(`${env.apiOrigin}/__realtime_mic_test__`);
// Through the proxy, so the test can drop the connection.
await page.evaluate(
({ origin, apiOrigin }) => {
@@ -695,7 +695,7 @@ describe('convo built SDK and browser media', () => {
tracks.push(...stream.getTracks());
return stream;
};
const c = await g.puter.ai.convo({ output: ['text'] });
const c = await g.puter.ai.realtime({ output: ['text'] });
const events: string[] = [];
c.on('*', (event) => events.push(event.type));
await c.connect();
@@ -800,7 +800,7 @@ describe('convo built SDK and browser media', () => {
{ ...manifest, apiOrigin: proxyOrigin },
env.users.user.token,
);
const conversation = await draining.ai.convo({ output: ['text'] });
const conversation = await draining.ai.realtime({ output: ['text'] });
const log = watch(conversation);
try {
await conversation.connect();
@@ -2,7 +2,7 @@ import WebSocket from 'ws';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { closeNodePuter, loadNodePuter } from '../harness/nodeSdkLoader';
import type { EnvManifest, PuterSDK } from '../harness/types';
import type { Conversation } from '../../../types/modules/ai/convo';
import type { Conversation } from '../../../types/modules/ai/realtime';
import type { RealtimeServerEvent } from '../../../types/modules/ai/types';
import { optionalEnv } from '@heyputer/backend/drivers/integrationTestUtil.ts';
import {
@@ -146,7 +146,7 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
'batchIncrementUsages',
);
const startedAt = performance.now();
const session = await puter.ai.convo({
const session = await puter.ai.realtime({
...(model ? { model } : {}),
output: ['text'],
instructions: 'Reply with exactly realtime-ok',
@@ -188,7 +188,7 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
.map((result) => result.value),
);
process.stdout.write(
`[live convo] model=${model ?? 'default'} ready=${Math.round(readyAt - startedAt)}ms firstText=${Math.round(firstDeltaAt! - readyAt)}ms reply=${JSON.stringify(text)} metering=recorded\n`,
`[live realtime] model=${model ?? 'default'} ready=${Math.round(readyAt - startedAt)}ms firstText=${Math.round(firstDeltaAt! - readyAt)}ms reply=${JSON.stringify(text)} metering=recorded\n`,
);
} finally {
session.close();
@@ -199,7 +199,7 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
45_000,
);
it('restores tool and text context on a new provider session after a restart notice', async () => {
const session = await puter.ai.convo({
const session = await puter.ai.realtime({
output: ['text'],
instructions: 'Answer in one short sentence.',
});
@@ -272,7 +272,7 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
expect(text.toLowerCase()).toContain('pineapple');
expect(errors).toEqual([]);
process.stdout.write(
`[live convo] context restored after reconnect, reply=${JSON.stringify(text)}\n`,
`[live realtime] context restored after reconnect, reply=${JSON.stringify(text)}\n`,
);
} finally {
session.close();
@@ -281,7 +281,7 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
}, 60_000);
it('receives live PCM audio and interrupts the current response', async () => {
const c = await puter.ai.convo({
const c = await puter.ai.realtime({
output: ['audio'],
instructions: 'Count slowly from one to fifty.',
});
@@ -301,7 +301,7 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
const response = await done;
expect(response.response).toMatchObject({ status: 'cancelled' });
process.stdout.write(
'[live convo] PCM audio received, response canceled, playback history truncated\n',
'[live realtime] PCM audio received, response canceled, playback history truncated\n',
);
} finally {
c.close();
+11 -11
View File
@@ -1743,10 +1743,10 @@ export default suite('ai', {
t.assert.equal(error.message, 'Input size cannot be larger than 25 MB');
},
'convo streams a live text turn': {
'realtime streams a live text turn': {
requires: ['ai.openai'],
fn: async (t) => {
const conversation = await t.puter.ai.convo({ output: ['text'], instructions: 'Reply with exactly realtime-ok.' });
const conversation = await t.puter.ai.realtime({ output: ['text'], instructions: 'Reply with exactly realtime-ok.' });
let text = '';
conversation.on('text', ({ delta }) => { text += delta; });
let timer: ReturnType<typeof setTimeout> | undefined;
@@ -1767,8 +1767,8 @@ export default suite('ai', {
},
},
'convo prepares and closes without opening a provider session': async (t) => {
const conversation = await t.puter.ai.convo({ output: ['text'] });
'realtime prepares and closes without opening a provider session': async (t) => {
const conversation = await t.puter.ai.realtime({ output: ['text'] });
t.assert.equal(conversation.state, 'prepared');
t.assert.equal(conversation.model, 'gpt-realtime-2.1-mini');
let closed = 0;
@@ -1779,9 +1779,9 @@ export default suite('ai', {
t.assert.equal(closed, 1);
},
'convo rejects invalid options and sending before connect': async (t) => {
await t.assert.rejects(() => t.puter.ai.convo({ model: 'not-a-realtime-model' }));
const conversation = await t.puter.ai.convo({ output: ['text'] });
'realtime rejects invalid options and sending before connect': async (t) => {
await t.assert.rejects(() => t.puter.ai.realtime({ model: 'not-a-realtime-model' }));
const conversation = await t.puter.ai.realtime({ output: ['text'] });
try {
await t.assert.rejects(async () => conversation.sendText('Hello'));
} finally {
@@ -1789,8 +1789,8 @@ export default suite('ai', {
}
},
'convo rejects interruption readiness through its Promise': async (t) => {
const conversation = await t.puter.ai.convo({ output: ['text'] });
'realtime rejects interruption readiness through its Promise': async (t) => {
const conversation = await t.puter.ai.realtime({ output: ['text'] });
try {
const prepared = conversation.interrupt();
t.assert.equal(typeof prepared.then, 'function');
@@ -1804,9 +1804,9 @@ export default suite('ai', {
}
},
'convo is the realtime SDK entry point': async (t) => {
'realtime is the only realtime SDK entry point': async (t) => {
t.assert.equal('assistant' in t.puter.ai, false);
t.assert.equal('conversation' in t.puter.ai, false);
t.assert.equal(typeof t.puter.ai.convo, 'function');
t.assert.equal(typeof t.puter.ai.realtime, 'function');
},
});