Compare commits

...
99 Commits
Author SHA1 Message Date
Daniel Salazar 0405de30bd update taskbar apps (#3286) 2026-06-22 14:26:35 -07:00
Nariman Jelveh 40c2799b06 Add 'builder' to recommended apps list
Include 'builder' in the RECOMMENDED_APP_NAMES array in src/backend/services/apps/RecommendedAppsService.ts so the RecommendedAppsService treats the builder app as a recommended application and surfaces it accordingly.
2026-06-22 14:22:27 -07:00
Daniel Salazar 4bcdeefe68 chore: bump up deps (#3285) 2026-06-22 13:58:45 -07:00
Daniel Salazar b6ba6dd90e fix: [PUT-1199] more prelude signals (#3282) 2026-06-22 08:16:59 -07:00
Daniel Salazar 954f8b7b8d fix: puter self host guides + default bug (#3281)
* fix: oss dev center html_encode hardening

* fix: puter self host guides + default bug
2026-06-21 22:32:43 -07:00
Neal Shah 24d700673c Compaction support for OpenAI and Anthropic (#3279)
* Alpha: compaction support for OpenAI and Anthropic

* update lock

* fix billing for anthropic compactions

* Fix max_tokens bug in together provider

* Fix responses compaction
2026-06-21 21:57:56 -04:00
c6d64029e0 feat: meter Gemini thinking tokens and grounding requests (#3178)
* feat: meter Gemini thinking tokens and grounding requests

- Thinking tokens: Extracted from standard completion tokens to ensure they are billed accurately at the correct model-specific rate.
- Grounding requests: Added flat-fee metering for Google Search by tracking grounding_metadata across both streaming and non-streaming responses.
- Pricing updates: Corrected stale rates for Gemini 2.5 Flash output, cached tokens, thinking tokens, and grounding requests.

* fix: correct Gemini 2.x metering rates and harden grounding capture

Pricing corrections (verified against ai.google.dev/gemini-api/docs/pricing):
- gemini-2.5-flash output is $2.50/M, not $1.00/M: restore
  completion_tokens to 250 and bill thinking_tokens at the same output
  rate (250). The previous 100 under-billed output ~60%, and this is the
  provider's default model.
- gemini-2.5-flash cache read is $0.03/M: restore cached_tokens to 3
  (the 7.5 value over-billed).
- Grounding with Google Search is $35 / 1,000 requests for Gemini 2.x
  models and $14 / 1,000 for 3.x. Set grounding_requests to 3_500_000
  for gemini-2.0-flash, gemini-2.5-flash, gemini-2.5-flash-lite and
  gemini-2.5-pro; 3.x models keep 1_400_000.

Streaming robustness:
- In create_chat_stream_handler, don't let a later extra_content chunk
  without grounding_metadata overwrite an earlier one that carried it,
  so grounding requests are still metered.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

---------

Co-authored-by: Daniel Salazar <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-20 15:08:38 -07:00
jelveh 40ff866598 Add 'butler' to recommended apps
Include 'butler' in the RECOMMENDED_APP_NAMES array in RecommendedAppsService so it appears in the recommended apps list. Updates src/backend/services/apps/RecommendedAppsService.ts.
2026-06-19 15:50:57 -07:00
Nariman Jelveh 36888e35d3 Add phone verification UI improvements and helpers (#3274)
Revamps the phone verification dialog and adds reusable phone helper modules.

- Heavily refactors UIWindowPhoneVerificationRequired.js to add a searchable inline country picker, national-number live formatting, country autodetect for "+" input, WebOTP auto-fill support, resend cooldown timer, accessibility improvements, proper teardown/cleanup, and last-used country persistence.
- Introduces helpers/src/helpers/country_codes.js (get_country_list, flag emoji) which builds a localized list of dialable countries from libphonenumber-js and Intl.DisplayNames.
- Introduces helpers/src/helpers/phone.js with thin wrappers around libphonenumber-js: format_phone_as_you_type, inspect_phone, phone_example_for, detect_country_from_input.
- Adds English i18n strings for the new UI in src/gui/src/i18n/translations/en.js.

These changes keep submission canonicalization using libphonenumber-js (E.164), improve UX for selecting/formatting numbers, and make the country list and phone utilities reusable across the app.
2026-06-19 10:28:30 -07:00
Reynaldi Chernando 30432b931a Add PUT and DELETE to preflight allowed methods (#3278) 2026-06-19 12:35:29 -04:00
Reynaldi Chernando e2174a14cf Update router cors preflight docs (#3277) 2026-06-19 20:50:41 +07:00
Daniel Salazar d3cb89d8c9 fix: import in extensions (#3276) 2026-06-18 21:30:00 -07:00
Reynaldi Chernando c8625b3396 Release puterjs npm (#3275)
* Release puterjs npm

* add package lock
2026-06-18 18:26:39 -07:00
Reynaldi Chernando b576f01d12 Sync Puter.js types and docs (#3194)
* AI chat types

* ai voice types

* update tts docs and jsdoc

* stt types and docs

* txt2img docs and types

* txt2vid and img2txt

* apps

* signin

* perms

* fs

* kv

* hosting

* workers

* net

* ui

* jsdoc
2026-06-18 09:34:13 -07:00
Vyasdev a87cafee3c Revise bug report and feature request instructions (#3220)
Updated the wording for reporting bugs and feature requests in Japanese.
2026-06-18 09:27:55 -07:00
Reynaldi Chernando 1c51fde163 Add minimax m3 direct integration + cleanup model list (#3263) 2026-06-18 10:12:44 -04:00
Daniel Salazar b05699db77 fix: AGENTS.md cleanup (#3273) 2026-06-17 17:07:20 -07:00
Daniel Salazar 163e48ab27 feat: viz over user verification types (#3271) 2026-06-17 14:21:55 -07:00
Reynaldi Chernando 0b41d992a4 Add glm 5.2 direct integration (#3269) 2026-06-17 09:40:10 -04:00
Miika Kuisma c2ddfd6867 Merge pull request #3270 from HeyPuter/fix-file-dialog
puter.js - fix issue where content of file dialog wasn't visible
2026-06-17 13:08:43 +03:00
Miika Kuisma 81b9dc1f06 fix issue where content of file dialog wasn't visible 2026-06-17 13:07:13 +03:00
Daniel Salazar 5506ddb2c2 feat: fp cleanup and better devex (#3268) 2026-06-16 22:48:54 -07:00
Neal Shah c74469b01a prelude RCS (#3267) 2026-06-15 19:08:30 -04:00
Daniel Salazar b706693f82 wip: hardening (#3266) 2026-06-15 16:07:44 -07:00
Daniel SalazarandNeal Shah cd800148d5 feat: optional additional verification gates (#3262)
* phone number verificiation initial

* finish phone verification

* feat: add card gate for signups

* chore: npm

* fix: modal order and wording

* fix:wording

---------

Co-authored-by: Neal Shah <[email protected]>
2026-06-15 09:16:00 -07:00
Ricardo Sawir 900b0d7aa6 fix: correct puter container healthcheck URL (#3255)
The healthcheck was using 'puter.localhost:4100/test' which fails for
two reasons:
- 'puter.localhost' does not resolve inside the container; the service
  listens on all interfaces at port 4100, so 'localhost' works
- '/test' endpoint does not exist; use '/' which serves the shell

This caused the container to be perpetually marked as unhealthy.

Fixes #3191
2026-06-13 23:17:44 -07:00
Daniel Salazar 7e21c1f888 fix: fs metadata sanitation (#3257) 2026-06-12 22:56:06 -07:00
Daniel Salazar 00d6f9c230 fix: disallow unverified outlook emails (#3256) 2026-06-12 18:52:24 -07:00
jelveh 4c03080efa Update mcp.md 2026-06-12 16:55:37 +02:00
jelveh 432a11d289 Add Cursor MCP docs and icons 2026-06-12 16:37:53 +02:00
jelveh e9e233404b Add icons for Claude, Codex and OpenCode 2026-06-12 13:20:15 +02:00
jelveh 8101d42604 Capitalize OpenCode in MCP docs 2026-06-12 13:16:41 +02:00
jelveh 88cedd43bc Add opencode MCP example to docs 2026-06-12 13:11:24 +02:00
Daniel Salazar d3a2934d9b feat: fingerprinting for signup checks (#3254) 2026-06-12 00:59:21 -07:00
Daniel Salazar 13a36b6199 fix: secure http calls (#3252) 2026-06-11 13:43:58 -07:00
Daniel Salazar a155464d46 fix: more data for puter site serve event (#3251) 2026-06-10 23:27:13 -07:00
Daniel Salazar 40d1c998bb feat: pass args to all events (#3248)
* feat: pass args to all events

* fix: alias app joining

* fix: actor in event
2026-06-10 14:13:04 -07:00
Daniel Salazar b4837d49b6 perf: some perf gains from fable (#3249) 2026-06-10 13:26:35 -07:00
ProgrammerIn-wonderland e23042b42e add preperation for reputation score persistence (#3250) 2026-06-10 16:09:42 -04:00
Daniel Salazar 240a733285 sec: misc fable hardening (#3244)
* sec: misc fable hardening

* more fixes

* more fixes

* fix: cors issue
2026-06-10 11:19:41 -07:00
Reynaldi Chernando bb202be12c Minor clarification local webserver in deployment (#3247) 2026-06-10 23:04:42 +07:00
Reynaldi Chernando 8c4cb601c9 Add cli docs (#3246) 2026-06-10 20:53:53 +07:00
Reynaldi Chernando 64a7c7b7c4 Add mcp docs (#3245) 2026-06-10 20:31:31 +07:00
wwb1942 2dd2887ff7 ci: add docs build workflow (#3235)
* ci: add docs build workflow

Add CI workflow that runs npm ci and npm run build in src/docs

whenever files under src/docs change in a pull request.

Closes #3225

* ci: also trigger docs build when workflow file changes

* ci: address docs build workflow review
2026-06-10 10:53:40 +07:00
ProgrammerIn-wonderland 4bbe8a2789 add protections for PUT-1094 (#3243) 2026-06-09 22:29:57 -04:00
Dilan Melvin T 3e9ceb673b fix: coerce Mistral image_url parts from object to string (#3177)
* fix: coerce Mistral image_url parts from object to string

Mistral's API expects image_url content parts to be a plain string URL,
not the OpenAI-style { url: string } object that process_input_messages
produces. Without this coercion, sending an image to any Mistral model
that supports vision (mistral-small, mistral-medium, mistral-large,
ministral-*) results in a request error from the Mistral SDK.

Add #coerceImageUrls() as a private method on MistralAIProvider that
maps { type: 'image_url', image_url: { url } } -> { type: 'image_url',
image_url: url } for every content part in every message. Messages with
plain string content are left untouched, as are parts whose image_url is
already a string.

Add four unit tests covering: object-to-string coercion, already-flat
strings, plain string message content, and mixed text+image content.

* style: use plain ASCII dashes in comment section divider
2026-06-09 20:12:44 -04:00
Daniel Salazar 9c4d1ef535 fix: misc bugs + new middleware for before/after routes (#3242) 2026-06-09 17:05:56 -07:00
ProgrammerIn-wonderland bf57518262 MCP: assert worker_exec ends with puter.work (#3241) 2026-06-09 17:43:34 -04:00
Reynaldi Chernando d48fb11de6 Add claude fable 5 (#3238) 2026-06-09 14:05:31 -04:00
ProgrammerIn-wonderland 7ed87f760b make max tokens less lenient (#3240) 2026-06-09 14:03:13 -04:00
Reynaldi Chernando ee37be16f0 Add cli deployment docs (#3239) 2026-06-10 00:40:19 +07:00
Reynaldi Chernando a8cfaeefbd Add subdomain and workers github actions deployment (#3226)
* Add subdomain and workers github actions deployment

* minor adjust
2026-06-09 23:35:34 +07:00
Badri467 58cb8ee048 skip github actions backend for docs only changes (#3236) 2026-06-09 23:28:48 +07:00
Daniel Salazar a72fbd1d47 fix: inform acl of full access flag (#3237) 2026-06-09 08:04:36 -07:00
Daniel Salazar f6408db74e fix: access tokens with full accesses allowed through with new middleware gate (#3234) 2026-06-08 18:47:07 -07:00
ProgrammerIn-wonderland df20993d8b set azure openai as the default (#3233) 2026-06-08 20:33:41 -04:00
ProgrammerIn-wonderland 708fe1bc4e azure ai provider (#3232) 2026-06-08 18:31:49 -04:00
Daniel Salazar e782fa832e fix: kv costs (#3231) 2026-06-08 13:32:36 -07:00
ProgrammerIn-wonderland c947c2c989 fix mcp server cloudfront cache bug (#3230) 2026-06-08 15:42:49 -04:00
Daniel Salazar 3de4613b7d Revert "feat (PUT-1091): have gui tokens labeled and be different than reques…" (#3229)
This reverts commit 1335b46404.
2026-06-08 11:15:24 -07:00
Daniel Salazar def7290c48 feat: emit user.deleted event on user deletion (#3228) 2026-06-08 10:30:24 -07:00
Daniel Salazar 1335b46404 feat (PUT-1091): have gui tokens labeled and be different than requestor ones (#3222)
* feat (PUT-1091): have gui tokens labeled and be different than creator ones

* fix: errors with auth token
2026-06-08 09:03:02 -07:00
Reynaldi Chernando 0e15ac98a9 Update cli readme and set version (#3227) 2026-06-08 22:23:32 +07:00
Reynaldi ChernandoandCopilot Autofix powered by AI f127b2509d Add docs fixes for deployment and workers pages (#3223)
* Add docs fixes for deployment and workers pages

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

* handle router edge case

* explicit info about wildcard requiring name

---------

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
2026-06-08 17:43:00 +07:00
Sourabh Sharma 15134c99bf Fix mkdir dedupeName for existing directories (#3215) 2026-06-07 12:49:00 -07:00
Daniel Salazar 8262e3193e perf: don't batch get app stats (#3219) 2026-06-06 18:28:59 -07:00
Reynaldi Chernando 98a21c4ac0 More workers docs improvement (#3218)
* More workers docs improvement

* Add mention of workers alongside mention of user scope
2026-06-06 01:27:24 +07:00
ProgrammerIn-wonderland c52eb1b903 Ns/bug fixes 060426 (#3214)
* Harden backend auth

* Puter.js auth hardening
2026-06-05 13:55:23 -04:00
mohit_parmar_007 d6a71e50bb fix: remove unnecessary cache console.log statements (#3216) 2026-06-05 10:01:59 -07:00
Reynaldi Chernando a4b6ee0dd2 [PUT-1090] Improve worker docs (#3217)
* Improve worker docs

* refine
2026-06-05 21:54:56 +07:00
Daniel Salazar bb61103080 feat: add apps api into mcp (#3213) 2026-06-04 17:19:53 -07:00
Daniel Salazar 9bbc76b2dd drop metering alerts to low severity (#3212) 2026-06-04 14:01:44 -07:00
Daniel Salazar 918aba9c80 fix: reduce alarms further (#3211)
* feat; update bug-bounty

* fix: reduce alarms further
2026-06-04 10:43:50 -07:00
Reynaldi ChernandoandCopilot Autofix powered by AI 3ceba9e3b3 Minor fix deployment docs (#3210)
* Minor fix deployment docs

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

---------

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
2026-06-04 23:48:25 +07:00
Daniel Salazar a480679a2d vis: make metering service alarm only go off on actual abuse (#3208) 2026-06-03 22:45:18 -07:00
Daniel Salazar e99ae40e93 chore: bump puter-js (#3207) 2026-06-03 18:59:07 -07:00
Daniel Salazar 3cd50fbaf3 move the token migration to root (#3206) 2026-06-03 18:54:22 -07:00
ProgrammerIn-wonderland 60dd1453d4 Fix issues (#3205) 2026-06-03 21:19:33 -04:00
ProgrammerIn-wonderland 07781cd681 allow overriding origin from workers.dev on mcp server (#3204) 2026-06-03 17:48:37 -04:00
ProgrammerIn-wonderland 2387d8dcc9 MCP whoami (#3203) 2026-06-03 17:13:13 -04:00
Reynaldi Chernando 723608f4db Add new puter cli (#3199)
* Add new cli

* update package lock

* fix readme and license
2026-06-04 01:51:31 +07:00
Reynaldi Chernando bd1dcde7b0 Add info for puterjs deployments (#3202)
* Add info for puterjs deployments

* update external links

* hide github actions method
2026-06-04 01:25:57 +07:00
ProgrammerIn-wonderland 5b8e41e66d MCP Server (#3197)
* beginnings of mcp

* oauth support

* add workers and better prompt engineering
2026-06-03 12:23:06 -04:00
Nariman Jelveh acc4796b99 Allow Cache-Control and Pragma headers (#3198) 2026-06-03 14:30:33 +02:00
jelveh 106e85d6ca Force revalidation on FileSystem reads 2026-06-03 14:15:22 +02:00
ProgrammerIn-wonderland 0ffc32f19e add whitelist for signed batch uploads (#3196) 2026-06-02 21:30:34 -04:00
Daniel Salazar d897cc5dd0 refactor: move subscriptions to webhook system (#3193)
* fix: dont sign write urls if they don't have write perm

* wip: new subscriptions

* refactor: move subscriptions to
2026-06-02 17:58:56 -07:00
Daniel Salazar 6d2f277ce2 fix: some error handling (#3190) 2026-05-31 18:58:27 -07:00
ProgrammerIn-wonderland 86f80b29c0 tighten RAO policy (#3188) 2026-05-29 20:52:21 -04:00
Daniel SalazarandProgrammerIn-wonderland 39243735e8 feat (put-1049): worker types (#3185)
* feat (put-1049): worket types

* Implement changes to worker types

---------

Co-authored-by: ProgrammerIn-wonderland <[email protected]>
2026-05-29 20:51:57 -04:00
ProgrammerIn-wonderland b3ff9d74b7 add browserjs to taskbar and RecommendedAppService (#3189) 2026-05-29 20:32:53 -04:00
ProgrammerIn-wonderland 7b7604440f Add serverless only filtering for togetherai chat models (#3187) 2026-05-29 14:33:55 -04:00
ProgrammerIn-wonderland 382d2b3c14 add minimax provider (#3171)
* add minimax provider
2026-05-29 13:36:10 -04:00
Anurag Pappula eda5379c17 fix(docs/search): handle search index loading and failure states (#3184)
* fix(docs/search): show loading state and replay query when index isn't ready

Previously performSearch rendered "No results found" while the
MiniSearch index was still loading, which is misleading on slow
networks. Track the last query as pendingQuery and replay it from
fetchSearchIndex once the index is ready; render a distinct
"Failed to load search index" message on fetch error so the
loading message doesn't persist indefinitely.

Fixes #3180.

* fix(docs/search): close stale-replay race and consolidate status messages

Drop pendingQuery synchronously in the input handler when the query
falls below the minimum length, so a fetchSearchIndex that resolves
within the 300ms debounce window after the user clears the field can
no longer replay the cleared query into an empty input.

Move the idle, loading, and failed copy into a single
renderSearchStatus(kind) helper so the four .search-no-results
renderings can only drift in one place.
2026-05-29 16:00:59 +07:00
Ron HernausandDaniel Salazar d2fee51844 Support PostgreSQL database backend (#3167)
* feat(database): add postgres database client

Adds the PostgreSQL database client, native bootstrap migration, SQL preparation helpers, and database config/factory wiring.\n\nRefs #3165.

* feat(database): make backend queries postgres-aware

Updates runtime SQL call sites for database-specific booleans, identifiers, insert-ignore, upserts, JSON extraction, intervals, and Postgres insert ids.\n\nRefs #3165.

* test(database): cover postgres client behavior

Adds unit coverage for SQL preparation, factory selection, write-result mapping, and transaction rollback/commit ordering, plus an env-gated PostgreSQL integration flow.\n\nRefs #3165.

* docs(self-hosting): document postgres database setup

Adds PostgreSQL configuration examples and migration path guidance for self-hosted deployments.\n\nRefs #3165.

* fix: harden postgres oidc tests

* fix(postgres): normalize query results and SQL prep

* fix(user): preserve normalized cache booleans

* test(postgres): run integration coverage with pgmock

* tests: add way to run all tests with postgres though slow

Also adding note that postgres is not in active use so might not work out the box

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-05-28 14:12:17 -07:00
Daniel Salazar 8ac4bad196 fix: readUrl perm check (#3183) 2026-05-28 11:44:16 -07:00
Reynaldi Chernando f140d7221c Add opus 4.8 (#3182)
* Add opus 4.8

* update alias

* handling opus 48 omit temperature
2026-05-28 13:27:46 -04:00
Daniel Salazar 631f0bda1b chore:deploy latest puter.js (#3181) 2026-05-28 09:58:33 -07:00
Reynaldi Chernando 1121314582 Docs search improvement (#3179)
* Init search improvement

* Add fuzzy implementation

* update package lock

* package lock matched with runner 24.16.0
2026-05-28 23:11:34 +07:00
324 changed files with 32210 additions and 6593 deletions

No files matched your search

+2
View File
@@ -3,6 +3,8 @@ name: Backend Tests
on:
pull_request:
types: [opened, synchronize, reopened]
paths-ignore:
- 'src/docs/**'
permissions:
contents: read
+27
View File
@@ -0,0 +1,27 @@
name: Docs Build
on:
pull_request:
paths:
- 'src/docs/**'
jobs:
build-docs:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '24'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Build docs
run: npm run build
working-directory: src/docs
+2
View File
@@ -70,3 +70,5 @@ coverage/
undefined
servers.json
config.*.json
volatile/
+2 -2
View File
@@ -64,9 +64,9 @@ Keep comments light. Prefer self-documenting code — clear names, small functio
- The _why_ is non-obvious (a hidden constraint, a workaround, a subtle invariant).
- A non-trivial usage detail would otherwise trip the next reader.
Don't restate what the code already says. Don't write comments that reference the current task or PR — those rot.
Don't restate what the code already says. Don't write comments that reference the current task or PR or version of something — those rot.
**No ticket references in code or test comments.** Don't write `PUT-1234`, `AUTH-3`, `GUI-2`, `// fix for FOO-99`, `// per ROLLOUT-1`, or section headers like `// -- PUT-1020: ... --`. The ticket exists in git history and the issue tracker; the code should describe the *why* in domain terms ("logout cascade transparently invalidates...") not project-management terms ("the cascade from AUTH-3..."). Same rule for test-block names and section dividers — describe what's being tested, not which ticket asked for it.
**No ticket references in code or test comments.** Don't write `PUT-1234`, `AUTH-3`, `GUI-2`, `// fix for FOO-99`, `// per ROLLOUT-1`, or section headers like `// -- PUT-1020: ... --`. The ticket exists in git history and the issue tracker; the code should describe the _why_ in domain terms ("logout cascade transparently invalidates...") not project-management terms ("the cascade from AUTH-3..."). Same rule for test-block names and section dividers — describe what's being tested, not which ticket asked for it.
**Use plain ASCII `-` in comment section dividers, not box-drawing characters.** Write `// -- Section name --`, not `// ── Section name ──`. The `─` (U+2500) character renders inconsistently in some editors and diff tools and adds nothing over `-`.
+34 -1
View File
@@ -17,18 +17,51 @@ Out-of-scope:
* Denial of Service (DoS), spam, or volumetric attacks.
* Physical security issues.
## Known Non-Issues (Please Check Before Submitting)
The following have already been reviewed and determined **not to be vulnerabilities**. Reports that only re-describe one of these are **not eligible for a reward** and will be closed as non-issues — even if they include new code references. Please review this list before submitting:
* **XSS / CORS / token issues scoped to `api.puter.com`.** The primary user session cookie lives on `puter.com`, not on the API origin, so reflected/stored XSS, CORS, or token handling on `api.puter.com` is generally out of scope. Two exceptions we *do* evaluate on their merits — please report these: (a) attacker-controlled content served **inline** (e.g. an HTML `Content-Type`) from API file/response endpoints, and (b) anything that abuses the app-scoped `puter_token_v2` companion cookie set on the API host.
* **SSRF via `secureFetch`.** Production routes outbound requests through an isolated proxy that has no access to internal/SSRF-sensitive resources.
* **Attacks that depend on guessing an `appInstanceID` or app UID.** These are random 128-bit secret values and are not considered guessable.
* **Apps invoking drivers, creating workers, or using KV.** Applications are intended to do this; worker permissions are scoped to the owning app. This is by design.
* **App metadata or app user-count "leaks".** This information is currently public by design.
* **General "token in a URL" / token-lifetime designs** — signed directory URLs exposing children, a write signature implying read, or app tokens outliving a web session. These are current intended behaviors.
* **Missing PKCE, unverified `id_token` signatures, or other OIDC hardening.** Provider tokens are obtained through a server-to-server authorization-code exchange with the provider's token endpoint over TLS, so the resulting `id_token` / userinfo claims are trusted from that channel rather than from local JWKS signature verification — the callback never accepts a caller-supplied `id_token`. Adding local signature / `aud` / `iss` / `exp` checks is welcome defense-in-depth (please open a GitHub issue/PR), but their absence is not an account-takeover vector on its own.
* **JWT "algorithm confusion" / unpinned `algorithms` in `jwt.verify`.** Puter's session tokens are HMAC-signed (HS256) with a server-side secret, and there is no asymmetric public key anywhere in the verification path, so `alg`-substitution attacks do not apply. Explicitly pinning `algorithms` is a fine hardening PR, but it is not a vulnerability.
* **Static-source "SQL injection" in internal pagination.** Findings such as `LIMIT ${limit}` in list/notification queries: the limit is numerically coerced and clamped before it reaches the query, so it is not reachable with attacker-controlled string input. Hardening PRs to the internal stores are welcome, but these are not exploitable as reported.
* **Deprecated `saveTo*` GUI app messages.** The legacy `saveToDesktop` / `saveToDocuments` / etc. app-IPC handlers can create — never overwrite — new files in standard user folders. The behavior is non-destructive, path-traversal-safe, and deprecated (slated for removal); it is not treated as a sandbox escape.
* **Best-practice suggestions** such as login/registration username enumeration (kept intentionally for UX) or unauthenticated unsubscribe links (industry norm).
* **Rate-limiting suggestions for TURN credential issuance** (intentional; not billed per tunnel).
If you believe you have a **genuinely new** exploit chain that defeats one of these rationales (for example, demonstrating a sensitive credential that really is reachable on `api.puter.com`), say so explicitly and show why the reasoning above does not apply.
## Rules of Engagement
To participate, you must:
1. **Report responsibly**: Provide detailed steps to reproduce the issue, including proof-of-concept code or screenshots where applicable.
1. **Report responsibly**: Provide detailed steps to reproduce the issue, including proof-of-concept code, screenshots, or a screen recording (see *Proof of Reproduction* below).
2. **Do no harm**: Do not exfiltrate, modify, or delete data. Only access your own account or test data.
3. **Respect availability**: Do not perform denial-of-service attacks or automated scans that degrade service.
4. **Follow disclosure policy**: Do not publicly disclose vulnerabilities until we have confirmed and patched the issue.
5. **Act in good faith**: Make every effort to avoid privacy violations, destruction of data, and interruption or degradation of services.
6. **Check the Known Non-Issues list**: Reports matching an item in the "Known Non-Issues" section above are not eligible and will be closed.
Reports that do not meet these guidelines may not be eligible for a reward.
## Proof of Reproduction
Reports must demonstrate a **working, reproducible exploit with real impact** — not a theoretical or static-source-review finding. Please include:
* Exact steps to reproduce, the relevant request/response or code path, and the commit or version you tested.
* The **observed** result versus the **expected** result.
* For client-side, UI, or authentication-flow bugs: a short screen recording (≤ 2 minutes) showing the exploit working end-to-end on a real Puter instance.
* For server-side bugs: a runnable proof-of-concept.
Reports based solely on reading the source ("source review only, not tested") or unverified AI/LLM-generated reports are the **lowest triage priority and are generally not eligible**. If you used an AI tool to help find an issue, you must personally verify that it actually reproduces before submitting.
Please submit **one issue per report**. Bundled "audit packs" of many speculative findings will be declined; send each confirmed issue separately.
## Reporting Process
To report a vulnerability, email us at: **[[email protected]](mailto:[email protected])**.
+26 -4
View File
@@ -159,17 +159,30 @@
// ── Database ────────────────────────────────────────────────────────
"database": {
// `sqlite` for single-node / dev; `mysql` for self-host with MariaDB.
// `sqlite` for single-node/dev, `mysql` for MariaDB/MySQL,
// or `postgres` for PostgreSQL.
//
// NOTE: `postgres` support is a community contribution and is not
// exercised by Puter.com production. It boots, passes its own
// integration tests against pgmock, and runs the common user/app/
// fsentry/session/permission/OIDC flows — but less-traveled code
// paths may surface MySQL/SQLite-isms that haven't been ported yet.
// Expect rough edges and please file issues if you hit one. For
// production self-hosting today, `mysql` (MariaDB) and `sqlite` are
// the supported defaults.
"engine": "sqlite",
// sqlite — file path on disk
"path": "volatile/runtime/puter-database.sqlite",
"targetVersion": 0,
// mysql — connection details
// mysql/postgres — connection details. PostgreSQL defaults to port
// 5432 when `engine` is `postgres`; MySQL/MariaDB normally use 3306.
"host": "",
"port": 3306,
"user": "",
"password": "",
"database": "",
// postgres may also use a URL instead of host/user/password fields:
// "connectionString": "postgres://puter:secret@localhost:5432/puter",
// Optional read-replica pool. Reads route here when populated.
"replica": {
"host": "",
@@ -178,9 +191,10 @@
"password": "",
"database": ""
}
// mysql self-host bootstrap: set `migrationPaths` to apply the bundled
// schema on first boot. Idempotent — safe to leave on.
// mysql/postgres self-host bootstrap: set `migrationPaths` to apply the
// bundled schema on first boot. Idempotent — safe to leave on.
// "migrationPaths": ["./src/backend/clients/database/migrations/mysql"]
// "migrationPaths": ["./src/backend/clients/database/migrations/postgres"]
},
// ── DynamoDB (KV store) ─────────────────────────────────────────────
@@ -246,12 +260,20 @@
// ─ Chat / completion ─
"claude": { "apiKey": "" },
"openai-completion": { "apiKey": "" },
"azure-openai": {
"apiKey": "",
"apiURL": ""
},
"gemini": { "apiKey": "" },
"groq": { "apiKey": "" },
"deepseek": { "apiKey": "" },
"mistral": { "apiKey": "" },
"xai": { "apiKey": "" },
"moonshot": { "apiKey": "" },
"minimax": {
"apiKey": "",
"apiBaseUrl": "https://api.minimax.io/v1"
},
"openrouter": {
"apiKey": "",
"apiBaseUrl": "https://openrouter.ai/api/v1"
+1 -1
View File
@@ -104,7 +104,7 @@ Puterは[**puter.com**](https://puter.com)でホストサービスとして利
メンテナーやコミュニティと以下のチャンネルを通じてつながりましょう:
- バグ報告や機能リクエストがありますか? [issueを開く](https://github.com/HeyPuter/puter/issues/new/choose) してください。
- バグ報告や機能リクエストは [新規Issue](https://github.com/HeyPuter/puter/issues/new/choose) まで
- Discord: [discord.com/invite/PQcx7Teh8u](https://discord.com/invite/PQcx7Teh8u)
- X (Twitter): [x.com/HeyPuter](https://x.com/HeyPuter)
- Reddit: [reddit.com/r/puter/](https://www.reddit.com/r/puter/)
+47 -1
View File
@@ -136,7 +136,7 @@ Why these knobs:
- `jwt_secret` + `jwt_secret_v2` — Puter signs new auth tokens with `jwt_secret_v2` (v2 token format, `kid: 'v2'` JWT header). `jwt_secret` is verify-only and lets tokens minted before this version (cookies still in users' browsers, stored API tokens) keep working. `allow_v1_tokens: true` keeps that fallback enabled. Fresh installs can leave it as-is; future versions will flip the flag to `false` and retire `jwt_secret` entirely once legacy tokens have drained.
- `env: "prod"` — the bundled `config.default.json` ships with `env: "dev"` (matches the source-tree `npm run start=gui` workflow, which expects webpack-dev-server emitting a CSS manifest). Self-host runs against pre-built static bundles, so `env: "prod"` makes the homepage emit the `/dist/bundle.min.css` `<link>` tag instead of waiting on a manifest that doesn't exist.
- `database.migrationPaths` — Puter applies the bundled MySQL schema on boot. `mysql_mig_1.sql` (tables) and `mysql_mig_2.sql` (default apps: editor, viewer, pdf, camera, player, recorder, git, dev-center, puter-linux). Idempotent — safe to re-run.
- `database.migrationPaths` — Puter applies the bundled MySQL/MariaDB schema on boot. The migration files are idempotent, so it is safe to leave this configured across restarts.
- `dynamo.bootstrapTables: true` — Puter creates its KV table on boot. **Only set against a local emulator**, never real AWS.
- `dynamo.aws` keys are dummies; DynamoDB-local doesn't validate them but the AWS SDK requires _something_. **Note:** DynamoDB uses `access_key` / `secret_key` (snake_case); S3 below uses `accessKeyId` / `secretAccessKey` (camelCase). Not interchangeable.
- `providers.ollama.enabled: false` — Puter auto-probes a local Ollama at `127.0.0.1:11434` by default; without one running you'd see `ECONNREFUSED` on every boot. To run a bundled Ollama, see [Optional: local LLM (Ollama)](#optional-local-llm-ollama) below.
@@ -199,6 +199,32 @@ Drop the resulting `fullchain.pem` and `privkey.pem` into `./puter/tls/`.
"s3": { "s3Config": { "publicEndpoint": "https://s3.puter.local", ... } }
```
## Running behind your own reverse proxy
The bundled `puter-nginx` mirrors production and is the supported default — it terminates TLS (Step 3) and forwards every Host to Puter. But plenty of self-hosters already run their own edge proxy (Caddy, Traefik, HAProxy, a cloud load balancer, another nginx). You can put Puter behind it instead; Puter doesn't terminate TLS itself in any setup, so "your proxy does TLS" is just a matter of pointing it at the Puter container and getting a handful of details right.
You can keep the bundled nginx as the single hop your proxy talks to, or bypass it and forward straight to the Puter container on port `4100` (uncomment the `4100:4100` mapping under the `puter` service in [docker-compose.yml](../docker-compose.yml), or attach your proxy to the compose network). Either works — what matters is the rules below.
**The rules that actually matter** (getting any of these wrong is what causes the redirect loops and "Invalid Host header" failures people hit):
1. **Don't rewrite the `Host` header.** Puter routes entirely on Host — `api.<domain>`, `site.<domain>`, `app.<domain>` are all distinguished by the incoming host. Forward the original host through unchanged. Do **not** rewrite external hostnames to an internal name like `puter.local`; that forces you to remap every subdomain by hand and still breaks signed-URL and CORS checks. Most proxies preserve Host by default — just don't override it.
2. **The external domain must equal `domain` in your `config.json`.** If users reach the box at `puter.example.com`, then `domain` must be `puter.example.com` and the hosting domains must be its real subdomains (`site.puter.example.com`, `app.puter.example.com`, …) — exactly as the installer/`config.json` lays them out. Puter rejects hosts it doesn't recognize with `Invalid Host header`.
3. **Set `protocol` to match the public scheme.** When your proxy terminates TLS, set `"protocol": "https"` (installer: `PUTER_PROTOCOL=https`). Puter builds its origins, redirects, signed S3 URLs, and OIDC callback URLs from this — leave it `http` behind an HTTPS proxy and you get redirect loops, broken logins, and mixed-content errors.
4. **Forward the standard proxy headers.** Puter needs:
- `Host` — the original request host (rule 1).
- `X-Forwarded-Proto` — the **external** scheme (`https`), so Puter knows TLS was terminated upstream.
- `X-Forwarded-For` — the real client IP (rate limiting + audit logs).
- `Upgrade` / `Connection` — passed through for WebSocket / socket.io upgrades, or the realtime connection silently fails.
5. **Set `trust_proxy` to the number of hops in front of Puter.** One external proxy talking directly to Puter → `"trust_proxy": 1` (installer: `PUTER_TRUST_PROXY=1`). A second proxy in front (e.g. Cloudflare → your proxy → Puter) → `2`. This is the count of proxies between the client and Puter, **including** the bundled nginx if you keep it. Too low and `req.ip` becomes a proxy address (breaks rate limiting); never set `true` (it trusts every hop and makes `X-Forwarded-For` forgeable).
6. **Route the wildcard to Puter.** Your proxy must forward `*.<domain>` (covering `api.*`, `app.*`, and `s3.<domain>`) and `*.site.<domain>` (and any other hosting domains) to Puter — same wildcard DNS as Step 2. Puter does the per-subdomain routing internally; the proxy just needs to hand it the traffic with the Host intact.
With those in place the rest of the stack is unchanged — `docker compose up -d` as below.
## Step 4 — Bring it up
```bash
@@ -232,6 +258,26 @@ Change it in Settings after first login.
All optional. Drop any of the blocks below into `puter/config/config.json` and `docker compose restart puter`. See [config.template.jsonc](../config.template.jsonc) for the full list. Per-key documentation lives in [src/backend/types.ts](../src/backend/types.ts).
### PostgreSQL database
> **Community contribution — expect rough edges.** PostgreSQL support was contributed by the community and is not run by Puter.com production. It boots, applies the bundled schema, and exercises the common user/app/fsentry/session/permission/OIDC flows in the integration tests, but less-traveled SQL call sites may still need porting. If you hit a query that doesn't work on Postgres, please open an issue. For production self-hosting today, MariaDB/MySQL and SQLite are the supported defaults.
The bundled Docker Compose stack still defaults to MariaDB. To use PostgreSQL instead, run a PostgreSQL service yourself, point Puter at it, and use the PostgreSQL migration path:
```json
"database": {
"engine": "postgres",
"host": "postgres",
"port": 5432,
"user": "puter",
"password": "...",
"database": "puter",
"migrationPaths": ["/opt/puter/dist/src/backend/clients/database/migrations/postgres"]
}
```
You may use `"connectionString": "postgres://puter:...@postgres:5432/puter"` instead of the host/user/password fields. Puter only needs normal PostgreSQL connection details and the migration path; CloudNativePG is one possible Kubernetes operator, but no operator-specific manifests are required by Puter.
### Email (SMTP)
Used for password resets, email confirmation, and notifications. Without it those flows silently fail.
+1 -1
View File
@@ -264,7 +264,7 @@ services:
# Persistent runtime data (anything your config points at /var/puter).
- ./puter/data/puter:/var/puter:z
healthcheck:
test: wget --no-verbose --tries=1 --spider http://puter.localhost:4100/test || exit 1
test: wget --no-verbose --tries=1 --spider http://localhost:4100/ || exit 1
interval: 30s
timeout: 3s
retries: 3
+1 -1
View File
@@ -84,7 +84,7 @@ export const handleAppTelemetryUsers = async (
const users = await clients.db.read(
`SELECT u.username, u.uuid FROM user_to_app_permissions p
INNER JOIN user u ON p.user_id = u.id
INNER JOIN ${clients.db.quoteIdentifier('user')} u ON p.user_id = u.id
WHERE p.permission = 'flag:app-is-authenticated' AND p.app_id = ?
ORDER BY (p.dt IS NOT NULL), p.dt, p.user_id
LIMIT ? OFFSET ?`,
+1 -1
View File
@@ -71,6 +71,6 @@ export const handleInstalledApps = async (
extension.get(
'/installedApps',
{ subdomain: 'api', requireUserActor: true },
{ subdomain: 'api', requireUserActor: true, allowFullAccessToken: true },
handleInstalledApps,
);
+1 -1
View File
@@ -1,4 +1,3 @@
import type { Request, Response } from 'express';
import { Context } from '@heyputer/backend/src/core';
import { HttpError } from '@heyputer/backend/src/core/http';
import {
@@ -6,6 +5,7 @@ import {
driversContainers,
} from '@heyputer/backend/src/exports';
import { extension } from '@heyputer/backend/src/extensions';
import type { Request, Response } from 'express';
const services = extension.import('service');
const clients = extension.import('client');
+3
View File
@@ -72,6 +72,9 @@ export const handleWhoami = async (
unconfirmed_email: user.email,
email_confirmed: user.email_confirmed || user.username === 'admin',
requires_email_confirmation: user.requires_email_confirmation,
phone: user.phone,
requires_phone_verification: user.requires_phone_verification,
requires_card_verification: user.requires_card_verification,
desktop_bg_url: user.desktop_bg_url,
desktop_bg_color: user.desktop_bg_color,
desktop_bg_fit: user.desktop_bg_fit,
+22 -10
View File
@@ -16,11 +16,20 @@
# the compose file + brings the stack up. Set PUTER_FORCE=1 to overwrite.
#
# Tunable env vars:
# PUTER_DIR install directory (default: ./puter-selfhosted)
# PUTER_URL base URL to fetch docker-compose.yml (default: GitHub raw, main branch)
# PUTER_DOMAIN domain Puter will serve on (default: puter.localhost)
# PUTER_PORT HTTP port for nginx (default: 80)
# PUTER_FORCE set to 1 to overwrite existing .env / config.json
# PUTER_DIR install directory (default: ./puter-selfhosted)
# PUTER_URL base URL to fetch docker-compose.yml (default: GitHub raw, main branch)
# PUTER_DOMAIN domain Puter will serve on (default: puter.localhost)
# PUTER_PORT HTTP port for nginx (default: 80)
# PUTER_PROTOCOL public scheme: http | https (default: http)
# Set https when a TLS-terminating reverse proxy (Caddy,
# Traefik, nginx, a cloud LB) sits in front — Puter then
# builds https:// origins, S3 URLs, and redirects. See
# "Running behind your own reverse proxy" in doc/self-hosting.md.
# PUTER_TRUST_PROXY number of reverse-proxy hops in front (default: 1)
# 1 = the bundled nginx (or a single external proxy);
# 2 = two hops (e.g. Cloudflare → your proxy → Puter).
# PUTER_ENV prod | dev (default: prod)
# PUTER_FORCE set to 1 to overwrite existing .env / config.json
set -eu
@@ -28,6 +37,9 @@ PUTER_DIR="${PUTER_DIR:-puter-selfhosted}"
PUTER_URL="${PUTER_URL:-https://raw.githubusercontent.com/HeyPuter/puter/main}"
PUTER_DOMAIN="${PUTER_DOMAIN:-puter.localhost}"
PUTER_PORT="${PUTER_PORT:-80}"
PUTER_PROTOCOL="${PUTER_PROTOCOL:-http}"
PUTER_TRUST_PROXY="${PUTER_TRUST_PROXY:-1}"
PUTER_ENV="${PUTER_ENV:-prod}"
PUTER_FORCE="${PUTER_FORCE:-0}"
log() { printf '\033[1;36m[puter-install]\033[0m %s\n' "$*"; }
@@ -117,9 +129,9 @@ EOF
cat > puter/config/config.json <<EOF
{
"domain": "$PUTER_DOMAIN",
"protocol": "http",
"protocol": "$PUTER_PROTOCOL",
"pub_port": $PUTER_PORT,
"env": "prod",
"env": "$PUTER_ENV",
"static_hosting_domain": "site.$PUTER_DOMAIN",
"static_hosting_domain_alt": "host.$PUTER_DOMAIN",
@@ -159,7 +171,7 @@ EOF
"s3": {
"s3Config": {
"endpoint": "http://s3:9000",
"publicEndpoint": "http://s3.$PUTER_DOMAIN",
"publicEndpoint": "$PUTER_PROTOCOL://s3.$PUTER_DOMAIN",
"accessKeyId": "puter",
"secretAccessKey": "$S3_SECRET_KEY",
"region": "us-east-1",
@@ -173,7 +185,7 @@ EOF
"ollama": { "enabled": false }
},
"trust_proxy": 1
"trust_proxy": $PUTER_TRUST_PROXY
}
EOF
fi
@@ -187,6 +199,6 @@ log "stack starting. first boot takes ~30s while MariaDB initialises."
log "follow puter logs:"
log " cd $PUTER_DIR && docker compose logs -f puter"
log ""
log "open http://$PUTER_DOMAIN:$PUTER_PORT once the puter container is healthy."
log "open $PUTER_PROTOCOL://$PUTER_DOMAIN:$PUTER_PORT once the puter container is healthy."
log "first-boot admin password is logged once — grab it with:"
log " cd $PUTER_DIR && docker compose logs puter | grep password"
+4738 -4861
View File
File diff suppressed because it is too large. Load diff
+12 -4
View File
@@ -44,17 +44,18 @@
"yaml": "^2.8.1"
},
"scripts": {
"test:backend": "vitest run --config src/backend/vitest.config.ts ",
"test:backend": "npm run setupExtensions && vitest run --config src/backend/vitest.config.ts ",
"test:backend:postgres": "npm run setupExtensions && PUTER_TEST_DB_ENGINE=postgres vitest run --config src/backend/vitest.config.ts ",
"start=gui": "nodemon --exec \"node dev-server.js\" ",
"start": "node --enable-source-maps -r ./dist/src/backend/telemetry.js ./dist/src/backend/index.js",
"prestart": "npm run build:ts",
"prestart": "npm run setupExtensions && npm run build:ts",
"dev": "npm start",
"build": "npm run build:ts && cd src/gui && node ./build.js && cd ../puter-js && npm run build",
"build": "npm run setupExtensions && npm run build:ts && cd src/gui && node ./build.js && cd ../puter-js && npm run build",
"build:workerLib": "cd src/puter-js && npm run build && cd ../worker && npm run build",
"check-translations": "node tools/check-translations.js",
"prepare": "husky",
"build:ts": "tsc -p tsconfig.build.json && node ./tools/write-dist-package-json.mjs",
"postinstall": "node ./tools/extensionSetup.mjs"
"setupExtensions": "node ./tools/extensionSetup.mjs"
},
"workspaces": [
"src/*",
@@ -76,9 +77,16 @@
"ai": "^6.0.73",
"dedent": "^1.5.3",
"javascript-time-ago": "^2.5.11",
"libphonenumber-js": "1.13.6",
"open": "^10.1.0"
},
"engines": {
"node": ">=24.0.0"
},
"allowScripts": {
"[email protected]": true,
"[email protected]": true,
"@google/genai": false,
"[email protected]": true
}
}
@@ -0,0 +1,50 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Optional ClickHouse client.
*
* ClickHouse is NOT a hard dependency. Core Puter runs without it — the app
* stats read path (open / unique-user counts) falls back to the primary SQL
* database. `this.clients.clickhouse` is therefore typed as optional and is
* `undefined` in default and self-hosted setups; callers MUST branch on its
* presence and fall back to SQL.
*
* A production deployment can register a real ClickHouse client via an
* extension (`extension.registerClient('clickhouse', client)`) to offload the
* analytics queries off the primary database and keep the stats path fast at
* scale. When registered, the instance flows into `this.clients.clickhouse`
* everywhere it's typed.
*
* Only the surface the stats path actually consumes is declared here. Extend
* this interface (don't widen to `any`) when a new query shape is needed.
*/
export interface ClickhouseQueryResult {
json<T = Record<string, unknown>>(): Promise<T[]>;
}
export interface ClickhouseQueryParams {
query: string;
query_params?: Record<string, unknown>;
format?: string;
}
export interface ClickhouseClient {
query(params: ClickhouseQueryParams): Promise<ClickhouseQueryResult>;
}
+90 -3
View File
@@ -31,6 +31,8 @@ export interface BatchEntry {
values: unknown[];
}
type SqlJsonPath = readonly [string, ...string[]];
/**
* Base database client. Subclasses must override every method that throws here.
*
@@ -101,9 +103,10 @@ export class AbstractDatabaseClient extends PuterClient {
const cols = Object.keys(data);
const values = Object.values(data);
const sql =
`INSERT INTO \`${tableName}\` ` +
`(${cols.map((c) => `\`${c}\``).join(', ')}) ` +
`VALUES (${cols.map(() => '?').join(', ')})`;
`INSERT INTO ${this.quoteIdentifier(tableName)} ` +
`(${cols.map((c) => this.quoteIdentifier(c)).join(', ')}) ` +
`VALUES (${cols.map(() => '?').join(', ')})` +
this.returningIdClause();
return this.write(sql, values);
}
@@ -161,4 +164,88 @@ export class AbstractDatabaseClient extends PuterClient {
}
return choices.otherwise as T;
}
quoteIdentifier(identifier: string): string {
return identifier
.split('.')
.map((part) => {
if (part === '*') return part;
return `\`${part.replaceAll('`', '``')}\``;
})
.join('.');
}
booleanLiteral(value: boolean): string {
return value ? '1' : '0';
}
booleanValue(value: boolean): boolean | 0 | 1 {
return value ? 1 : 0;
}
insertIgnoreInto(tableName: string): string {
const table = this.quoteIdentifier(tableName);
return this.case({
sqlite: `INSERT OR IGNORE INTO ${table}`,
postgres: `INSERT INTO ${table}`,
otherwise: `INSERT IGNORE INTO ${table}`,
});
}
insertIgnoreSuffix(): string {
return this.case({
postgres: ' ON CONFLICT DO NOTHING',
otherwise: '',
});
}
upsertClause(
conflictColumns: readonly string[],
updateColumns: readonly string[],
): string {
if (updateColumns.length === 0) {
throw new Error('upsertClause requires at least one update column');
}
const updateList = updateColumns
.map((column) => `${this.quoteIdentifier(column)} = ?`)
.join(', ');
return this.case({
mysql: `ON DUPLICATE KEY UPDATE ${updateList}`,
otherwise: `ON CONFLICT(${conflictColumns
.map((column) => this.quoteIdentifier(column))
.join(', ')}) DO UPDATE SET ${updateList}`,
});
}
jsonTextExtract(jsonExpression: string, path: SqlJsonPath): string {
const sqlitePath = `$${path.map((part) => `.${part}`).join('')}`;
return this.case({
sqlite: `json_extract(${jsonExpression}, ${this.sqlStringLiteral(sqlitePath)})`,
mysql: `JSON_UNQUOTE(JSON_EXTRACT(${jsonExpression}, ${this.sqlStringLiteral(sqlitePath)}))`,
postgres: `${jsonExpression} #>> ARRAY[${path
.map((part) => this.sqlStringLiteral(part))
.join(', ')}]`,
otherwise: `JSON_UNQUOTE(JSON_EXTRACT(${jsonExpression}, ${this.sqlStringLiteral(sqlitePath)}))`,
});
}
nullCoalesce(...expressions: readonly string[]): string {
if (expressions.length === 0) {
throw new Error('nullCoalesce requires at least one expression');
}
return `COALESCE(${expressions.join(', ')})`;
}
returningIdClause(): string {
return this.case({
postgres: ' RETURNING id',
otherwise: '',
});
}
protected sqlStringLiteral(value: string): string {
return `'${value.replaceAll("'", "''")}'`;
}
}
@@ -23,6 +23,7 @@ import { createPool, type Pool } from 'mysql2';
import { AbstractDatabaseClient, type WriteResult } from './DatabaseClient';
import { SQLBatcher } from './SQLBatcher.js';
import { splitMysqlStatements } from './splitMysqlStatements.js';
import { compareMigrationFilenames } from './migrationFilenames.js';
import type { IConfig } from '../../types';
const RETRIABLE_ERROR_CODES = new Set([
@@ -44,36 +45,7 @@ const RETRIABLE_ERROR_MESSAGES = [
'ETIMEDOUT',
];
/**
* Comparator for MySQL migration filenames.
*
* Existing files are named `mysql_mig_<N>.sql` with unpadded N, so a
* plain lexical sort puts `mysql_mig_10.sql` BEFORE `mysql_mig_2.sql`.
* Pull the trailing integer out and sort numerically. Anything that
* doesn't match the `_<digits>.sql` shape (one-off, vendor dump) falls
* back to lexical comparison so the order stays deterministic, and
* unmatched names sort *after* numbered files so future numbered
* migrations don't get interleaved into a one-off's namespace.
*
* Exported only for the unit test — the production caller is the
* `runMigrations()` loop inside this file.
*/
export const compareMigrationFilenames = (a: string, b: string): number => {
const numericIndex = (name: string): number => {
const m = /_(\d+)\.sql$/.exec(name);
return m ? Number.parseInt(m[1], 10) : Number.NaN;
};
const na = numericIndex(a);
const nb = numericIndex(b);
if (Number.isFinite(na) && Number.isFinite(nb)) {
if (na !== nb) return na - nb;
} else if (Number.isFinite(na)) {
return -1;
} else if (Number.isFinite(nb)) {
return 1;
}
return a.localeCompare(b);
};
export { compareMigrationFilenames };
type PoolConfig = Parameters<typeof createPool>[0];
@@ -0,0 +1,328 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { Pool } from 'pg';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import type { IConfig } from '../../types';
import type { PuterServer } from '../../server';
import { generateDefaultFsentries } from '../../util/userProvisioning.js';
import {
createPgMockPostgresDatabaseClient,
POSTGRES_TEST_MIGRATIONS_PATH,
setupTestServer,
} from '../../testUtil.js';
import { PostgresDatabaseClient } from './PostgresDatabaseClient.js';
const postgresUrl = process.env.PUTER_TEST_POSTGRES_URL;
const postgresMigrationsPath = POSTGRES_TEST_MIGRATIONS_PATH;
const postgresTestSchemaPattern = /^puter_test_[a-f0-9]{32}$/u;
const postgresIntegrationTimeoutMs = 180_000;
let postgresTestSchema: string | undefined;
let postgresTestUrl: string | undefined;
const postgresConfig = (overrides: Partial<IConfig> = {}): IConfig => {
if (postgresUrl && !postgresTestUrl) {
throw new Error('Postgres test schema was not initialized');
}
const { database: databaseOverrides, ...rootOverrides } = overrides;
const database = postgresUrl
? {
engine: 'postgres' as const,
inMemory: false,
connectionString: postgresTestUrl,
migrationPaths: [postgresMigrationsPath],
}
: {
engine: 'postgres' as const,
inMemory: true,
migrationPaths: [postgresMigrationsPath],
};
return {
port: 0,
extensions: [],
database: { ...database, ...(databaseOverrides ?? {}) },
...rootOverrides,
};
};
const quoteTestSchemaIdentifier = (schema: string): string => {
if (!postgresTestSchemaPattern.test(schema)) {
throw new Error(`Unsafe Postgres test schema name: ${schema}`);
}
return `"${schema}"`;
};
const postgresConnectionStringForSchema = (
connectionString: string,
schema: string,
): string => {
if (!postgresTestSchemaPattern.test(schema)) {
throw new Error(`Unsafe Postgres test schema name: ${schema}`);
}
const url = new URL(connectionString);
url.searchParams.set('options', `-c search_path=${schema}`);
return url.toString();
};
const createPostgresTestSchema = async (): Promise<void> => {
if (!postgresUrl) return;
const schema = `puter_test_${uuidv4().replaceAll('-', '')}`;
const pool = new Pool({ connectionString: postgresUrl });
try {
await pool.query(`CREATE SCHEMA ${quoteTestSchemaIdentifier(schema)}`);
postgresTestSchema = schema;
postgresTestUrl = postgresConnectionStringForSchema(
postgresUrl,
schema,
);
} finally {
await pool.end();
}
};
const dropPostgresTestSchema = async (): Promise<void> => {
if (!postgresUrl || !postgresTestSchema) return;
const schema = postgresTestSchema;
postgresTestSchema = undefined;
postgresTestUrl = undefined;
const pool = new Pool({ connectionString: postgresUrl });
try {
await pool.query(
`DROP SCHEMA IF EXISTS ${quoteTestSchemaIdentifier(schema)} CASCADE`,
);
} finally {
await pool.end();
}
};
describe('PostgresDatabaseClient integration', () => {
let server: PuterServer | undefined;
beforeEach(async () => {
await createPostgresTestSchema();
});
afterEach(async () => {
try {
await server?.shutdown();
} finally {
server = undefined;
await dropPostgresTestSchema();
}
});
it(
'applies the native migrations idempotently to an empty database',
async () => {
const config = postgresConfig();
const pgMockClient = postgresUrl
? undefined
: await createPgMockPostgresDatabaseClient(config);
try {
const firstClient =
pgMockClient?.client ?? new PostgresDatabaseClient(config);
try {
await firstClient.onServerStart();
} finally {
await firstClient.onServerShutdown();
}
const secondClient =
pgMockClient?.createClient() ??
new PostgresDatabaseClient(config);
try {
await secondClient.onServerStart();
const [systemUser] = await secondClient.read(
'SELECT `id`, `username` FROM `user` WHERE `username` = ?',
['system'],
);
const [devCenter] = await secondClient.read(
'SELECT `name`, `index_url` FROM `apps` WHERE `name` = ?',
['dev-center'],
);
expect(systemUser).toMatchObject({
id: 1,
username: 'system',
});
expect(devCenter).toMatchObject({
name: 'dev-center',
index_url:
'https://builtins.namespaces.puter.com/dev-center',
});
} finally {
await secondClient.onServerShutdown();
}
} finally {
pgMockClient?.destroy();
}
},
postgresIntegrationTimeoutMs,
);
it('starts the server and exercises user, app, fsentry, permission, and session flows', async () => {
const userStorageAllowance = 123_456_789;
server = await setupTestServer(
postgresConfig({
no_default_user: false,
is_storage_limited: true,
}),
);
const admin = await server.stores.user.getByUsername('admin');
expect(admin?.username).toBe('admin');
const username = `pg-${uuidv4().slice(0, 8)}`;
const createdUser = await server.stores.user.create({
username,
uuid: uuidv4(),
password: null,
email: `${username}@test.local`,
free_storage: userStorageAllowance,
requires_email_confirmation: false,
});
await generateDefaultFsentries(
server.clients.db,
server.stores.user,
createdUser,
);
const user = await server.stores.user.getById(createdUser.id);
if (!user) throw new Error('created user was not readable');
expect(user.username).toBe(username);
await expect(
server.stores.fsEntry.getUserStorageAllowance(user.id),
).resolves.toMatchObject({
max: userStorageAllowance,
});
const otherUsername = `pg-other-${uuidv4().slice(0, 8)}`;
const otherUser = await server.stores.user.create({
username: otherUsername,
uuid: uuidv4(),
password: null,
email: `${otherUsername}@test.local`,
free_storage: 100 * 1024 * 1024,
requires_email_confirmation: false,
});
const authResult = await server.services.auth.createSessionToken(
user,
{
ip: '127.0.0.1',
user_agent: 'postgres-integration-test',
},
);
const authenticated =
await server.services.auth.authenticateFromToken(authResult.token);
expect(authenticated?.user?.id).toBe(user?.id);
const devCenter = await server.stores.app.getByName('dev-center');
if (!devCenter) throw new Error('dev-center app was not seeded');
expect(devCenter.name).toBe('dev-center');
const documents = await server.stores.fsEntry.getEntryByPath(
`/${username}/Documents`,
);
if (!documents) throw new Error('Documents directory was not created');
expect(documents.isDir).toBe(true);
const folder = await server.stores.fsEntry.createNonFileEntry({
userId: user.id,
parent: documents,
name: 'postgres-folder',
kind: 'directory',
});
const renamedFolder = await server.stores.fsEntry.updateEntry(
folder.uuid,
{
name: 'postgres-folder-renamed',
path: `/${username}/Documents/postgres-folder-renamed`,
},
);
expect(renamedFolder.name).toBe('postgres-folder-renamed');
await server.stores.permission.upsertUserAppPerm(
user.id,
Number(devCenter.id),
'driver:postgres-integration',
{ ok: true },
);
await expect(
server.stores.permission.hasUserAppPerm(
user.id,
Number(devCenter.id),
'driver:postgres-integration',
),
).resolves.toBe(true);
await server.stores.oidc.link(
user.id,
'postgres-integration-test',
'subject-1',
null,
);
await expect(
server.stores.oidc.link(
user.id,
'postgres-integration-test',
'subject-1',
null,
),
).resolves.toBeUndefined();
await expect(
server.stores.oidc.link(
otherUser.id,
'postgres-integration-test',
'subject-1',
null,
),
).rejects.toMatchObject({
statusCode: 409,
legacyCode: 'conflict',
});
const session = await server.stores.session.create(user.id, {
meta: { source: 'postgres-integration-test' },
});
const activeSession = await server.stores.session.getByUuid(
session.uuid,
);
expect(activeSession?.uuid).toBe(session.uuid);
const workerName = `worker-${uuidv4().slice(0, 8)}`;
const workerSession = await server.stores.session.getOrCreateWorker(
user.id,
{ workerName },
);
expect(workerSession?.kind).toBe('worker');
expect(workerSession?.meta?.worker_name).toBe(workerName);
await server.stores.session.removeByUuid(session.uuid);
await expect(
server.stores.session.getByUuid(session.uuid),
).resolves.toBeNull();
}, postgresIntegrationTimeoutMs);
});
@@ -0,0 +1,360 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { FieldDef, QueryResult } from 'pg';
import { describe, expect, it } from 'vitest';
import type { IConfig } from '../../types';
import { DatabaseClientFactory } from './index.js';
import {
mapPostgresWriteResult,
PostgresDatabaseClient,
type PostgresPool,
type PostgresPoolClient,
} from './PostgresDatabaseClient.js';
type QueryCall = {
text: string;
values?: unknown[];
};
const postgresConfig = (): IConfig => ({
port: 0,
extensions: [],
database: {
engine: 'postgres',
migrationPaths: [],
},
});
const postgresReplicaConfig = (): IConfig => ({
port: 0,
extensions: [],
database: {
engine: 'postgres',
migrationPaths: [],
replica: {},
},
});
const field = (name: string, dataTypeID: number): FieldDef => ({
name,
tableID: 0,
columnID: 0,
dataTypeID,
dataTypeSize: -1,
dataTypeModifier: -1,
format: 'text',
});
const int8Field = (name: string): FieldDef => field(name, 20);
const textField = (name: string): FieldDef => field(name, 25);
const queryResult = (
rows: Record<string, unknown>[] = [],
rowCount = rows.length,
fields: FieldDef[] = [],
): QueryResult<Record<string, unknown>> => ({
command: '',
fields,
oid: 0,
rowCount,
rows,
});
class RecordingPoolClient implements PostgresPoolClient {
readonly calls: QueryCall[] = [];
released = false;
constructor(private readonly failOnText?: string) {}
async query(
text: string,
values?: unknown[],
): Promise<QueryResult<Record<string, unknown>>> {
this.calls.push({ text, values });
if (this.failOnText && text.includes(this.failOnText)) {
throw new Error(`forced query failure: ${text}`);
}
return queryResult([], text === 'ROLLBACK' ? 0 : 1);
}
release(): void {
this.released = true;
}
}
class RecordingPool implements PostgresPool {
readonly calls: QueryCall[] = [];
constructor(
private readonly client: RecordingPoolClient =
new RecordingPoolClient(),
private readonly nextResult: QueryResult<Record<string, unknown>> =
queryResult(),
) {}
async query(
text: string,
values?: unknown[],
): Promise<QueryResult<Record<string, unknown>>> {
this.calls.push({ text, values });
if (text === 'SELECT 1') return queryResult([{ ok: 1 }], 1);
return this.nextResult;
}
async connect(): Promise<PostgresPoolClient> {
return this.client;
}
async end(): Promise<void> {}
}
describe('PostgresDatabaseClient', () => {
it('is selected by the database factory', () => {
const client = new DatabaseClientFactory(postgresConfig());
expect(client).toBeInstanceOf(PostgresDatabaseClient);
});
it('maps pg write results to the shared WriteResult shape', () => {
expect(
mapPostgresWriteResult(queryResult([{ id: '42' }], 1)),
).toEqual({
insertId: 42,
affectedRows: 1,
anyRowsAffected: true,
});
expect(mapPostgresWriteResult(queryResult([], 0))).toEqual({
insertId: 0,
affectedRows: 0,
anyRowsAffected: false,
});
});
it('prepares write SQL at the pg boundary', async () => {
const pool = new RecordingPool(
new RecordingPoolClient(),
queryResult([{ id: '7' }], 1),
);
const client = new PostgresDatabaseClient(postgresConfig(), () => pool);
await client.onServerStart();
const result = await client.write(
'INSERT INTO `apps` (`name`) VALUES (?) RETURNING id',
['editor'],
);
expect(result.insertId).toBe(7);
expect(pool.calls.at(-1)).toEqual({
text: 'INSERT INTO "apps" ("name") VALUES ($1) RETURNING id',
values: ['editor'],
});
});
it('normalizes int8 fields on read and primary read rows', async () => {
const pool = new RecordingPool(
new RecordingPoolClient(),
queryResult(
[
{
uuid: 'session-1',
created_at: '1710000000',
last_activity: '1710000001',
expires_at: '1710000002',
revoked_at: null,
},
],
1,
[
textField('uuid'),
int8Field('created_at'),
int8Field('last_activity'),
int8Field('expires_at'),
int8Field('revoked_at'),
],
),
);
const client = new PostgresDatabaseClient(postgresConfig(), () => pool);
await client.onServerStart();
await expect(client.read('SELECT * FROM `sessions`')).resolves.toEqual(
[
{
uuid: 'session-1',
created_at: 1710000000,
last_activity: 1710000001,
expires_at: 1710000002,
revoked_at: null,
},
],
);
await expect(client.pread('SELECT * FROM `sessions`')).resolves.toEqual(
[
{
uuid: 'session-1',
created_at: 1710000000,
last_activity: 1710000001,
expires_at: 1710000002,
revoked_at: null,
},
],
);
});
it('rejects unsafe int8 values instead of losing precision', async () => {
const pool = new RecordingPool(
new RecordingPoolClient(),
queryResult(
[{ id: '9007199254740992' }],
1,
[int8Field('id')],
),
);
const client = new PostgresDatabaseClient(postgresConfig(), () => pool);
await client.onServerStart();
await expect(client.read('SELECT `id` FROM `sessions`')).rejects.toThrow(
'safe integer',
);
});
it('normalizes tryHardRead rows returned by a replica', async () => {
const primaryPool = new RecordingPool(
new RecordingPoolClient(),
queryResult([{ created_at: '1' }], 1, [
int8Field('created_at'),
]),
);
const replicaPool = new RecordingPool(
new RecordingPoolClient(),
queryResult([{ created_at: '2' }], 1, [
int8Field('created_at'),
]),
);
const pools = [primaryPool, replicaPool];
let nextPoolIndex = 0;
const client = new PostgresDatabaseClient(
postgresReplicaConfig(),
() => {
const pool = pools[nextPoolIndex];
nextPoolIndex += 1;
if (!pool) throw new Error('unexpected pool factory call');
return pool;
},
);
await client.onServerStart();
await expect(
client.tryHardRead('SELECT `created_at` FROM `sessions`'),
).resolves.toEqual([{ created_at: 2 }]);
});
it('normalizes tryHardRead rows returned by primary fallback', async () => {
const primaryPool = new RecordingPool(
new RecordingPoolClient(),
queryResult([{ created_at: '3' }], 1, [
int8Field('created_at'),
]),
);
const replicaPool = new RecordingPool(
new RecordingPoolClient(),
queryResult([], 0, [int8Field('created_at')]),
);
const pools = [primaryPool, replicaPool];
let nextPoolIndex = 0;
const client = new PostgresDatabaseClient(
postgresReplicaConfig(),
() => {
const pool = pools[nextPoolIndex];
nextPoolIndex += 1;
if (!pool) throw new Error('unexpected pool factory call');
return pool;
},
);
await client.onServerStart();
await expect(
client.tryHardRead('SELECT `created_at` FROM `sessions`'),
).resolves.toEqual([{ created_at: 3 }]);
});
it('runs batch writes in order and commits', async () => {
const conn = new RecordingPoolClient();
const client = new PostgresDatabaseClient(
postgresConfig(),
() => new RecordingPool(conn),
);
await client.onServerStart();
await client.batchWrite([
{
statement:
'UPDATE `user` SET `username` = ? WHERE `id` = ?',
values: ['ada', 1],
},
{
statement: 'DELETE FROM `sessions` WHERE `uuid` = ?',
values: ['session-1'],
},
]);
expect(conn.calls).toEqual([
{ text: 'BEGIN', values: undefined },
{
text: 'UPDATE "user" SET "username" = $1 WHERE "id" = $2',
values: ['ada', 1],
},
{
text: 'DELETE FROM "sessions" WHERE "uuid" = $1',
values: ['session-1'],
},
{ text: 'COMMIT', values: undefined },
]);
expect(conn.released).toBe(true);
});
it('rolls back and releases the pg connection when a batch write fails', async () => {
const conn = new RecordingPoolClient('UPDATE "user"');
const client = new PostgresDatabaseClient(
postgresConfig(),
() => new RecordingPool(conn),
);
await client.onServerStart();
await expect(
client.batchWrite([
{
statement:
'UPDATE `user` SET `username` = ? WHERE `id` = ?',
values: ['ada', 1],
},
]),
).rejects.toThrow('forced query failure');
expect(conn.calls).toEqual([
{ text: 'BEGIN', values: undefined },
{
text: 'UPDATE "user" SET "username" = $1 WHERE "id" = $2',
values: ['ada', 1],
},
{ text: 'ROLLBACK', values: undefined },
]);
expect(conn.released).toBe(true);
});
});
@@ -0,0 +1,390 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { readdirSync, readFileSync } from 'fs';
import { isAbsolute, resolve as resolvePath } from 'path';
import { Pool, type PoolConfig, type QueryResult } from 'pg';
import {
AbstractDatabaseClient,
type BatchEntry,
type WriteResult,
} from './DatabaseClient';
import { compareMigrationFilenames } from './migrationFilenames.js';
import { preparePostgresSql } from './preparePostgresSql.js';
import { splitPostgresStatements } from './splitPostgresStatements.js';
import type { IConfig } from '../../types';
type PostgresEndpointConfig = {
host?: string;
port?: number;
user?: string;
password?: string;
database?: string;
connectionString?: string;
url?: string;
};
export interface PostgresQueryable {
query(query: string, values?: unknown[]): Promise<QueryResult>;
}
export interface PostgresPoolClient extends PostgresQueryable {
release(): void;
}
export interface PostgresPool extends PostgresQueryable {
connect(): Promise<PostgresPoolClient>;
end(): Promise<void>;
}
type PostgresPoolFactory = (poolConfig: PoolConfig) => PostgresPool;
enum Configuration {
SINGLE,
REPLICA,
}
const POSTGRES_INT8_OID = 20;
const INTEGER_TEXT_PATTERN = /^-?\d+$/u;
const normalizePostgresInt8 = (
value: unknown,
columnName: string,
): number | null | undefined => {
if (value === null || value === undefined) return value;
const parsed =
typeof value === 'bigint'
? Number(value)
: typeof value === 'number'
? value
: typeof value === 'string' && INTEGER_TEXT_PATTERN.test(value)
? Number(value)
: Number.NaN;
if (!Number.isSafeInteger(parsed)) {
throw new Error(
`[postgres] int8 column ${columnName} is outside JavaScript's safe integer range`,
);
}
return parsed;
};
const normalizePostgresRows = (
result: QueryResult,
): Record<string, unknown>[] => {
const int8Fields = result.fields.filter(
(field) => field.dataTypeID === POSTGRES_INT8_OID,
);
if (int8Fields.length === 0) {
return result.rows as Record<string, unknown>[];
}
return result.rows.map((row) => {
const normalized: Record<string, unknown> = { ...row };
for (const field of int8Fields) {
normalized[field.name] = normalizePostgresInt8(
normalized[field.name],
field.name,
);
}
return normalized;
});
};
export const mapPostgresWriteResult = (result: QueryResult): WriteResult => {
const affectedRows = result.rowCount ?? 0;
const rowId = result.rows[0]?.id;
const insertId =
typeof rowId === 'bigint'
? rowId
: typeof rowId === 'number'
? rowId
: typeof rowId === 'string' && rowId !== ''
? Number(rowId)
: 0;
const normalizedInsertId =
typeof insertId === 'number' && Number.isNaN(insertId) ? 0 : insertId;
return {
insertId: normalizedInsertId,
affectedRows,
anyRowsAffected: affectedRows > 0,
};
};
export class PostgresDatabaseClient extends AbstractDatabaseClient {
override readonly engineName = 'postgres';
private primaryPool!: PostgresPool;
private replicaPool!: PostgresPool;
private configuration = Configuration.SINGLE;
private shutdownStarted = false;
private shutdownTimer: ReturnType<typeof setTimeout> | null = null;
constructor(
config: IConfig,
private readonly poolFactory: PostgresPoolFactory = (poolConfig) =>
new Pool(poolConfig) as unknown as PostgresPool,
) {
super(config);
}
override async onServerStart(): Promise<void> {
const dbConf = this.config.database!;
this.primaryPool = this.createPool(dbConf);
await this.primaryPool.query('SELECT 1');
console.log('[postgres] connected to primary');
if (dbConf.replica) {
this.replicaPool = this.createPool(dbConf.replica);
await this.replicaPool.query('SELECT 1');
this.configuration = Configuration.REPLICA;
console.log('[postgres] connected to read-replica');
} else {
this.replicaPool = this.primaryPool;
this.configuration = Configuration.SINGLE;
}
await this.runMigrations();
}
override async onServerPrepareShutdown(): Promise<void> {
if (this.shutdownStarted) return;
this.shutdownStarted = true;
const drainMs = 60_000;
console.log(
`[postgres] draining in-flight queries (${drainMs}ms) before closing pools`,
);
this.shutdownTimer = setTimeout(() => {
this.shutdownTimer = null;
this.closeCurrentPools().catch((e) =>
console.error('[postgres] error closing pools after drain', e),
);
}, drainMs);
if (typeof this.shutdownTimer.unref === 'function') {
this.shutdownTimer.unref();
}
}
override async onServerShutdown(): Promise<void> {
if (this.shutdownTimer) {
clearTimeout(this.shutdownTimer);
this.shutdownTimer = null;
}
await this.closeCurrentPools();
}
override quoteIdentifier(identifier: string): string {
return identifier
.split('.')
.map((part) => {
if (part === '*') return part;
return `"${part.replaceAll('"', '""')}"`;
})
.join('.');
}
override booleanLiteral(value: boolean): string {
return value ? 'TRUE' : 'FALSE';
}
override booleanValue(value: boolean): boolean {
return value;
}
override async read(
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
const result = await this.query(this.replicaPool, query, params);
return normalizePostgresRows(result);
}
override async pread(
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
const result = await this.query(this.primaryPool, query, params);
return normalizePostgresRows(result);
}
override async write(
query: string,
params: unknown[] = [],
): Promise<WriteResult> {
const result = await this.query(this.primaryPool, query, params);
return mapPostgresWriteResult(result);
}
override async batchWrite(entries: BatchEntry[]): Promise<void> {
if (entries.length === 0) return;
const conn = await this.primaryPool.connect();
try {
await conn.query('BEGIN');
try {
for (const { statement, values } of entries) {
await this.query(conn, statement, values);
}
await conn.query('COMMIT');
} catch (err) {
await conn.query('ROLLBACK').catch(() => {});
throw err;
}
} finally {
conn.release();
}
}
override async tryHardRead(
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
if (this.configuration === Configuration.SINGLE) {
return this.read(query, params);
}
const primaryPromise = this.query(this.primaryPool, query, params);
try {
const replicaResult = await this.query(
this.replicaPool,
query,
params,
);
if (replicaResult.rows.length > 0) {
primaryPromise.catch(() => {});
return normalizePostgresRows(replicaResult);
}
} catch {
// fall through to primary
}
const primaryResult = await primaryPromise;
return normalizePostgresRows(primaryResult);
}
private async runMigrations(): Promise<void> {
const paths = this.config.database?.migrationPaths;
if (!paths || paths.length === 0) return;
const conn = await this.primaryPool.connect();
try {
for (const rawPath of paths) {
const dir = isAbsolute(rawPath)
? rawPath
: resolvePath(process.cwd(), rawPath);
let files: string[];
try {
files = readdirSync(dir)
.filter(
(f) =>
f.endsWith('.sql') && f.startsWith('postgres'),
)
.sort(compareMigrationFilenames);
} catch (e) {
throw new Error(
`[postgres] migration path is unreadable: ${dir}`,
{ cause: e },
);
}
if (files.length === 0) {
console.log(`[postgres] no migrations in ${dir}`);
continue;
}
console.log(
`[postgres] running migrations from ${dir}: ${files.length} file(s)`,
);
for (const file of files) {
const filePath = resolvePath(dir, file);
const contents = readFileSync(filePath, 'utf8');
const statements = splitPostgresStatements(contents);
await conn.query('BEGIN');
try {
for (let i = 0; i < statements.length; i++) {
try {
await conn.query(statements[i]);
} catch (e) {
throw new Error(
`[postgres] failed to apply ${file} at statement ${i}`,
{ cause: e },
);
}
}
await conn.query('COMMIT');
} catch (e) {
await conn.query('ROLLBACK').catch(() => {});
throw e;
}
console.log(
`[postgres] applied ${file} (${statements.length} statements)`,
);
}
}
} finally {
conn.release();
}
}
private createPool(dbConf: PostgresEndpointConfig): PostgresPool {
const connectionString = dbConf.connectionString ?? dbConf.url;
if (connectionString) {
return this.poolFactory({
connectionString,
max: 30,
});
}
return this.poolFactory({
host: dbConf.host ?? '127.0.0.1',
port: dbConf.port ?? 5432,
user: dbConf.user ?? 'postgres',
password: dbConf.password ?? '',
database: dbConf.database ?? 'puter',
max: 30,
});
}
private async query(
target: PostgresQueryable,
query: string,
params: unknown[] = [],
): Promise<QueryResult> {
const prepared = preparePostgresSql(query);
return target.query(prepared.text, params);
}
private async closeCurrentPools(): Promise<void> {
const tasks: Promise<void>[] = [];
if (this.primaryPool) tasks.push(this.primaryPool.end());
if (this.replicaPool && this.replicaPool !== this.primaryPool) {
tasks.push(this.replicaPool.end());
}
await Promise.all(tasks);
}
}
@@ -85,6 +85,9 @@ const AVAILABLE_MIGRATIONS: [number, string[]][] = [
[50, ['0054_sessions_workers.sql']],
[50, ['0055_username_nocase_unique.sql']],
[51, ['0056_sessions_kind_worker.sql']],
[52, ['0057_add_user_reputation.sql']],
[53, ['0058_add_phone_verification.sql']],
[54, ['0059_add_card_verification.sql']],
];
export class SqliteDatabaseClient extends AbstractDatabaseClient {
+4
View File
@@ -24,10 +24,12 @@ export {
} from './DatabaseClient';
export { SqliteDatabaseClient } from './SqliteDatabaseClient';
export { MySQLDatabaseClient } from './MySQLDatabaseClient';
export { PostgresDatabaseClient } from './PostgresDatabaseClient';
import type { IConfig } from '../../types';
import { AbstractDatabaseClient } from './DatabaseClient';
import { MySQLDatabaseClient } from './MySQLDatabaseClient';
import { PostgresDatabaseClient } from './PostgresDatabaseClient';
import { SqliteDatabaseClient } from './SqliteDatabaseClient';
/**
@@ -41,6 +43,8 @@ export const DatabaseClientFactory = class DatabaseClientFactory {
switch (engine) {
case 'mysql':
return new MySQLDatabaseClient(config);
case 'postgres':
return new PostgresDatabaseClient(config);
case 'sqlite':
return new SqliteDatabaseClient(config);
default:
@@ -0,0 +1,43 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Comparator for migration filenames named `<engine>_mig_<N>.sql`.
*
* Existing MySQL files use unpadded integers, so lexical sorting places
* `*_mig_10.sql` before `*_mig_2.sql`. Pull the trailing integer out and sort
* numerically. Anything that does not match the `_<digits>.sql` shape falls
* back to lexical comparison and sorts after numbered files.
*/
export const compareMigrationFilenames = (a: string, b: string): number => {
const numericIndex = (name: string): number => {
const m = /_(\d+)\.sql$/.exec(name);
return m ? Number.parseInt(m[1], 10) : Number.NaN;
};
const na = numericIndex(a);
const nb = numericIndex(b);
if (Number.isFinite(na) && Number.isFinite(nb)) {
if (na !== nb) return na - nb;
} else if (Number.isFinite(na)) {
return -1;
} else if (Number.isFinite(nb)) {
return 1;
}
return a.localeCompare(b);
};
@@ -24,7 +24,9 @@
--
--
-- Idempotent column-ensure helper (used by CALLs below; dropped at end of file)
-- Idempotent column-ensure helper. Used by the CALLs below AND by later
-- migration files, which CALL it as a one-line idempotent column add. Left
-- resident (not dropped at end of file) so it outlives this migration.
--
DROP PROCEDURE IF EXISTS _puter_add_col;
@@ -1333,7 +1335,10 @@ CALL _puter_add_col('user_update_audit', 'new_username', '`new_username` varchar
CALL _puter_add_col('user_update_audit', 'reason', '`reason` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL');
DROP PROCEDURE IF EXISTS _puter_add_col;
-- _puter_add_col is intentionally NOT dropped here: later migration files CALL
-- it as a one-line idempotent column add, and migrations replay on every boot
-- with no applied-state tracking, so the helper must outlive this file. The
-- DROP-before-CREATE at the top keeps this migration itself replay-safe.
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
@@ -0,0 +1,47 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- SMS phone verification columns. Mirrors SQLite migration 0058. `phone` is the
-- E.164 number collected during verification (indexed like `email`);
-- `requires_phone_verification` gates account use for low-reputation signups
-- (not indexed, mirroring `requires_email_confirmation`).
--
-- Idempotent: column adds use _puter_add_col (defined in mig_1, which leaves it
-- resident for later migrations); the index add is guarded against
-- INFORMATION_SCHEMA.STATISTICS so the directory replays safely.
CALL _puter_add_col('user', 'phone', '`phone` varchar(20) DEFAULT NULL');
CALL _puter_add_col('user', 'requires_phone_verification', '`requires_phone_verification` tinyint(1) NOT NULL DEFAULT ''0''');
DROP PROCEDURE IF EXISTS _puter_add_user_phone_index;
DELIMITER //
CREATE PROCEDURE _puter_add_user_phone_index()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user'
AND INDEX_NAME = 'idx_user_phone'
) THEN
ALTER TABLE `user` ADD INDEX `idx_user_phone` (`phone`);
END IF;
END//
DELIMITER ;
CALL _puter_add_user_phone_index();
DROP PROCEDURE IF EXISTS _puter_add_user_phone_index;
@@ -0,0 +1,26 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Credit-card verification column. Mirrors SQLite migration 0059.
-- `requires_card_verification` gates account use for low-reputation signups;
-- the card itself never touches our DB, so this is the only column (not
-- indexed, mirroring `requires_phone_verification`).
--
-- Idempotent: the column add uses _puter_add_col (from mig_1) so the
-- directory replays safely.
CALL _puter_add_col('user', 'requires_card_verification', '`requires_card_verification` tinyint(1) NOT NULL DEFAULT ''0''');
@@ -0,0 +1,47 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Card fingerprint column. Mirrors SQLite migration 0060. `card_fingerprint`
-- is the Stripe card fingerprint (stable per card number) recorded when a user
-- clears card verification — the card sibling of `phone`, indexed like it so
-- admin tooling can find the accounts that verified with a given card. The card
-- itself never touches our DB, only Stripe's fingerprint for it.
--
-- Idempotent: the column add uses _puter_add_col (defined in mig_1, which
-- leaves it resident for later migrations); the index add is guarded against
-- INFORMATION_SCHEMA.STATISTICS so the directory replays safely.
CALL _puter_add_col('user', 'card_fingerprint', '`card_fingerprint` varchar(128) DEFAULT NULL');
DROP PROCEDURE IF EXISTS _puter_add_user_card_fingerprint_index;
DELIMITER //
CREATE PROCEDURE _puter_add_user_card_fingerprint_index()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user'
AND INDEX_NAME = 'idx_user_card_fingerprint'
) THEN
ALTER TABLE `user` ADD INDEX `idx_user_card_fingerprint` (`card_fingerprint`);
END IF;
END//
DELIMITER ;
CALL _puter_add_user_card_fingerprint_index();
DROP PROCEDURE IF EXISTS _puter_add_user_card_fingerprint_index;
@@ -0,0 +1,724 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
CREATE TABLE IF NOT EXISTS "user" (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
uuid varchar(36) NOT NULL UNIQUE,
username varchar(50) UNIQUE,
email varchar(256),
password varchar(225),
free_storage bigint,
max_subdomains integer,
taskbar_items text,
desktop_uuid varchar(36),
appdata_uuid varchar(36),
documents_uuid varchar(36),
pictures_uuid varchar(36),
videos_uuid varchar(36),
trash_uuid varchar(36),
trash_id integer,
appdata_id integer,
desktop_id integer,
documents_id integer,
pictures_id integer,
videos_id integer,
referrer varchar(64),
desktop_bg_url text,
desktop_bg_color varchar(20),
desktop_bg_fit varchar(16),
pass_recovery_token varchar(36),
requires_email_confirmation boolean NOT NULL DEFAULT FALSE,
email_confirm_code varchar(8),
email_confirm_token varchar(36),
email_confirmed boolean NOT NULL DEFAULT FALSE,
dev_first_name varchar(100),
dev_last_name varchar(100),
dev_paypal varchar(100),
dev_approved_for_incentive_program boolean DEFAULT FALSE,
dev_joined_incentive_program boolean DEFAULT FALSE,
suspended boolean,
unsubscribed boolean NOT NULL DEFAULT FALSE,
"timestamp" timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
last_activity_ts timestamp,
referral_code varchar(16) UNIQUE,
referred_by integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
unconfirmed_change_email varchar(256),
change_email_confirm_token varchar(256),
otp_secret text,
otp_enabled boolean DEFAULT FALSE,
otp_recovery_codes text,
stripe_customer_id varchar(40),
public_uuid varchar(36),
public_id integer,
clean_email varchar(256),
audit_metadata jsonb,
signup_ip varchar(45),
signup_ip_forwarded varchar(45),
signup_user_agent varchar(512),
signup_origin varchar(255),
signup_server varchar(255),
metadata jsonb DEFAULT '{}'::jsonb,
reputation smallint DEFAULT 100
);
CREATE INDEX IF NOT EXISTS idx_user_email ON "user" (email);
CREATE INDEX IF NOT EXISTS idx_user_pass_recovery_token ON "user" (pass_recovery_token);
CREATE INDEX IF NOT EXISTS idx_user_referrer ON "user" (referrer);
CREATE INDEX IF NOT EXISTS idx_user_email_confirm_token ON "user" (email_confirm_token);
CREATE INDEX IF NOT EXISTS idx_user_last_activity_ts ON "user" (last_activity_ts);
CREATE INDEX IF NOT EXISTS idx_user_referred_by ON "user" (referred_by);
CREATE INDEX IF NOT EXISTS idx_user_stripe_customer_id ON "user" (stripe_customer_id);
CREATE INDEX IF NOT EXISTS idx_user_clean_email ON "user" (clean_email);
CREATE INDEX IF NOT EXISTS idx_user_signup_ip ON "user" (signup_ip);
CREATE INDEX IF NOT EXISTS idx_user_signup_ip_forwarded ON "user" (signup_ip_forwarded);
CREATE INDEX IF NOT EXISTS idx_user_signup_user_agent ON "user" (signup_user_agent);
CREATE INDEX IF NOT EXISTS idx_user_signup_origin ON "user" (signup_origin);
CREATE INDEX IF NOT EXISTS idx_user_signup_server ON "user" (signup_server);
CREATE TABLE IF NOT EXISTS apps (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
uid varchar(40) NOT NULL UNIQUE,
owner_user_id integer REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
icon text,
name varchar(100) NOT NULL UNIQUE,
title varchar(100) NOT NULL,
description text,
godmode boolean DEFAULT FALSE,
maximize_on_start boolean DEFAULT FALSE,
index_url text NOT NULL,
approved_for_listing boolean DEFAULT FALSE,
approved_for_opening_items boolean DEFAULT FALSE,
approved_for_incentive_program boolean DEFAULT FALSE,
"timestamp" timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
last_review timestamp,
tags varchar(255),
app_owner integer REFERENCES apps (id) ON DELETE SET NULL ON UPDATE CASCADE,
background boolean DEFAULT FALSE,
metadata jsonb,
protected boolean DEFAULT FALSE,
is_private boolean DEFAULT FALSE
);
CREATE INDEX IF NOT EXISTS idx_apps_owner_user_id ON apps (owner_user_id);
CREATE INDEX IF NOT EXISTS idx_apps_app_owner ON apps (app_owner);
CREATE INDEX IF NOT EXISTS idx_apps_owner_timestamp ON apps (owner_user_id, "timestamp" DESC);
CREATE INDEX IF NOT EXISTS idx_apps_listing_timestamp ON apps (approved_for_listing, "timestamp" DESC);
CREATE TABLE IF NOT EXISTS "group" (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
uid varchar(40) UNIQUE,
owner_user_id integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
extra jsonb,
metadata jsonb,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_group_owner_user_id ON "group" (owner_user_id);
CREATE TABLE IF NOT EXISTS app_filetype_association (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
app_id integer NOT NULL REFERENCES apps (id) ON DELETE CASCADE ON UPDATE CASCADE,
type varchar(60) NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_app_filetype_association_app_id ON app_filetype_association (app_id);
CREATE INDEX IF NOT EXISTS idx_app_filetype_association_type ON app_filetype_association (type);
CREATE TABLE IF NOT EXISTS app_opens (
_id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
app_uid varchar(40) NOT NULL REFERENCES apps (uid) ON DELETE CASCADE ON UPDATE CASCADE,
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
ts integer NOT NULL,
human_ts timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_app_opens_user_id ON app_opens (user_id);
CREATE INDEX IF NOT EXISTS idx_app_opens_app_uid ON app_opens (app_uid);
CREATE INDEX IF NOT EXISTS idx_app_opens_uid_ts ON app_opens (app_uid, ts);
CREATE INDEX IF NOT EXISTS idx_app_opens_app_user ON app_opens (app_uid, user_id);
CREATE TABLE IF NOT EXISTS fsentries (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
uuid varchar(36) NOT NULL UNIQUE,
bucket varchar(50),
bucket_region varchar(30),
public_token varchar(36) UNIQUE,
file_request_token varchar(36) UNIQUE,
is_shortcut boolean DEFAULT FALSE,
shortcut_to integer REFERENCES fsentries (id) ON DELETE SET NULL ON UPDATE CASCADE,
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
parent_id integer REFERENCES fsentries (id) ON DELETE CASCADE ON UPDATE CASCADE,
associated_app_id integer REFERENCES apps (id) ON DELETE SET NULL ON UPDATE CASCADE,
is_dir boolean DEFAULT FALSE,
layout varchar(30),
sort_by varchar(20),
sort_order varchar(10),
is_public boolean,
thumbnail text,
immutable boolean NOT NULL DEFAULT FALSE,
name varchar(767) NOT NULL,
metadata text,
modified integer NOT NULL,
created integer,
accessed integer,
size bigint,
symlink_path varchar(260),
is_symlink boolean DEFAULT FALSE,
parent_uid varchar(36),
path varchar(4096),
CONSTRAINT fsentries_parent_name_unique UNIQUE (parent_id, name)
);
CREATE INDEX IF NOT EXISTS idx_fsentries_name ON fsentries (name);
CREATE INDEX IF NOT EXISTS idx_fsentries_modified ON fsentries (modified);
CREATE INDEX IF NOT EXISTS idx_fsentries_parent_id ON fsentries (parent_id);
CREATE INDEX IF NOT EXISTS idx_fsentries_is_dir ON fsentries (is_dir);
CREATE INDEX IF NOT EXISTS idx_fsentries_user_id ON fsentries (user_id);
CREATE INDEX IF NOT EXISTS idx_fsentries_shortcut_to ON fsentries (shortcut_to);
CREATE INDEX IF NOT EXISTS idx_fsentries_associated_app_id ON fsentries (associated_app_id);
CREATE INDEX IF NOT EXISTS idx_fsentries_bucket ON fsentries (bucket);
CREATE INDEX IF NOT EXISTS idx_fsentries_bucket_region ON fsentries (bucket_region);
CREATE INDEX IF NOT EXISTS idx_fsentries_parent_uid ON fsentries (parent_uid);
CREATE INDEX IF NOT EXISTS idx_fsentries_path ON fsentries (path);
CREATE INDEX IF NOT EXISTS idx_fsentries_accessed ON fsentries (accessed);
CREATE INDEX IF NOT EXISTS idx_fsentries_user_parent_name ON fsentries (user_id, parent_uid, name);
CREATE INDEX IF NOT EXISTS idx_fsentries_parent_uid_name ON fsentries (parent_uid, name);
CREATE TABLE IF NOT EXISTS fsentry_versions (
id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
fsentry_id integer NOT NULL REFERENCES fsentries (id) ON DELETE CASCADE ON UPDATE CASCADE,
fsentry_uuid varchar(36) NOT NULL,
version_id varchar(60) NOT NULL,
user_id integer REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
message text,
ts_epoch integer
);
CREATE INDEX IF NOT EXISTS idx_fsentry_versions_fsentry_id ON fsentry_versions (fsentry_id);
CREATE INDEX IF NOT EXISTS idx_fsentry_versions_fsentry_uuid ON fsentry_versions (fsentry_uuid);
CREATE INDEX IF NOT EXISTS idx_fsentry_versions_user_id ON fsentry_versions (user_id);
CREATE TABLE IF NOT EXISTS subdomains (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
uuid varchar(40) UNIQUE,
subdomain varchar(64) NOT NULL UNIQUE,
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
root_dir_id integer REFERENCES fsentries (id) ON DELETE SET NULL ON UPDATE CASCADE,
associated_app_id integer REFERENCES apps (id) ON DELETE CASCADE ON UPDATE CASCADE,
ts timestamp DEFAULT CURRENT_TIMESTAMP,
app_owner integer REFERENCES apps (id) ON DELETE SET NULL ON UPDATE CASCADE,
protected boolean DEFAULT FALSE,
domain varchar(265),
database_id varchar(40),
preamble_version varchar(64)
);
CREATE INDEX IF NOT EXISTS idx_subdomains_user_id ON subdomains (user_id);
CREATE INDEX IF NOT EXISTS idx_subdomains_root_dir_id ON subdomains (root_dir_id);
CREATE INDEX IF NOT EXISTS idx_subdomains_associated_app_id ON subdomains (associated_app_id);
CREATE INDEX IF NOT EXISTS idx_subdomains_app_owner ON subdomains (app_owner);
CREATE INDEX IF NOT EXISTS idx_subdomains_domain ON subdomains (domain);
CREATE INDEX IF NOT EXISTS idx_subdomains_root_user ON subdomains (root_dir_id, user_id);
CREATE INDEX IF NOT EXISTS idx_subdomains_app_user ON subdomains (associated_app_id, user_id);
CREATE TABLE IF NOT EXISTS kv (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
app varchar(40),
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
kkey_hash bigint NOT NULL,
kkey text NOT NULL,
value text,
migrated boolean DEFAULT FALSE,
"expireAt" timestamp,
CONSTRAINT kv_app_user_kkey_hash_unique UNIQUE (app, user_id, kkey_hash)
);
CREATE INDEX IF NOT EXISTS idx_kv_app ON kv (app);
CREATE INDEX IF NOT EXISTS idx_kv_user_id ON kv (user_id);
CREATE INDEX IF NOT EXISTS idx_kv_kkey_hash ON kv (kkey_hash);
CREATE TABLE IF NOT EXISTS sessions (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
uuid varchar(40) NOT NULL,
meta jsonb,
created_at bigint DEFAULT 0,
last_activity bigint DEFAULT 0,
kind varchar(32) NOT NULL DEFAULT 'web'
CHECK (kind IN ('web', 'app', 'access_token', 'asset', 'worker')),
label varchar(255),
parent_session_id varchar(64),
last_ip varchar(64),
last_user_agent varchar(512),
revoked_at bigint,
expires_at bigint,
app_uid varchar(64),
legacy_token_uid varchar(64),
created_via varchar(32),
auth_id varchar(64),
access_token_uid varchar(64)
);
CREATE INDEX IF NOT EXISTS idx_sessions_user_id ON sessions (user_id);
CREATE INDEX IF NOT EXISTS idx_sessions_uuid ON sessions (uuid);
CREATE INDEX IF NOT EXISTS idx_sessions_user_revoked ON sessions (user_id, revoked_at);
CREATE INDEX IF NOT EXISTS idx_sessions_parent ON sessions (parent_session_id);
CREATE UNIQUE INDEX IF NOT EXISTS idx_sessions_user_app_active
ON sessions (user_id, app_uid)
WHERE kind = 'app' AND revoked_at IS NULL;
CREATE UNIQUE INDEX IF NOT EXISTS idx_sessions_legacy_token_active
ON sessions (legacy_token_uid)
WHERE legacy_token_uid IS NOT NULL AND revoked_at IS NULL;
CREATE INDEX IF NOT EXISTS idx_sessions_kind_user ON sessions (kind, user_id);
CREATE INDEX IF NOT EXISTS idx_sessions_access_token_uid
ON sessions (access_token_uid)
WHERE access_token_uid IS NOT NULL;
CREATE UNIQUE INDEX IF NOT EXISTS idx_sessions_user_worker_active
ON sessions (user_id, COALESCE(app_uid, ''), (meta #>> ARRAY['worker_name']))
WHERE kind = 'worker' AND revoked_at IS NULL;
CREATE TABLE IF NOT EXISTS user_to_app_permissions (
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
app_id integer NOT NULL REFERENCES apps (id) ON DELETE CASCADE ON UPDATE CASCADE,
permission varchar(255) NOT NULL,
extra jsonb,
dt timestamp DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (user_id, app_id, permission)
);
CREATE INDEX IF NOT EXISTS idx_utap_user_permission ON user_to_app_permissions (user_id, permission);
CREATE INDEX IF NOT EXISTS idx_utap_app_permission ON user_to_app_permissions (app_id, permission);
CREATE TABLE IF NOT EXISTS user_to_group_permissions (
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
group_id integer NOT NULL REFERENCES "group" (id) ON DELETE CASCADE ON UPDATE CASCADE,
permission varchar(255) NOT NULL,
extra jsonb,
PRIMARY KEY (user_id, group_id, permission)
);
CREATE INDEX IF NOT EXISTS idx_user_to_group_permissions_group_id ON user_to_group_permissions (group_id);
CREATE TABLE IF NOT EXISTS user_to_user_permissions (
issuer_user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
holder_user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
permission varchar(255) NOT NULL,
extra jsonb,
PRIMARY KEY (issuer_user_id, holder_user_id, permission)
);
CREATE INDEX IF NOT EXISTS idx_user_to_user_permissions_holder_user_id ON user_to_user_permissions (holder_user_id);
CREATE TABLE IF NOT EXISTS dev_to_app_permissions (
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
app_id integer NOT NULL REFERENCES apps (id) ON DELETE CASCADE ON UPDATE CASCADE,
permission varchar(255) NOT NULL,
extra jsonb,
PRIMARY KEY (user_id, app_id, permission)
);
CREATE INDEX IF NOT EXISTS idx_dev_to_app_permissions_app_id ON dev_to_app_permissions (app_id);
CREATE INDEX IF NOT EXISTS idx_dev_to_app_permissions_permission ON dev_to_app_permissions (permission);
CREATE TABLE IF NOT EXISTS access_token_permissions (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
token_uid varchar(40) NOT NULL,
authorizer_user_id integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
authorizer_app_id integer REFERENCES apps (id) ON DELETE SET NULL ON UPDATE CASCADE,
permission varchar(255) NOT NULL,
extra jsonb,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_access_token_permissions_token_uid ON access_token_permissions (token_uid);
CREATE TABLE IF NOT EXISTS old_app_names (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
app_uid varchar(40) NOT NULL REFERENCES apps (uid) ON DELETE CASCADE,
name varchar(255) NOT NULL,
"timestamp" timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT unique_old_app_names_app_uid_name UNIQUE (app_uid, name)
);
CREATE INDEX IF NOT EXISTS idx_old_app_names_app_uid ON old_app_names (app_uid);
CREATE INDEX IF NOT EXISTS idx_old_app_names_name ON old_app_names (name);
CREATE TABLE IF NOT EXISTS jct_user_group (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
group_id integer NOT NULL REFERENCES "group" (id) ON DELETE CASCADE ON UPDATE CASCADE,
extra jsonb,
metadata jsonb,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_jct_user_group_user_id ON jct_user_group (user_id);
CREATE INDEX IF NOT EXISTS idx_jct_user_group_group_id ON jct_user_group (group_id);
CREATE TABLE IF NOT EXISTS notification (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
uid varchar(40) UNIQUE,
value jsonb NOT NULL,
acknowledged bigint,
shown bigint,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_notification_user_id ON notification (user_id);
CREATE TABLE IF NOT EXISTS share (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
uid varchar(40) UNIQUE,
issuer_user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
recipient_email varchar(255) NOT NULL,
data jsonb,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_share_issuer_user_id ON share (issuer_user_id);
CREATE INDEX IF NOT EXISTS idx_share_recipient_email ON share (recipient_email);
CREATE TABLE IF NOT EXISTS feedback (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
message text,
ts timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_feedback_user_id ON feedback (user_id);
CREATE TABLE IF NOT EXISTS thread (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
uid varchar(40) NOT NULL UNIQUE,
parent_uid varchar(40) REFERENCES thread (uid) ON DELETE CASCADE ON UPDATE CASCADE,
owner_user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
"schema" text,
text text NOT NULL,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_thread_parent_uid ON thread (parent_uid);
CREATE INDEX IF NOT EXISTS idx_thread_owner_user_id ON thread (owner_user_id);
CREATE TABLE IF NOT EXISTS user_comments (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
uid varchar(40) NOT NULL UNIQUE,
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
metadata jsonb,
text text NOT NULL,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_user_comments_user_id ON user_comments (user_id);
CREATE INDEX IF NOT EXISTS idx_user_comments_uid ON user_comments (uid);
CREATE TABLE IF NOT EXISTS user_fsentry_comments (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_comment_id integer NOT NULL REFERENCES user_comments (id) ON DELETE CASCADE ON UPDATE CASCADE,
fsentry_id integer NOT NULL REFERENCES fsentries (id) ON DELETE CASCADE ON UPDATE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_user_fsentry_comments_user_comment_id ON user_fsentry_comments (user_comment_id);
CREATE INDEX IF NOT EXISTS idx_user_fsentry_comments_fsentry_id ON user_fsentry_comments (fsentry_id);
CREATE TABLE IF NOT EXISTS user_fsentry_version_comments (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_comment_id integer NOT NULL REFERENCES user_comments (id) ON DELETE CASCADE ON UPDATE CASCADE,
fsentry_version_id integer NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_user_fsentry_version_comments_user_comment_id ON user_fsentry_version_comments (user_comment_id);
CREATE INDEX IF NOT EXISTS idx_user_fsentry_version_comments_version_id ON user_fsentry_version_comments (fsentry_version_id);
CREATE TABLE IF NOT EXISTS user_group_comments (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_comment_id integer NOT NULL REFERENCES user_comments (id) ON DELETE CASCADE ON UPDATE CASCADE,
group_id integer NOT NULL REFERENCES "group" (id) ON DELETE CASCADE ON UPDATE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_user_group_comments_user_comment_id ON user_group_comments (user_comment_id);
CREATE INDEX IF NOT EXISTS idx_user_group_comments_group_id ON user_group_comments (group_id);
CREATE TABLE IF NOT EXISTS user_user_comments (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_comment_id integer NOT NULL REFERENCES user_comments (id) ON DELETE CASCADE ON UPDATE CASCADE,
commented_user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_user_user_comments_user_comment_id ON user_user_comments (user_comment_id);
CREATE INDEX IF NOT EXISTS idx_user_user_comments_commented_user_id ON user_user_comments (commented_user_id);
CREATE TABLE IF NOT EXISTS user_oidc_providers (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
provider varchar(64) NOT NULL,
provider_sub varchar(255) NOT NULL,
refresh_token text,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_user_oidc_providers_user_id ON user_oidc_providers (user_id);
CREATE INDEX IF NOT EXISTS idx_user_oidc_providers_provider ON user_oidc_providers (provider);
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_oidc_providers_provider_sub_unique ON user_oidc_providers (provider, provider_sub);
CREATE TABLE IF NOT EXISTS app_update_audit (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
app_id integer REFERENCES apps (id) ON DELETE SET NULL ON UPDATE CASCADE,
app_id_keep integer NOT NULL,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
old_name varchar(50),
new_name varchar(50),
reason varchar(255)
);
CREATE INDEX IF NOT EXISTS idx_app_update_audit_app_id ON app_update_audit (app_id);
CREATE TABLE IF NOT EXISTS user_update_audit (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
user_id_keep integer NOT NULL,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
old_email varchar(256),
new_email varchar(256),
old_username varchar(50),
new_username varchar(50),
reason varchar(255)
);
CREATE INDEX IF NOT EXISTS idx_user_update_audit_user_id ON user_update_audit (user_id);
CREATE TABLE IF NOT EXISTS storage_audit (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
user_id_keep integer NOT NULL,
is_subtract boolean NOT NULL DEFAULT FALSE,
amount bigint NOT NULL,
field_a varchar(16),
field_b varchar(16),
reason varchar(255),
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_storage_audit_user_id ON storage_audit (user_id);
CREATE TABLE IF NOT EXISTS audit_user_to_app_permissions (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
user_id_keep integer NOT NULL,
app_id integer REFERENCES apps (id) ON DELETE SET NULL ON UPDATE CASCADE,
app_id_keep integer NOT NULL,
permission varchar(255) NOT NULL,
extra jsonb,
action varchar(16),
reason varchar(255),
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_audit_user_to_app_permissions_user_id ON audit_user_to_app_permissions (user_id);
CREATE INDEX IF NOT EXISTS idx_audit_user_to_app_permissions_app_id ON audit_user_to_app_permissions (app_id);
CREATE TABLE IF NOT EXISTS audit_dev_to_app_permissions (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
user_id_keep integer NOT NULL,
app_id integer REFERENCES apps (id) ON DELETE SET NULL ON UPDATE CASCADE,
app_id_keep integer NOT NULL,
permission varchar(255) NOT NULL,
extra jsonb,
action varchar(16),
reason varchar(255),
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_audit_dev_to_app_permissions_user_id ON audit_dev_to_app_permissions (user_id);
CREATE INDEX IF NOT EXISTS idx_audit_dev_to_app_permissions_app_id ON audit_dev_to_app_permissions (app_id);
CREATE TABLE IF NOT EXISTS audit_user_to_group_permissions (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
user_id_keep integer NOT NULL,
group_id integer REFERENCES "group" (id) ON DELETE SET NULL ON UPDATE CASCADE,
group_id_keep integer NOT NULL,
permission varchar(255) NOT NULL,
extra jsonb,
action varchar(255),
reason varchar(255),
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_audit_user_to_group_permissions_user_id ON audit_user_to_group_permissions (user_id);
CREATE INDEX IF NOT EXISTS idx_audit_user_to_group_permissions_group_id ON audit_user_to_group_permissions (group_id);
CREATE TABLE IF NOT EXISTS audit_user_to_user_permissions (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
issuer_user_id integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
issuer_user_id_keep integer NOT NULL,
holder_user_id integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
holder_user_id_keep integer NOT NULL,
permission varchar(255) NOT NULL,
extra jsonb,
action varchar(16),
reason varchar(255),
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_audit_user_to_user_permissions_issuer_user_id ON audit_user_to_user_permissions (issuer_user_id);
CREATE INDEX IF NOT EXISTS idx_audit_user_to_user_permissions_holder_user_id ON audit_user_to_user_permissions (holder_user_id);
CREATE TABLE IF NOT EXISTS ai_usage (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id integer NOT NULL REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
app_id integer REFERENCES apps (id) ON DELETE SET NULL ON UPDATE CASCADE,
service_name varchar(64),
model_name varchar(128),
price_modifier varchar(40),
cost integer,
value_uint_1 integer,
value_uint_2 integer,
value_uint_3 integer,
value_uint_4 integer,
value_uint_5 integer,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_ai_usage_app_id ON ai_usage (app_id);
CREATE INDEX IF NOT EXISTS idx_ai_usage_service_name ON ai_usage (service_name);
CREATE INDEX IF NOT EXISTS idx_ai_usage_model_name ON ai_usage (model_name);
CREATE INDEX IF NOT EXISTS idx_ai_usage_price_modifier ON ai_usage (price_modifier);
CREATE INDEX IF NOT EXISTS idx_ai_usage_created_at ON ai_usage (created_at);
CREATE INDEX IF NOT EXISTS idx_ai_usage_user_timestamp ON ai_usage (user_id, created_at);
CREATE TABLE IF NOT EXISTS general_analytics (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
uid varchar(40) NOT NULL,
trace_id varchar(40),
user_id integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
user_id_keep integer,
app_id integer REFERENCES apps (id) ON DELETE SET NULL ON UPDATE CASCADE,
app_id_keep integer,
server_id varchar(40),
actor_type varchar(40),
tags jsonb,
fields jsonb
);
CREATE INDEX IF NOT EXISTS idx_general_analytics_user_id ON general_analytics (user_id);
CREATE INDEX IF NOT EXISTS idx_general_analytics_app_id ON general_analytics (app_id);
CREATE TABLE IF NOT EXISTS monthly_usage_counts (
year integer NOT NULL,
month integer NOT NULL,
service_type varchar(40) NOT NULL,
service_name varchar(40) NOT NULL,
actor_key varchar(255) NOT NULL,
pricing_category jsonb NOT NULL,
pricing_category_hash bytea NOT NULL,
"count" integer DEFAULT 0,
value_uint_1 integer,
value_uint_2 integer,
value_uint_3 integer,
PRIMARY KEY (year, month, service_type, service_name, actor_key, pricing_category_hash)
);
CREATE TABLE IF NOT EXISTS service_usage_monthly (
"key" varchar(255) NOT NULL,
year integer NOT NULL,
month integer NOT NULL,
user_id integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
app_id integer REFERENCES apps (id) ON DELETE SET NULL ON UPDATE CASCADE,
"count" integer NOT NULL,
extra jsonb,
PRIMARY KEY ("key", year, month)
);
CREATE INDEX IF NOT EXISTS idx_service_usage_monthly_user_id ON service_usage_monthly (user_id);
CREATE INDEX IF NOT EXISTS idx_service_usage_monthly_app_id ON service_usage_monthly (app_id);
CREATE TABLE IF NOT EXISTS per_user_credit (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id integer NOT NULL UNIQUE REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
amount bigint NOT NULL,
last_updated_at bigint NOT NULL
);
INSERT INTO "user" (id, uuid, username, metadata)
VALUES (1, '5d4adce0-a381-4982-9c02-6e2540026238', 'system', '{}'::jsonb)
ON CONFLICT (uuid) DO UPDATE SET username = EXCLUDED.username;
SELECT setval(pg_get_serial_sequence('"user"', 'id'), COALESCE((SELECT MAX(id) FROM "user"), 1), true);
INSERT INTO "group" (uid, owner_user_id, extra, metadata)
VALUES
('26bfb1fb-421f-45bc-9aa4-d81ea569e7a5', 1, '{"critical": true, "type": "default", "name": "system"}'::jsonb, '{"title": "System", "color": "#000000"}'::jsonb),
('ca342a5e-b13d-4dee-9048-58b11a57cc55', 1, '{"critical": true, "type": "default", "name": "admin"}'::jsonb, '{"title": "Admin", "color": "#a83232"}'::jsonb),
('78b1b1dd-c959-44d2-b02c-8735671f9997', 1, '{"critical": true, "type": "default", "name": "user"}'::jsonb, '{"title": "User", "color": "#3254a8"}'::jsonb),
('b7220104-7905-4985-b996-649fdcdb3c8f', 1, '{"critical": true, "type": "default", "name": "temp"}'::jsonb, '{"title": "Temp", "color": "#888888"}'::jsonb),
('3c2dfff7-d22a-41aa-a193-59a61dac4b64', 1, '{"type": "default", "name": "moderator"}'::jsonb, '{"title": "Moderator", "color": "#a432a8"}'::jsonb),
('5e8f251d-3382-4b0d-932c-7bb82f48652f', 1, '{"type": "default", "name": "developer"}'::jsonb, '{"title": "Developer", "color": "#32a852"}'::jsonb)
ON CONFLICT (uid) DO UPDATE SET
owner_user_id = EXCLUDED.owner_user_id,
extra = EXCLUDED.extra,
metadata = EXCLUDED.metadata;
INSERT INTO apps (
uid, owner_user_id, icon, name, title, description, index_url,
godmode, maximize_on_start, background,
approved_for_listing, approved_for_opening_items,
approved_for_incentive_program, tags, "timestamp"
)
VALUES
('app-3920851d-bda8-479b-9407-8517293c7d44', 1, NULL, 'pdf', 'PDF', '', 'https://pdf.puter.com', FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, 'productivity', '2020-01-01 00:00:00'),
('app-5584fbf7-ed69-41fc-99cd-85da21b1ef51', 1, NULL, 'camera', 'Camera', 'Camera in the browser.', 'https://online-camera.com', FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, NULL, '2020-01-01 00:00:00'),
('app-11edfba2-1ed3-4e22-8573-47e88fb87d70', 1, NULL, 'player', 'Player', 'A free video player app in the browser.', 'https://simple-player.puter.com', FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, NULL, '2020-01-01 00:00:00'),
('app-7bdca1a4-6373-4c98-ad97-03ff2d608ca1', 1, NULL, 'recorder', 'Recorder', 'Online voice recorder in the browser with cloud storage.', 'https://voice-recorder.com', FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, NULL, '2020-01-01 00:00:00'),
('app-d7e9471f-e441-4d72-a5ab-75e96573b76b', 1, NULL, 'music-player', 'Music Player', 'A free music player app in the browser.', 'https://player.puter.com', FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, NULL, '2026-05-10 00:00:00'),
('app-e3ac5486-da8c-42ad-8377-8728086e0980', 1, NULL, 'git', 'Git', 'Puter Git client', 'https://builtins.namespaces.puter.com/git', FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, 'productivity', '2020-01-01 00:00:00'),
('app-0b37f054-07d4-4627-8765-11bd23e889d4', 1, NULL, 'dev-center', 'Dev Center', 'This is the app that makes apps', 'https://builtins.namespaces.puter.com/dev-center', TRUE, TRUE, FALSE, TRUE, TRUE, FALSE, NULL, '2020-01-01 00:00:00'),
('app-fbbdb72b-ad08-4cb4-86a1-de0f27cf2e1e', 1, NULL, 'puter-linux', 'Puter Linux', 'Linux emulator for Puter', 'https://builtins.namespaces.puter.com/emulator', TRUE, FALSE, FALSE, TRUE, TRUE, FALSE, NULL, '2020-01-01 00:00:00'),
('app-838dfbc4-bf8b-48c2-b47b-c4adc77fab58', 1, NULL, 'editor', 'Editor', 'Text editor', 'https://online-notepad.com', TRUE, TRUE, FALSE, TRUE, TRUE, FALSE, 'productivity', '2020-01-01 00:00:00'),
('app-58282b08-990a-4906-95f7-fa37ff92452b', 1, NULL, 'draw', 'Draw', 'Image editor', 'https://draw.puter.com', TRUE, TRUE, FALSE, TRUE, TRUE, FALSE, 'graphics', '2020-01-01 00:00:00'),
('app-0bef044f-918f-4cbf-a0c0-b4a17ee81085', 1, NULL, 'about', 'About', 'About Puter', 'https://about.puter.com', TRUE, FALSE, FALSE, FALSE, TRUE, FALSE, NULL, '2020-01-01 00:00:00'),
('app-a2ae72a4-1ba3-4a29-b5c0-6de1be5cf178', 1, NULL, 'app-center', 'App Center', 'Discover apps for Puter', 'https://app-center.puter.com', TRUE, TRUE, FALSE, TRUE, TRUE, FALSE, NULL, '2020-01-01 00:00:00'),
('app-93005ce0-80d1-50d9-9b1e-9c453c375d56', 1, NULL, 'markus', 'Markus', 'Markdown editor', 'https://markus.puter.com', TRUE, TRUE, FALSE, TRUE, TRUE, FALSE, 'productivity', '2020-01-01 00:00:00'),
('app-6f79ef7b-52b7-4b31-91c6-fc07b62e9396', 1, NULL, 'code', 'Code', 'Code editor', 'https://code.puter.com', TRUE, TRUE, FALSE, TRUE, TRUE, FALSE, 'productivity', '2020-01-01 00:00:00'),
('app-862fc09e-c7b8-4c30-b5b4-47b3cc5f5232', 1, NULL, 'memos', 'Memos', 'Notes and memos', 'https://memos.puter.com', FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, 'productivity', '2020-01-01 00:00:00'),
('app-aeb8c03e-1144-4c57-bff5-3a1a7c17f9f0', 1, NULL, 'word-processor', 'Word Processor', 'Write documents', 'https://word-processor.puter.com', FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, 'productivity', '2020-01-01 00:00:00'),
('app-c10a1999-22f5-4644-9960-9aaac0d4934e', 1, NULL, 'spreadsheet', 'Spreadsheet', 'Work with spreadsheets', 'https://spreadsheet.puter.com', FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, 'productivity', '2020-01-01 00:00:00'),
('app-e9c1d58d-3d8d-4f0a-a85c-e383ef63bc29', 1, NULL, 'presentation', 'Presentation', 'Create presentations', 'https://presentation.puter.com', FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, 'productivity', '2020-01-01 00:00:00'),
('app-fb7d9e42-8207-4fa0-b680-f0239327097f', 1, NULL, 'pdf-editor', 'PDF Editor', 'Edit PDF documents', 'https://pdf-editor.puter.com', FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, 'productivity', '2020-01-01 00:00:00')
ON CONFLICT (uid) DO UPDATE SET
owner_user_id = EXCLUDED.owner_user_id,
name = EXCLUDED.name,
title = EXCLUDED.title,
description = EXCLUDED.description,
index_url = EXCLUDED.index_url,
godmode = EXCLUDED.godmode,
maximize_on_start = EXCLUDED.maximize_on_start,
background = EXCLUDED.background,
approved_for_listing = EXCLUDED.approved_for_listing,
approved_for_opening_items = EXCLUDED.approved_for_opening_items,
approved_for_incentive_program = EXCLUDED.approved_for_incentive_program,
tags = EXCLUDED.tags;
INSERT INTO user_to_group_permissions (user_id, group_id, permission, extra)
SELECT u.id, g.id, 'driver', '{}'::jsonb
FROM "user" u, "group" g
WHERE u.username = 'system'
AND g.uid = 'ca342a5e-b13d-4dee-9048-58b11a57cc55'
ON CONFLICT (user_id, group_id, permission) DO UPDATE SET extra = EXCLUDED.extra;
@@ -0,0 +1,26 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- SMS phone verification columns. Mirrors SQLite migration 0058. `phone` is the
-- E.164 number collected during verification (indexed like `email`);
-- `requires_phone_verification` gates account use for low-reputation signups
-- (not indexed, mirroring `requires_email_confirmation`).
-- Idempotent via IF NOT EXISTS.
ALTER TABLE "user" ADD COLUMN IF NOT EXISTS phone varchar(20);
ALTER TABLE "user" ADD COLUMN IF NOT EXISTS requires_phone_verification boolean NOT NULL DEFAULT FALSE;
CREATE INDEX IF NOT EXISTS idx_user_phone ON "user" (phone);
@@ -0,0 +1,24 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Credit-card verification column. Mirrors SQLite migration 0059.
-- `requires_card_verification` gates account use for low-reputation signups;
-- the card itself never touches our DB, so this is the only column (not
-- indexed, mirroring `requires_phone_verification`).
-- Idempotent via IF NOT EXISTS.
ALTER TABLE "user" ADD COLUMN IF NOT EXISTS requires_card_verification boolean NOT NULL DEFAULT FALSE;
@@ -0,0 +1,26 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Card fingerprint column. Mirrors SQLite migration 0060. `card_fingerprint`
-- is the Stripe card fingerprint (stable per card number) recorded when a user
-- clears card verification — the card sibling of `phone`, indexed like it so
-- admin tooling can find the accounts that verified with a given card. The card
-- itself never touches our DB, only Stripe's fingerprint for it.
-- Idempotent via IF NOT EXISTS.
ALTER TABLE "user" ADD COLUMN IF NOT EXISTS card_fingerprint varchar(128);
CREATE INDEX IF NOT EXISTS idx_user_card_fingerprint ON "user" (card_fingerprint);
@@ -0,0 +1,21 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Abuse v2 reputation score (0–100+) recorded at signup. Mirrors the
-- `reputation smallint DEFAULT 100` column already present in the mysql and
-- postgres baseline migrations.
ALTER TABLE `user` ADD COLUMN `reputation` SMALLINT DEFAULT 100;
@@ -0,0 +1,25 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- SMS phone verification. `phone` holds the E.164 number collected during
-- verification (indexed like `email`). `requires_phone_verification` gates
-- account use until verified — the abuse v2 harness sets it for low-reputation
-- signups instead of blocking them. (Not indexed, mirroring
-- `requires_email_confirmation`.)
ALTER TABLE `user` ADD COLUMN `phone` varchar(20) DEFAULT NULL;
ALTER TABLE `user` ADD COLUMN `requires_phone_verification` tinyint(1) NOT NULL DEFAULT 0;
CREATE INDEX IF NOT EXISTS idx_user_phone ON `user` (`phone`);
@@ -0,0 +1,23 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Credit-card verification ($0 authorization handled by a payments
-- extension). `requires_card_verification` gates account use until verified —
-- the abuse v2 harness sets it for low-reputation signups instead of blocking
-- them. The card itself never touches our DB, so this is the only column.
-- (Not indexed, mirroring `requires_phone_verification`.)
ALTER TABLE `user` ADD COLUMN `requires_card_verification` tinyint(1) NOT NULL DEFAULT 0;
@@ -0,0 +1,25 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Card fingerprint column. `card_fingerprint` is the Stripe card fingerprint
-- (stable per card number) recorded when a user clears card verification — the
-- card sibling of `phone`. Indexed like `phone` so admin tooling can find the
-- accounts that verified with a given card. The card itself never touches our
-- DB, only Stripe's fingerprint for it. Server-only (not on the whoami
-- allowlist), so it never reaches clients.
ALTER TABLE `user` ADD COLUMN `card_fingerprint` varchar(128) DEFAULT NULL;
CREATE INDEX IF NOT EXISTS idx_user_card_fingerprint ON `user` (`card_fingerprint`);
@@ -0,0 +1,85 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { describe, expect, it } from 'vitest';
import { preparePostgresSql } from './preparePostgresSql.js';
describe('preparePostgresSql', () => {
it('converts placeholders and backtick identifiers outside SQL literals', () => {
const prepared = preparePostgresSql(
'SELECT `user`.`id` FROM `user` WHERE `email` = ? AND `username` = ?',
);
expect(prepared).toEqual({
text: 'SELECT "user"."id" FROM "user" WHERE "email" = $1 AND "username" = $2',
parameterCount: 2,
});
});
it('leaves question marks inside strings and comments untouched', () => {
const prepared = preparePostgresSql(`
SELECT '?' AS literal, ?
-- ? in a comment
FROM \`apps\`
WHERE \`description\` = 'is this ok?'
/* and ? in a block comment */
AND \`name\` = ?
`);
expect(prepared.text).toContain("SELECT '?' AS literal, $1");
expect(prepared.text).toContain('-- ? in a comment');
expect(prepared.text).toContain('"description" = \'is this ok?\'');
expect(prepared.text).toContain('/* and ? in a block comment */');
expect(prepared.text).toContain('"name" = $2');
expect(prepared.parameterCount).toBe(2);
});
it('escapes double quotes inside converted identifiers', () => {
const prepared = preparePostgresSql(
'SELECT `odd"name` FROM `odd``table` WHERE `id` = ?',
);
expect(prepared).toEqual({
text: 'SELECT "odd""name" FROM "odd`table" WHERE "id" = $1',
parameterCount: 1,
});
});
it('does not scan placeholders inside dollar-quoted strings', () => {
const prepared = preparePostgresSql(
"SELECT $$?$$, $tag$`not_ident` ?$tag$, `id` FROM `user` WHERE `id` = ?",
);
expect(prepared).toEqual({
text: 'SELECT $$?$$, $tag$`not_ident` ?$tag$, "id" FROM "user" WHERE "id" = $1',
parameterCount: 1,
});
});
it('does not treat Postgres JSON operators as comments', () => {
const prepared = preparePostgresSql(
"SELECT * FROM `sessions` WHERE `user_id` = ? AND `meta` #>> ARRAY['worker_name'] = ? AND `expires_at` > ?",
);
expect(prepared).toEqual({
text: 'SELECT * FROM "sessions" WHERE "user_id" = $1 AND "meta" #>> ARRAY[\'worker_name\'] = $2 AND "expires_at" > $3',
parameterCount: 3,
});
});
});
@@ -0,0 +1,169 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
export interface PreparedPostgresSql {
text: string;
parameterCount: number;
}
type ScannerState =
| { kind: 'normal' }
| { kind: 'singleQuote' }
| { kind: 'doubleQuote' }
| { kind: 'backtickIdentifier' }
| { kind: 'lineComment' }
| { kind: 'blockComment' }
| { kind: 'dollarQuote'; tag: string };
const isLineCommentStart = (sql: string, index: number): boolean => {
if (sql[index] !== '-' || sql[index + 1] !== '-') return false;
const after = sql[index + 2];
return after === undefined || /\s/u.test(after);
};
const readDollarQuoteTag = (sql: string, index: number): string | null => {
const match = /^\$[A-Za-z_][A-Za-z0-9_]*\$|^\$\$/u.exec(sql.slice(index));
return match?.[0] ?? null;
};
export const preparePostgresSql = (sql: string): PreparedPostgresSql => {
let state: ScannerState = { kind: 'normal' };
let parameterCount = 0;
let out = '';
for (let i = 0; i < sql.length; i++) {
const char = sql[i]!;
const next = sql[i + 1];
switch (state.kind) {
case 'normal': {
const dollarTag =
char === '$' ? readDollarQuoteTag(sql, i) : null;
if (dollarTag) {
out += dollarTag;
i += dollarTag.length - 1;
state = { kind: 'dollarQuote', tag: dollarTag };
break;
}
if (char === "'") {
out += char;
state = { kind: 'singleQuote' };
break;
}
if (char === '"') {
out += char;
state = { kind: 'doubleQuote' };
break;
}
if (char === '`') {
out += '"';
state = { kind: 'backtickIdentifier' };
break;
}
if (isLineCommentStart(sql, i)) {
out += char;
if (char === '-') {
out += next ?? '';
i += 1;
}
state = { kind: 'lineComment' };
break;
}
if (char === '/' && next === '*') {
out += '/*';
i += 1;
state = { kind: 'blockComment' };
break;
}
if (char === '?') {
parameterCount += 1;
out += `$${parameterCount}`;
break;
}
out += char;
break;
}
case 'singleQuote':
out += char;
if (char === "'" && next === "'") {
out += next;
i += 1;
} else if (char === '\\' && next !== undefined) {
out += next;
i += 1;
} else if (char === "'") {
state = { kind: 'normal' };
}
break;
case 'doubleQuote':
out += char;
if (char === '"' && next === '"') {
out += next;
i += 1;
} else if (char === '"') {
state = { kind: 'normal' };
}
break;
case 'backtickIdentifier':
if (char === '`' && next === '`') {
out += '`';
i += 1;
} else if (char === '`') {
out += '"';
state = { kind: 'normal' };
} else if (char === '"') {
out += '""';
} else {
out += char;
}
break;
case 'lineComment':
out += char;
if (char === '\n') {
state = { kind: 'normal' };
}
break;
case 'blockComment':
out += char;
if (char === '*' && next === '/') {
out += '/';
i += 1;
state = { kind: 'normal' };
}
break;
case 'dollarQuote':
if (sql.startsWith(state.tag, i)) {
out += state.tag;
i += state.tag.length - 1;
state = { kind: 'normal' };
} else {
out += char;
}
break;
}
}
return { text: out, parameterCount };
};
@@ -0,0 +1,120 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
type ScannerState =
| { kind: 'normal' }
| { kind: 'singleQuote' }
| { kind: 'doubleQuote' }
| { kind: 'lineComment' }
| { kind: 'blockComment' }
| { kind: 'dollarQuote'; tag: string };
const readDollarQuoteTag = (sql: string, index: number): string | null => {
const match = /^\$[A-Za-z_][A-Za-z0-9_]*\$|^\$\$/u.exec(sql.slice(index));
return match?.[0] ?? null;
};
export const splitPostgresStatements = (sql: string): string[] => {
const statements: string[] = [];
let state: ScannerState = { kind: 'normal' };
let start = 0;
for (let i = 0; i < sql.length; i++) {
const char = sql[i]!;
const next = sql[i + 1];
switch (state.kind) {
case 'normal': {
const dollarTag =
char === '$' ? readDollarQuoteTag(sql, i) : null;
if (dollarTag) {
i += dollarTag.length - 1;
state = { kind: 'dollarQuote', tag: dollarTag };
break;
}
if (char === "'") {
state = { kind: 'singleQuote' };
break;
}
if (char === '"') {
state = { kind: 'doubleQuote' };
break;
}
if (char === '-' && next === '-') {
i += 1;
state = { kind: 'lineComment' };
break;
}
if (char === '/' && next === '*') {
i += 1;
state = { kind: 'blockComment' };
break;
}
if (char === ';') {
const statement = sql.slice(start, i).trim();
if (statement !== '') statements.push(statement);
start = i + 1;
}
break;
}
case 'singleQuote':
if (char === "'" && next === "'") {
i += 1;
} else if (char === '\\' && next !== undefined) {
i += 1;
} else if (char === "'") {
state = { kind: 'normal' };
}
break;
case 'doubleQuote':
if (char === '"' && next === '"') {
i += 1;
} else if (char === '"') {
state = { kind: 'normal' };
}
break;
case 'lineComment':
if (char === '\n') {
state = { kind: 'normal' };
}
break;
case 'blockComment':
if (char === '*' && next === '/') {
i += 1;
state = { kind: 'normal' };
}
break;
case 'dollarQuote':
if (sql.startsWith(state.tag, i)) {
i += state.tag.length - 1;
state = { kind: 'normal' };
}
break;
}
}
const finalStatement = sql.slice(start).trim();
if (finalStatement !== '') statements.push(finalStatement);
return statements;
};
+192 -1
View File
@@ -14,6 +14,10 @@
* the SocketService fans out to user-scoped channels.
*/
import type {
Request as ExpressRequest,
Response as ExpressResponse,
} from 'express';
import { Actor } from '../../core';
import { FSEntry } from '../../stores/fs/FSEntry';
@@ -126,7 +130,20 @@ export type EventMap = {
req?: unknown;
data?: unknown;
abuse?: unknown;
custom?: unknown;
trail?: Array<string>;
/**
* Set by the abuse harness for flagged signups — the id under which the
* decision trail is persisted to KV (`abuse:trail:<id>`), shared back on
* the request for log / support correlation.
*/
trail_id?: string;
/** Device signals forwarded verbatim from the signup request body. */
fingerprint?: string | null;
dfp_telemetry_id?: string | null;
/** Set by the abuse harness — require SMS phone verification post-signup. */
requires_phone_verification?: boolean;
/** Set by the abuse harness — require card verification post-signup. */
requires_card_verification?: boolean;
[key: string]: unknown;
};
'puter.signup.success': {
@@ -135,6 +152,9 @@ export type EventMap = {
email: string;
username: string;
ip?: string | null;
fingerprint?: string | null;
/** True when the created account is a temp user (no email/password). */
is_temp?: boolean;
[key: string]: unknown;
};
'email.validate': {
@@ -154,12 +174,93 @@ export type EventMap = {
user_uid: string;
email: string;
};
// Phone-reuse cap is pure mechanism here — the abuse extension answers
// (emitted via `emitAndWait`) by counting how many OTHER accounts have
// already verified this number and flipping `allowed` to false when the
// cross-account limit is hit. No extension listening → `allowed` stays
// true and the send proceeds.
'puter.phone-verification.check': {
user_id: number;
user_uid: string;
phone: string;
// Client-supplied device fingerprint for this request, or null. Lets
// the abuse extension cap sends per device (across accounts) so a
// device farm can't dodge the per-account cap with fresh signups.
device_fingerprint: string | null;
allowed: boolean;
reason: string | null;
[key: string]: unknown;
};
// Fire-and-forget signal that a code was actually sent — the abuse
// extension bumps its per-number / per-account / per-device send-velocity
// counters off this. No-op with no extension listening.
'puter.phone-verification.sent': {
user_id: number;
user_uid: string;
phone: string;
device_fingerprint: string | null;
};
'user.phone-verified': {
user_id: number;
user_uid: string;
phone: string;
};
// Card verification is pure mechanism here — a payments extension fills
// these in (emitted via `emitAndWait`). `enabled` stays null when no
// extension is installed; the extension always sets it (true/false) so
// the endpoints can distinguish "disabled" from "not installed".
'puter.card-verification.setup': {
user_id: number;
user_uid: string;
ip?: string | null;
// Client-supplied device fingerprint for this request, or null. Lets
// the abuse extension cap card-verification setups per device (across
// accounts) before any Stripe SetupIntent is created.
device_fingerprint: string | null;
enabled: boolean | null;
// Set false by the extension to refuse this setup (e.g. the per-device
// setup-velocity cap); `reason` carries the opaque code. Stays true
// when allowed or with no extension listening.
allowed: boolean;
reason: string | null;
client_secret: string | null;
publishable_key: string | null;
[key: string]: unknown;
};
'puter.card-verification.confirm': {
user_id: number;
user_uid: string;
setup_intent_id: string;
enabled: boolean | null;
verified: boolean;
reason: string | null;
fingerprint: string | null;
funding: string | null;
country: string | null;
[key: string]: unknown;
};
'user.card-verified': {
user_id: number;
user_uid: string;
fingerprint: string | null;
funding: string | null;
country: string | null;
};
'user.username-changed': {
user_id: number;
old_username: string;
new_username: string;
};
'user.email-changed': { user_id: number; new_email: string };
// Fired after an account is torn down (self-serve, admin, or temp-user
// logout cleanup). Listeners purge external state tied to the account —
// e.g. the marketplace extension cancels the user's Stripe subscriptions.
// The row is already gone by emit time, so identifiers ride the payload.
'user.delete': {
user_id: number;
user_uuid?: string;
stripe_customer_id?: string | null;
};
// ---- Filesystem ----
'fs.copy.node': {
@@ -211,6 +312,8 @@ export type EventMap = {
entry: unknown;
host: string;
requestPath: string;
requestUrl?: string;
requestHash?: string;
mime: string;
};
@@ -257,6 +360,94 @@ export type EventMap = {
// SocketService re-emits each fanout-eligible event under
// `sent-to-user.<wireName>` so per-user channels can subscribe by wire name.
[K in `sent-to-user.${string}`]: { user_id: number; response: unknown };
} & {
// Generic per-driver-method lifecycle, emitted by DriverController for
// EVERY driver call under a key scoped to the interface + method:
// `driver.<iface>.<method>.before|after|error|reject`. Wildcard
// subscribers can listen to `driver.*` for everything, `driver.<iface>.*`
// for one interface, or the exact key for one method. The `.before` phase
// is emitted via `emitAndWait`, so a listener may set `allow = false`
// (with an optional `rejectReason`) to veto the call before it runs — a
// vetoed call emits `.reject` instead of running.
[K in `driver.${string}`]: DriverMethodLifecycleEvent;
} & {
// Generic per-route-endpoint lifecycle, emitted by the route materializer
// for EVERY non-middleware route under a key scoped to the HTTP method +
// normalized path: `route.<method>.<path>.before|after|error|reject`. Same
// wildcard + veto semantics as the driver lifecycle above.
[K in `route.${string}`]: RouteLifecycleEvent;
};
/**
* Phase of a request/method lifecycle. `reject` is emitted when a `before`
* listener vetoes the call (sets `allow = false`); the call never runs and no
* `after`/`error` follows.
*/
export type LifecyclePhase = 'before' | 'after' | 'error' | 'reject';
/**
* Payload for `driver.<iface>.<method>.<phase>` events.
*
* One shape across all phases; read `phase` (or the key suffix) to
* branch. `allow`/`rejectReason` are only meaningful on the `before` phase
* (emitted via `emitAndWait`).
*/
export type DriverMethodLifecycleEvent = {
phase: LifecyclePhase;
iface: string;
method: string;
/** Resolved concrete driver name. */
driver: string;
/** Full actor object, if the request is authenticated. */
actor?: Actor;
/** Stable actor id (see `actorUid`), if the request is authenticated. */
actorUid?: string;
/** Call arguments. Present on every phase. */
args?: unknown;
/** Return value. Present on `after`. */
result?: unknown;
/** Thrown error. Present on `error`. */
error?: unknown;
/** Wall-clock duration of the invocation. Present on `after`/`error`. */
durationMs?: number;
/** Veto channel for `before`: set `false` to block the call. */
allow?: boolean;
/** Optional human-readable reason surfaced to the caller when vetoed. */
rejectReason?: string;
};
/**
* Payload for `route.<method>.<path>.<phase>` events. Mirrors
* {@link DriverMethodLifecycleEvent} for HTTP endpoints.
*/
export type RouteLifecycleEvent = {
phase: LifecyclePhase;
/** HTTP method, lowercased (`get`, `post`, ...). */
method: string;
/** Full route path including the controller prefix. */
path: string;
/**
* The live express request/response for this call. Present on every phase.
* On `before` a listener can read the parsed body/headers or write its own
* response; on terminal phases they're useful for logging. These are real
* in-process objects — never serialize or forward them across nodes.
*/
req: ExpressRequest;
res: ExpressResponse;
/** Full actor object, if the request is authenticated. */
actor?: Actor;
/** Stable actor id (see `actorUid`), if the request is authenticated. */
actorUid?: string;
/** Response status code. Present on `after`/`error`. */
statusCode?: number;
/** Wall-clock duration from `before` to terminal phase. */
durationMs?: number;
/** Set when the request did not complete normally (abort / >=500). */
error?: unknown;
/** Veto channel for `before`: set `false` to block the request. */
allow?: boolean;
/** Optional human-readable reason surfaced to the caller when vetoed. */
rejectReason?: string;
};
export type EventKey = keyof EventMap & string;
+2
View File
@@ -24,6 +24,7 @@ import { EventClient } from './event/EventClient';
import { DDBClient } from './dynamodb/DDBClient';
import { RedisClient } from './redis/RedisClient';
import { S3Client } from './s3/S3Client';
import { PreludeClient } from './prelude/PreludeClient';
import type { IPuterClientRegistry } from './types';
export const puterClients = {
@@ -34,4 +35,5 @@ export const puterClients = {
dynamo: DDBClient,
redis: RedisClient,
s3: S3Client,
prelude: PreludeClient,
} satisfies IPuterClientRegistry;
@@ -0,0 +1,173 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { PreludeClient } from './PreludeClient';
import type { IConfig } from '../../types';
const makeClient = (apiKey?: string) =>
new PreludeClient({
prelude: apiKey ? { apiKey } : undefined,
} as unknown as IConfig);
const okJson = (body: unknown) =>
({
ok: true,
status: 200,
json: async () => body,
}) as unknown as Response;
describe('PreludeClient', () => {
let fetchMock: ReturnType<typeof vi.fn>;
beforeEach(() => {
fetchMock = vi.fn();
vi.stubGlobal('fetch', fetchMock);
});
afterEach(() => {
vi.unstubAllGlobals();
});
it('isConfigured reflects whether an apiKey is set', () => {
expect(makeClient('sk_test').isConfigured()).toBe(true);
expect(makeClient().isConfigured()).toBe(false);
});
describe('isCountrySupported (€0.07 cap)', () => {
const client = makeClient('sk_test');
it('allows revenue markets up to the cap (incl. the priciest)', () => {
expect(client.isCountrySupported('US')).toBe(true); // €0.0043
expect(client.isCountrySupported('DE')).toBe(true); // €0.0598
expect(client.isCountrySupported('SA')).toBe(true); // €0.0638
expect(client.isCountrySupported('us')).toBe(true); // case-insensitive
});
it('rejects countries above the cap, with no SMS, or unknown', () => {
expect(client.isCountrySupported('PK')).toBe(false); // €0.3548
expect(client.isCountrySupported('ID')).toBe(false); // €0.2430
expect(client.isCountrySupported('LI')).toBe(false); // null (no SMS)
expect(client.isCountrySupported('ZZ')).toBe(false); // unknown
expect(client.isCountrySupported(undefined)).toBe(false);
});
it('honors a configured maxSmsCostEur override', () => {
const strict = new PreludeClient({
prelude: { apiKey: 'sk', maxSmsCostEur: 0.01 },
} as unknown as IConfig);
expect(strict.isCountrySupported('US')).toBe(true); // €0.0043
expect(strict.isCountrySupported('DE')).toBe(false); // €0.0598 > 0.01
const loose = new PreludeClient({
prelude: { apiKey: 'sk', maxSmsCostEur: 0.5 },
} as unknown as IConfig);
expect(loose.isCountrySupported('PK')).toBe(true); // €0.3548 <= 0.5
});
});
it('createVerification POSTs the phone target + ip signal with bearer auth', async () => {
fetchMock.mockResolvedValue(
okJson({ id: 'vrf_1', status: 'success' }),
);
const client = makeClient('sk_test');
const res = await client.createVerification('+14155550123', {
ip: '203.0.113.7',
});
expect(res).toEqual({ id: 'vrf_1', status: 'success' });
const [url, init] = fetchMock.mock.calls[0];
expect(url).toBe('https://api.prelude.dev/v2/verification');
expect(init.method).toBe('POST');
expect(init.headers.Authorization).toBe('Bearer sk_test');
expect(JSON.parse(init.body)).toEqual({
target: { type: 'phone_number', value: '+14155550123' },
// Defaults to RCS (cheaper); Prelude falls back to SMS.
options: { code_size: 6, preferred_channel: 'rcs' },
signals: { ip: '203.0.113.7' },
});
});
it('forwards device_id and user_agent signals when supplied', async () => {
fetchMock.mockResolvedValue(okJson({ id: 'v', status: 'success' }));
const client = makeClient('sk_test');
await client.createVerification('+14155550123', {
ip: '203.0.113.7',
device_id: 'thumb_abc123',
user_agent: 'Mozilla/5.0',
});
const [, init] = fetchMock.mock.calls[0];
expect(JSON.parse(init.body).signals).toEqual({
ip: '203.0.113.7',
device_id: 'thumb_abc123',
user_agent: 'Mozilla/5.0',
});
});
it('omits the signals object entirely when none are supplied', async () => {
fetchMock.mockResolvedValue(okJson({ id: 'v', status: 'success' }));
const client = makeClient('sk_test');
await client.createVerification('+14155550123');
expect(JSON.parse(fetchMock.mock.calls[0][1].body)).not.toHaveProperty(
'signals',
);
});
it('includes a configured template_id + sender_id + preferred channel', async () => {
fetchMock.mockResolvedValue(okJson({ id: 'v', status: 'success' }));
const client = new PreludeClient({
prelude: {
apiKey: 'sk_test',
templateId: 'tmpl_puter',
senderId: 'Puter',
preferredChannel: 'sms',
},
} as unknown as IConfig);
await client.createVerification('+14155550123');
const [, init] = fetchMock.mock.calls[0];
expect(JSON.parse(init.body).options).toEqual({
code_size: 6,
preferred_channel: 'sms', // config override beats the rcs default
template_id: 'tmpl_puter',
sender_id: 'Puter',
});
});
it('checkVerification POSTs target + code and returns the status', async () => {
fetchMock.mockResolvedValue(okJson({ status: 'success' }));
const client = makeClient('sk_test');
const res = await client.checkVerification('+14155550123', '123456');
expect(res).toEqual({ status: 'success' });
const [url, init] = fetchMock.mock.calls[0];
expect(url).toBe('https://api.prelude.dev/v2/verification/check');
expect(JSON.parse(init.body)).toEqual({
target: { type: 'phone_number', value: '+14155550123' },
code: '123456',
});
});
it('throws (does not call fetch) when not configured', async () => {
const client = makeClient();
await expect(
client.createVerification('+14155550123'),
).rejects.toThrow(/not configured/i);
expect(fetchMock).not.toHaveBeenCalled();
});
it('throws on a non-2xx Prelude response', async () => {
fetchMock.mockResolvedValue({
ok: false,
status: 400,
json: async () => ({ message: 'bad target' }),
} as unknown as Response);
await expect(
makeClient('sk_test').checkVerification('+1', 'x'),
).rejects.toThrow(/Prelude .* failed: 400/);
});
});
@@ -0,0 +1,222 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { IConfig } from '../../types';
import { PuterClient } from '../types';
import { COUNTRY_SMS_PRICES } from './countries.js';
const PRELUDE_API_BASE = 'https://api.prelude.dev/v2';
const REQUEST_TIMEOUT_MS = 8000;
/** OTP length — matches the 6-box code UI. Prelude allows 4–8. */
const PRELUDE_CODE_SIZE = 6;
/**
* Channel Prelude prioritizes for delivery. RCS is far cheaper than SMS, so we
* prefer it by default; Prelude routes to the next reachable channel (SMS) when
* RCS isn't available on the destination. (Requires an RCS agent provisioned in
* the Prelude account to actually use RCS — otherwise it just falls through to
* SMS.) Override with `config.prelude.preferredChannel`.
*/
const DEFAULT_PREFERRED_CHANNEL: PreludeChannel = 'rcs';
/** Delivery channels Prelude can prioritize via `options.preferred_channel`. */
export type PreludeChannel =
| 'sms'
| 'rcs'
| 'whatsapp'
| 'viber'
| 'zalo'
| 'telegram';
/**
* Default per-SMS cost ceiling (EUR). Countries whose Prelude SMS rate exceeds
* this — or that have no SMS channel — are not offered phone verification. The
* cap covers every realistic revenue market (priciest are Germany €0.0598 and
* Saudi Arabia €0.0638) while excluding the expensive, high-fraud long tail.
* Override per-deployment with `config.prelude.maxSmsCostEur`.
*/
const DEFAULT_MAX_SMS_COST_EUR = 0.07;
/** Status returned by Prelude when creating/retrying a verification. */
export type PreludeCreateStatus =
| 'success'
| 'retry'
| 'challenged'
| 'blocked'
| 'shadow_blocked';
/** Status returned by Prelude when checking a code. */
export type PreludeCheckStatus =
| 'success'
| 'failure'
| 'expired_or_not_found'
| 'transaction_missing'
| 'transaction_mismatch';
/**
* Prelude Verify v2 client (https://docs.prelude.so/verify/v2). Sends and checks
* SMS one-time codes — Prelude generates, delivers, and validates the code, so
* we never store one ourselves. No-ops with a warning when no API key is
* configured (so dev environments don't crash); `isConfigured()` lets callers
* surface a clean "phone verification unavailable" error instead.
*/
export class PreludeClient extends PuterClient {
private apiKey: string | null = null;
constructor(config: IConfig) {
super(config);
this.apiKey = config.prelude?.apiKey ?? null;
}
override onServerStart(): void {
if (!this.apiKey) {
console.warn(
'[prelude] no apiKey configured — SMS phone verification is disabled',
);
}
}
/** True when an API key is configured and verification can be attempted. */
isConfigured(): boolean {
return !!this.apiKey;
}
/** ISO region used to parse local-format numbers (config-driven). */
get defaultCountry(): string | undefined {
return this.config.prelude?.defaultCountry;
}
/** Per-SMS cost ceiling in EUR (config override or the default cap). */
get maxSmsCostEur(): number {
return this.config.prelude?.maxSmsCostEur ?? DEFAULT_MAX_SMS_COST_EUR;
}
/**
* Whether SMS verification should be offered for a country. False when the
* country is unknown, has no SMS channel, or its rate exceeds the cost cap.
* @param iso ISO-3166 alpha-2 (e.g. 'US') — from the parsed phone number.
*/
isCountrySupported(iso: string | undefined): boolean {
if (!iso) return false; // couldn't determine country → can't price it
const price = COUNTRY_SMS_PRICES[iso.toUpperCase()];
if (!price || price.sms == null) return false;
return price.sms <= this.maxSmsCostEur;
}
/**
* Create (or retry) a verification: Prelude sends an OTP to `target`.
* @param target E.164 phone number, e.g. "+14155550123".
* @param signals Optional anti-fraud signals. The more we forward, the
* better Prelude's fraud model scores the attempt (per their docs IP +
* device + platform are the highest-value). We pass what the backend
* already has: the request `ip`, the client `device_id` (ThumbmarkJS
* hash), and the `user_agent` (Prelude infers platform/model/OS from it).
* Richer client-collected signals require Prelude's frontend SDK, which
* produces a `dispatch_id` the backend would forward here.
*/
async createVerification(
target: string,
signals: {
ip?: string;
device_id?: string;
user_agent?: string;
} = {},
): Promise<{ id?: string; status: PreludeCreateStatus }> {
// Match the 6-box code UI (UIWindowPhoneVerificationRequired). Without
// code_size Prelude uses the dashboard default (4). preferred_channel
// prioritizes RCS (cheaper); Prelude falls back to SMS when unavailable.
const options: Record<string, unknown> = {
code_size: PRELUDE_CODE_SIZE,
preferred_channel:
this.config.prelude?.preferredChannel ??
DEFAULT_PREFERRED_CHANNEL,
};
// Branding lives in the Prelude dashboard (the message text is a
// template); these just select a Puter-branded template / sender when
// configured. See IPreludeConfig.
const { templateId, senderId } = this.config.prelude ?? {};
if (templateId) options.template_id = templateId;
if (senderId) options.sender_id = senderId;
const body: Record<string, unknown> = {
target: { type: 'phone_number', value: target },
options,
};
// Only attach signals we actually have — Prelude treats the object as
// optional and an empty one adds nothing.
const sig: Record<string, string> = {};
if (signals.ip) sig.ip = signals.ip;
if (signals.device_id) sig.device_id = signals.device_id;
if (signals.user_agent) sig.user_agent = signals.user_agent;
if (Object.keys(sig).length > 0) body.signals = sig;
return this.#post('/verification', body) as Promise<{
id?: string;
status: PreludeCreateStatus;
}>;
}
/**
* Check a code the user entered against the active verification for `target`.
* @returns `{ status }` — `'success'` means verified.
*/
async checkVerification(
target: string,
code: string,
): Promise<{ status: PreludeCheckStatus }> {
return this.#post('/verification/check', {
target: { type: 'phone_number', value: target },
code,
}) as Promise<{ status: PreludeCheckStatus }>;
}
async #post(
path: string,
body: Record<string, unknown>,
): Promise<Record<string, unknown>> {
if (!this.apiKey) {
throw new Error('Prelude is not configured');
}
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
try {
const res = await fetch(`${PRELUDE_API_BASE}${path}`, {
method: 'POST',
headers: {
Authorization: `Bearer ${this.apiKey}`,
'Content-Type': 'application/json',
Accept: 'application/json',
},
body: JSON.stringify(body),
signal: controller.signal,
});
const json = (await res.json().catch(() => ({}))) as Record<
string,
unknown
>;
if (!res.ok) {
throw new Error(
`Prelude ${path} failed: ${res.status} ${
(json as { message?: string })?.message ?? ''
}`.trim(),
);
}
return json;
} finally {
clearTimeout(timer);
}
}
}
+952
View File
@@ -0,0 +1,952 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/** Per-country Prelude SMS pricing, hardcoded from the Prelude global price
* sheet. `sms` is the average cost in EUR, or null when SMS is unavailable
* there. Used to cap phone verification to affordable countries (see
* PreludeClient.isCountrySupported). Regenerate from the price sheet when
* Prelude updates rates. */
export interface CountryPrice {
name: string;
/** Prelude SMS average in EUR, or null when SMS is not available. */
sms: number | null;
}
export const COUNTRY_SMS_PRICES: Record<string, CountryPrice> = {
AD: {
name: 'Andorra',
sms: 0.0515,
},
AE: {
name: 'United Arab Emirates',
sms: 0.0145,
},
AF: {
name: 'Afghanistan',
sms: 0.2409,
},
AG: {
name: 'Antigua and Barbuda',
sms: 0.0755,
},
AI: {
name: 'Anguilla',
sms: 0.0765,
},
AL: {
name: 'Albania',
sms: 0.0345,
},
AM: {
name: 'Armenia',
sms: 0.118,
},
AN: {
name: 'Netherlands Antilles',
sms: null,
},
AO: {
name: 'Angola',
sms: 0.0347,
},
AR: {
name: 'Argentina',
sms: 0.0614,
},
AS: {
name: 'American Samoa',
sms: 0.077,
},
AT: {
name: 'Austria',
sms: 0.014,
},
AU: {
name: 'Australia',
sms: 0.0091,
},
AW: {
name: 'Aruba',
sms: 0.0617,
},
AZ: {
name: 'Azerbaijan',
sms: 0.0969,
},
BA: {
name: 'Bosnia and Herzegovina',
sms: 0.0801,
},
BB: {
name: 'Barbados',
sms: 0.2021,
},
BD: {
name: 'Bangladesh',
sms: 0.1844,
},
BE: {
name: 'Belgium',
sms: 0.0297,
},
BF: {
name: 'Burkina Faso',
sms: 0.063,
},
BG: {
name: 'Bulgaria',
sms: 0.0799,
},
BH: {
name: 'Bahrain',
sms: 0.014,
},
BI: {
name: 'Burundi',
sms: 0.217,
},
BJ: {
name: 'Benin',
sms: 0.13,
},
BM: {
name: 'Bermuda',
sms: 0.21,
},
BN: {
name: 'Brunei',
sms: 0.035,
},
BO: {
name: 'Bolivia',
sms: 0.0708,
},
BQ: {
name: 'Caribbean Netherlands',
sms: 0.0565,
},
BR: {
name: 'Brazil',
sms: 0.0086,
},
BS: {
name: 'Bahamas',
sms: 0.0672,
},
BT: {
name: 'Bhutan',
sms: null,
},
BW: {
name: 'Botswana',
sms: 0.0235,
},
BY: {
name: 'Belarus',
sms: 0.118,
},
BZ: {
name: 'Belize',
sms: 0.1529,
},
CA: {
name: 'Canada',
sms: 0.0052,
},
CD: {
name: 'Congo RDC',
sms: 0.1112,
},
CF: {
name: 'Central African Republic',
sms: 0.2,
},
CG: {
name: 'Congo',
sms: 0.2507,
},
CH: {
name: 'Switzerland',
sms: 0.0163,
},
CI: {
name: "Cote d'Ivoire",
sms: 0.19,
},
CK: {
name: 'Cook Islands',
sms: 0.1041,
},
CL: {
name: 'Chile',
sms: 0.0027,
},
CM: {
name: 'Cameroon',
sms: 0.1504,
},
CN: {
name: 'China',
sms: 0.0053,
},
CO: {
name: 'Colombia',
sms: 0.0008,
},
CR: {
name: 'Costa Rica',
sms: 0.0045,
},
CU: {
name: 'Cuba',
sms: 0.0512,
},
CV: {
name: 'Cabo Verde',
sms: 0.2031,
},
CW: {
name: 'Curacao',
sms: 0.0138,
},
CY: {
name: 'Northern Cyprus',
sms: 0.0065,
},
CZ: {
name: 'Czech Republic',
sms: 0.0299,
},
DE: {
name: 'Germany',
sms: 0.0598,
},
DJ: {
name: 'Djibouti',
sms: 0.0897,
},
DK: {
name: 'Denmark',
sms: 0.0301,
},
DM: {
name: 'Dominica',
sms: 0.0824,
},
DO: {
name: 'Dominican Republic',
sms: 0.0351,
},
DZ: {
name: 'Algeria',
sms: 0.196,
},
EC: {
name: 'Ecuador',
sms: 0.1041,
},
EE: {
name: 'Estonia',
sms: 0.0234,
},
EG: {
name: 'Egypt',
sms: 0.1561,
},
ER: {
name: 'Eritrea',
sms: 0.0659,
},
ES: {
name: 'Spain',
sms: 0.0193,
},
ET: {
name: 'Ethiopia',
sms: 0.2741,
},
FI: {
name: 'Finland',
sms: 0.043,
},
FJ: {
name: 'Fiji',
sms: 0.069,
},
FK: {
name: 'Falkland Islands',
sms: 0.0713,
},
FM: {
name: 'Micronesia',
sms: 0.0118,
},
FO: {
name: 'Faroe Islands',
sms: 0.0319,
},
FR: {
name: 'France',
sms: 0.03,
},
GA: {
name: 'Gabon',
sms: 0.15,
},
GB: {
name: 'United Kingdom',
sms: 0.026,
},
GD: {
name: 'Grenada',
sms: null,
},
GE: {
name: 'Georgia',
sms: 0.0788,
},
GF: {
name: 'French Guiana',
sms: 0.05,
},
GG: {
name: 'Guernsey',
sms: 0.025,
},
GH: {
name: 'Ghana',
sms: 0.15,
},
GI: {
name: 'Gibraltar',
sms: 0.0134,
},
GL: {
name: 'Greenland',
sms: 0.0048,
},
GM: {
name: 'Gambia',
sms: 0.1283,
},
GN: {
name: 'Guinea',
sms: 0.2,
},
GP: {
name: 'Guadeloupe',
sms: 0.0455,
},
GQ: {
name: 'Equatorial Guinea',
sms: 0.099,
},
GR: {
name: 'Greece',
sms: 0.0296,
},
GT: {
name: 'Guatemala',
sms: 0.1202,
},
GU: {
name: 'Guam',
sms: 0.02,
},
GW: {
name: 'Guinea-Bissau',
sms: 0.1566,
},
GY: {
name: 'Guyana',
sms: 0.2178,
},
HK: {
name: 'Hong Kong',
sms: 0.038,
},
HN: {
name: 'Honduras',
sms: 0.162,
},
HR: {
name: 'Croatia',
sms: 0.03,
},
HT: {
name: 'Haiti',
sms: 0.09,
},
HU: {
name: 'Hungary',
sms: 0.029,
},
ID: {
name: 'Indonesia',
sms: 0.243,
},
IE: {
name: 'Ireland',
sms: 0.0305,
},
IL: {
name: 'Israel',
sms: 0.01,
},
IM: {
name: 'Isle of Man',
sms: 0.0385,
},
IN: {
name: 'India',
sms: 0.0375,
},
IQ: {
name: 'Iraq',
sms: 0.151,
},
IR: {
name: 'Iran',
sms: 0.14,
},
IS: {
name: 'Iceland',
sms: 0.0493,
},
IT: {
name: 'Italy',
sms: 0.0245,
},
JE: {
name: 'Jersey',
sms: 0.025,
},
JM: {
name: 'Jamaica',
sms: 0.1525,
},
JO: {
name: 'Jordan',
sms: 0.2094,
},
JP: {
name: 'Japan',
sms: 0.017,
},
KE: {
name: 'Kenya',
sms: 0.125,
},
KG: {
name: 'Kyrgyzstan',
sms: 0.15,
},
KH: {
name: 'Cambodia',
sms: 0.1529,
},
KI: {
name: 'Kiribati',
sms: 0.025,
},
KM: {
name: 'Comoros',
sms: 0.15,
},
KN: {
name: 'Saint Kitts and Nevis',
sms: 0.1295,
},
KR: {
name: 'South Korea',
sms: 0.006,
},
KW: {
name: 'Kuwait',
sms: 0.145,
},
KY: {
name: 'Cayman Islands',
sms: 0.2172,
},
KZ: {
name: 'Kazakhstan',
sms: 0.202,
},
LA: {
name: 'Laos',
sms: 0.15,
},
LB: {
name: 'Lebanon',
sms: 0.153,
},
LC: {
name: 'Saint Lucia',
sms: 0.0892,
},
LI: {
name: 'Liechtenstein',
sms: null,
},
LK: {
name: 'Sri Lanka',
sms: 0.3593,
},
LR: {
name: 'Liberia',
sms: 0.1448,
},
LS: {
name: 'Lesotho',
sms: 0.0301,
},
LT: {
name: 'Lithuania',
sms: 0.0261,
},
LU: {
name: 'Luxembourg',
sms: 0.034,
},
LV: {
name: 'Latvia',
sms: 0.028,
},
LY: {
name: 'Libya',
sms: 0.1898,
},
MA: {
name: 'Morocco',
sms: 0.105,
},
MC: {
name: 'Monaco',
sms: 0.1219,
},
MD: {
name: 'Moldova',
sms: 0.065,
},
ME: {
name: 'Montenegro',
sms: 0.07,
},
MG: {
name: 'Madagascar',
sms: 0.24,
},
MH: {
name: '',
sms: 0.032,
},
MK: {
name: 'Macedonia',
sms: 0.0096,
},
ML: {
name: 'Mali',
sms: 0.136,
},
MM: {
name: 'Myanmar',
sms: 0.3037,
},
MN: {
name: 'Mongolia',
sms: 0.183,
},
MO: {
name: 'Macao',
sms: 0.005,
},
MP: {
name: 'Northern Mariana Islands',
sms: 0.0669,
},
MQ: {
name: 'Martinique',
sms: 0.0455,
},
MR: {
name: 'Mauritania',
sms: 0.161,
},
MS: {
name: 'Montserrat',
sms: 0.0725,
},
MT: {
name: 'Malta',
sms: 0.0343,
},
MU: {
name: 'Mauritius',
sms: 0.1431,
},
MV: {
name: 'Maldives',
sms: 0.145,
},
MW: {
name: 'Malawi',
sms: 0.19,
},
MX: {
name: 'Mexico',
sms: 0.0021,
},
MY: {
name: 'Malaysia',
sms: 0.08,
},
MZ: {
name: 'Mozambique',
sms: 0.2266,
},
NA: {
name: 'Namibia',
sms: 0.0173,
},
NC: {
name: 'New Caledonia',
sms: 0.052,
},
NE: {
name: 'Niger',
sms: 0.143,
},
NG: {
name: 'Nigeria',
sms: 0.198,
},
NI: {
name: 'Nicaragua',
sms: 0.1033,
},
NL: {
name: 'Netherlands',
sms: 0.046,
},
NO: {
name: 'Norway',
sms: 0.03,
},
NP: {
name: 'Nepal',
sms: 0.1645,
},
NR: {
name: 'Nauru',
sms: null,
},
NU: {
name: 'Niue',
sms: null,
},
NZ: {
name: 'New Zealand',
sms: 0.0372,
},
OM: {
name: 'Oman',
sms: 0.0713,
},
PA: {
name: 'Panama',
sms: 0.07,
},
PE: {
name: 'Peru',
sms: 0.13,
},
PF: {
name: 'French Polynesia',
sms: 0.0518,
},
PG: {
name: 'Papua New Guinea',
sms: 0.15,
},
PH: {
name: 'Philippines',
sms: 0.1237,
},
PK: {
name: 'Pakistan',
sms: 0.3548,
},
PL: {
name: 'Poland',
sms: 0.011,
},
PM: {
name: 'Saint Pierre and Miquelon',
sms: 0.0905,
},
PR: {
name: 'Puerto Rico',
sms: 0.02,
},
PS: {
name: 'Palestine',
sms: 0.2541,
},
PT: {
name: 'Portugal',
sms: 0.009,
},
PW: {
name: 'Palau',
sms: null,
},
PY: {
name: 'Paraguay',
sms: 0.0236,
},
QA: {
name: 'Qatar',
sms: 0.1393,
},
RE: {
name: 'Reunion',
sms: 0.034,
},
RO: {
name: 'Romania',
sms: 0.0255,
},
RS: {
name: 'Serbia',
sms: 0.1935,
},
RU: {
name: 'Russia',
sms: 0.2028,
},
RW: {
name: 'Rwanda',
sms: 0.1177,
},
SA: {
name: 'Saudi Arabia',
sms: 0.0638,
},
SB: {
name: 'Solomon Islands',
sms: 0.0357,
},
SC: {
name: 'Seychelles',
sms: 0.0404,
},
SD: {
name: 'Sudan',
sms: 0.224,
},
SE: {
name: 'Sweden',
sms: 0.026,
},
SG: {
name: 'Singapore',
sms: 0.0256,
},
SI: {
name: 'Slovenia',
sms: 0.1,
},
SK: {
name: 'Slovakia',
sms: 0.0245,
},
SL: {
name: 'Sierra Leone',
sms: 0.2322,
},
SM: {
name: 'San Marino',
sms: null,
},
SN: {
name: 'Senegal',
sms: 0.1151,
},
SO: {
name: 'Somalia',
sms: 0.06,
},
SR: {
name: 'Suriname',
sms: 0.119,
},
SS: {
name: 'South Sudan',
sms: 0.15,
},
ST: {
name: 'Sao Tome and Principe',
sms: 0.0133,
},
SV: {
name: 'El Salvador',
sms: 0.06,
},
SX: {
name: 'Sint Maarten',
sms: 0.0623,
},
SY: {
name: 'Syria',
sms: 0.221,
},
SZ: {
name: 'Eswatini',
sms: 0.12,
},
TC: {
name: 'Turks and Caicos Islands',
sms: null,
},
TD: {
name: 'Chad',
sms: 0.1711,
},
TG: {
name: 'Togo',
sms: 0.1806,
},
TH: {
name: 'Thailand',
sms: 0.003,
},
TJ: {
name: 'Tajikistan',
sms: 0.2626,
},
TL: {
name: 'Timor-Leste',
sms: 0.0675,
},
TM: {
name: 'Turkmenistan',
sms: 0.1677,
},
TN: {
name: 'Tunisia',
sms: 0.225,
},
TO: {
name: 'Tonga',
sms: 0.0902,
},
TR: {
name: 'Turkey',
sms: 0.0008,
},
TT: {
name: 'Trinidad and Tobago',
sms: 0.1556,
},
TW: {
name: 'Taiwan',
sms: 0.0165,
},
TZ: {
name: 'Tanzania',
sms: 0.2753,
},
UA: {
name: 'Ukraine',
sms: 0.094,
},
UG: {
name: 'Uganda',
sms: 0.1707,
},
US: {
name: 'United States',
sms: 0.0043,
},
UY: {
name: 'Uruguay',
sms: 0.0174,
},
UZ: {
name: 'Uzbekistan',
sms: 0.292,
},
VC: {
name: 'Saint Vincent and The Grenadines',
sms: 0.1216,
},
VE: {
name: 'Venezuela',
sms: 0.0427,
},
VG: {
name: 'British Virgin Islands',
sms: 0.083,
},
VI: {
name: 'Virgin Island',
sms: 0.0017,
},
VN: {
name: 'Vietnam',
sms: 0.0885,
},
VU: {
name: 'Vanuatu',
sms: 0.1347,
},
WF: {
name: 'Wallis and Futuna',
sms: 0.09,
},
WS: {
name: 'Samoa',
sms: 0.2,
},
XK: {
name: 'Kosovo',
sms: 0.1394,
},
YE: {
name: 'Yemen',
sms: 0.1486,
},
YT: {
name: 'Mayotte',
sms: 0.06,
},
ZA: {
name: 'South Africa',
sms: 0.039,
},
ZM: {
name: 'Zambia',
sms: 0.21,
},
ZW: {
name: 'Zimbabwe',
sms: 0.13,
},
};
+9
View File
@@ -18,6 +18,7 @@
*/
import type { IConfig, WithLifecycle } from '../types';
import type { ClickhouseClient } from './clickhouse/ClickhouseClient';
/**
* Extension-augmentable client registry. Extensions add their own client
@@ -40,6 +41,14 @@ export interface IExtensionClientInstances {
* override this for that name.
*/
[key: string]: unknown;
/**
* Optional ClickHouse analytics client. Absent by default — a production
* deployment registers it via an extension to speed up the app-stats path
* at scale (see {@link ClickhouseClient}). Always branch on its presence
* and fall back to SQL.
*/
clickhouse?: ClickhouseClient;
}
export interface IPuterClient<T extends WithLifecycle = WithLifecycle> {
+36 -1
View File
@@ -17,6 +17,7 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { isAccessTokenActor, isAppActor } from '../../core/actor.js';
import { HttpError } from '../../core/http/HttpError.js';
import { driversContainers } from '../../exports.js';
import {
@@ -55,6 +56,7 @@ export class AppController extends PuterController {
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
},
async (req, res) => {
const apps = await this.appDriver.select({
@@ -70,6 +72,7 @@ export class AppController extends PuterController {
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
},
async (req, res) => {
const name = req.query?.name;
@@ -89,6 +92,16 @@ export class AppController extends PuterController {
// actor calls this, the app id is already on the token — clients
// don't re-send it in the body. Fall back to `actor.app.uid`
// before 400-ing for a missing body field.
//
// Authorization: only two callers are trusted to report opens —
// 1. a root user actor (plain session, no `.app` and no access
// token), e.g. the GUI launching apps on behalf of the user;
// 2. the app-under-user actor for the app being reported, i.e.
// `actor.app.uid === app_uid`.
// Everything else — access tokens (regardless of issuer), asset
// tokens, app actors reporting for a *different* app — is denied,
// otherwise any authenticated party could inflate another app's
// open count.
router.post(
'/rao',
{
@@ -96,8 +109,9 @@ export class AppController extends PuterController {
requireAuth: true,
},
async (req, res) => {
const actor = req.actor;
const bodyAppUid = req.body?.app_uid;
const actorAppUid = req.actor?.app?.uid;
const actorAppUid = actor?.app?.uid;
const app_uid =
typeof bodyAppUid === 'string' && bodyAppUid.length > 0
? bodyAppUid
@@ -108,6 +122,26 @@ export class AppController extends PuterController {
});
}
// Access tokens (and any other non-user/non-app identity,
// e.g. asset tokens) are not allowed to report opens —
// they're shared / scoped credentials and shouldn't drive
// analytics counters.
if (isAccessTokenActor(actor)) {
throw new HttpError(
403,
'Access tokens cannot report app opens',
{ legacyCode: 'forbidden' },
);
}
if (isAppActor(actor) && app_uid !== actorAppUid) {
throw new HttpError(
403,
'App actors can only report opens for their own app',
{ legacyCode: 'forbidden' },
);
}
const app = await this.appStore.getByUid(app_uid);
if (!app)
throw new HttpError(404, 'App not found', {
@@ -153,6 +187,7 @@ export class AppController extends PuterController {
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
},
async (req, res) => {
const raw = req.params.name;
File diff suppressed because it is too large. Load diff
+728 -38
View File
@@ -47,6 +47,7 @@ import {
} from '../../services/auth/OTPUtil.js';
import { sessionCookieFlags } from '../../util/cookieFlags.js';
import { cleanEmail, isBlockedEmail } from '../../util/email.js';
import { parsePhone } from '../../util/phone.js';
import { generate_identifier } from '../../util/identifier.js';
import { getTaskbarItems } from '../../util/taskbarItems.js';
import {
@@ -57,6 +58,8 @@ import { PuterController } from '../types.js';
const USERNAME_REGEX = /^\w{1,}$/;
const USERNAME_MAX_LENGTH = 45;
const FINGERPRINT_MAX_LENGTH = 128;
const DFP_TELEMETRY_ID_MAX_LENGTH = 64;
const RESERVED_USERNAMES = new Set([
'admin',
'administrator',
@@ -99,7 +102,16 @@ export class AuthController extends PuterController {
@Post('/login', {
captcha: true,
rateLimit: { scope: 'login', limit: 10, window: 15 * 60_000 },
// Two limits: per-fingerprint keeps users behind a shared IP
// (offices, campuses) from throttling each other, while the
// coarser per-IP backstop stops an attacker from minting fresh
// fingerprint buckets by rotating client-controlled headers
// (User-Agent etc.). Same pattern on the other unauthenticated
// credential endpoints below.
rateLimit: [
{ scope: 'login', limit: 10, window: 15 * 60_000 },
{ scope: 'login-ip', limit: 50, window: 15 * 60_000, key: 'ip' },
],
})
async handleLogin(req: Request, res: Response): Promise<void> {
const { username, email, password } = req.body;
@@ -206,11 +218,15 @@ export class AuthController extends PuterController {
@Post('/login/otp', {
captcha: true,
rateLimit: {
scope: 'login-otp',
limit: 15,
window: 30 * 60_000,
},
rateLimit: [
{ scope: 'login-otp', limit: 15, window: 30 * 60_000 },
{
scope: 'login-otp-ip',
limit: 60,
window: 30 * 60_000,
key: 'ip',
},
],
})
async handleLoginOtp(req: Request, res: Response): Promise<void> {
const { token, code } = req.body;
@@ -266,11 +282,15 @@ export class AuthController extends PuterController {
@Post('/login/recovery-code', {
captcha: true,
rateLimit: {
scope: 'login-recovery',
limit: 10,
window: 60 * 60_000,
},
rateLimit: [
{ scope: 'login-recovery', limit: 10, window: 60 * 60_000 },
{
scope: 'login-recovery-ip',
limit: 40,
window: 60 * 60_000,
key: 'ip',
},
],
})
async handleLoginRecoveryCode(req: Request, res: Response): Promise<void> {
const { token, code } = req.body;
@@ -339,7 +359,10 @@ export class AuthController extends PuterController {
@Post('/signup', {
captcha: true,
rateLimit: { scope: 'signup', limit: 10, window: 15 * 60_000 },
rateLimit: [
{ scope: 'signup', limit: 10, window: 15 * 60_000 },
{ scope: 'signup-ip', limit: 50, window: 15 * 60_000, key: 'ip' },
],
})
async handleSignup(req: Request, res: Response): Promise<void> {
const body = req.body ?? {};
@@ -355,6 +378,43 @@ export class AuthController extends PuterController {
return;
}
// Optional device signals (browser fingerprint hash, DFP telemetry
// id). Core only enforces shape and forwards the values verbatim —
// signup-abuse policy built on them lives in extensions. Checked
// before the reauth short-circuit so a malformed value is rejected
// on every /signup path.
if (body.fingerprint !== undefined && body.fingerprint !== null) {
if (typeof body.fingerprint !== 'string')
throw new HttpError(400, 'fingerprint must be a string.', {
legacyCode: 'bad_request',
});
if (body.fingerprint.length > FINGERPRINT_MAX_LENGTH)
throw new HttpError(
400,
`fingerprint cannot be longer than ${FINGERPRINT_MAX_LENGTH} characters.`,
{ legacyCode: 'bad_request' },
);
}
if (
body.dfp_telemetry_id !== undefined &&
body.dfp_telemetry_id !== null
) {
if (typeof body.dfp_telemetry_id !== 'string')
throw new HttpError(400, 'dfp_telemetry_id must be a string.', {
legacyCode: 'bad_request',
});
if (body.dfp_telemetry_id.length > DFP_TELEMETRY_ID_MAX_LENGTH)
throw new HttpError(
400,
`dfp_telemetry_id cannot be longer than ${DFP_TELEMETRY_ID_MAX_LENGTH} characters.`,
{ legacyCode: 'bad_request' },
);
}
// Empty strings are treated as absent — a signal that wasn't
// collected, not a malformed request.
const fingerprint: string | null = body.fingerprint || null;
const dfp_telemetry_id: string | null = body.dfp_telemetry_id || null;
// Temp-user reauth short-circuit: when an existing temp user is
// forced through the reauth flow, the GUI re-submits /signup with
// is_temp=true plus the server-signed reauth_token from the 401.
@@ -534,23 +594,45 @@ export class AuthController extends PuterController {
allow: true,
no_temp_user: false,
requires_email_confirmation: false,
// Set by the abuse harness for low-reputation signups: the account is
// created + logged in but gated behind SMS phone verification (in
// addition to email confirmation) instead of being blocked.
requires_phone_verification: false,
// Same idea, one rung up the ladder: gate the account behind
// credit-card verification (a $0 auth handled by an extension).
requires_card_verification: false,
message: null,
code: null,
user_agent: req?.headers?.['user-agent'] ?? null,
fingerprint,
dfp_telemetry_id,
// Populated by the abuse extension's v2 harness; persisted to the
// user row below so the signup-time reputation is referable later.
reputation: null as number | null,
// Stamped by the abuse harness for flagged signups — the id keying
// the `abuse:trail:<id>` decision trail (carrying both the live and
// shadow trails). Surfaced to a blocked user as the Request Code so
// the code they quote support leads straight to their trail.
trail_id: undefined as string | undefined,
};
try {
await this.clients.event?.emitAndWait(
'puter.signup.validate' as never,
validateEvent as never,
'puter.signup.validate',
validateEvent,
{},
);
} catch (e) {
console.warn('[signup] validate hook failed:', e);
}
if (!validateEvent.allow) {
// Pass the trail id back to a blocked user as the Request Code (when
// the harness stamped one), embedded in the message so the existing
// signup-block UI surfaces it without a GUI change.
const requestCode = validateEvent.trail_id;
throw new HttpError(
403,
validateEvent.message ?? 'Signup blocked',
(validateEvent.message ?? 'Signup blocked') +
(requestCode ? ` Request Code: ${requestCode}` : ''),
{
...(validateEvent.code
? { legacyCode: validateEvent.code as never }
@@ -573,6 +655,17 @@ export class AuthController extends PuterController {
const force_email_confirmation = Boolean(
validateEvent.requires_email_confirmation,
);
const force_phone_verification =
Boolean(validateEvent.requires_phone_verification) ||
// Test/QA switch: force the SMS gate on every signup regardless of
// reputation (see config.always_require_phone_verification).
Boolean(this.config.always_require_phone_verification);
const force_card_verification = Boolean(
validateEvent.requires_card_verification ||
// Test/QA switch: force the card gate on every signup regardless of
// reputation (see config.always_require_card_verification).
this.config.always_require_card_verification,
);
// Prepare shared fields
const user_uuid = uuidv4();
@@ -597,11 +690,13 @@ export class AuthController extends PuterController {
email_confirm_code,
email_confirm_token,
email_confirmed: 0,
// Pseudo claims always require email confirmation — the
// validate hook can only tighten, not loosen, so `1`
// stays hardcoded here.
requires_email_confirmation: 1,
last_activity_ts: signupSqlTs,
...(validateEvent.reputation != null
? { reputation: validateEvent.reputation }
: {}),
requires_phone_verification: force_phone_verification ? 1 : 0,
requires_card_verification: force_card_verification ? 1 : 0,
});
// Move from temp group to regular user group
@@ -650,6 +745,8 @@ export class AuthController extends PuterController {
ip_fwd: proxyIpChain,
user_agent: req.headers?.['user-agent'],
origin: req.headers?.origin,
fingerprint,
dfp_telemetry_id,
},
signup_ip: clientIp,
signup_ip_forwarded: proxyIpChain,
@@ -658,6 +755,11 @@ export class AuthController extends PuterController {
signup_server: (this.config as { serverId?: string }).serverId,
referrer: req.body.referrer ?? null,
last_activity_ts: signupSqlTs,
reputation: validateEvent.reputation,
// Phone collected later in the verification dialog (null now).
phone: null,
requires_phone_verification: force_phone_verification,
requires_card_verification: force_card_verification,
} as never);
// Add to default group
@@ -728,6 +830,11 @@ export class AuthController extends PuterController {
user_uuid: user!.uuid,
email: user!.email,
username: user!.username,
fingerprint,
// Reflects the row that was actually created/claimed —
// a pseudo-user claim ends up with credentials, so it
// reports false here. Same signal completeLogin uses.
is_temp: user!.password === null && user!.email === null,
ip:
(req?.headers?.['x-forwarded-for'] as
| string
@@ -764,7 +871,7 @@ export class AuthController extends PuterController {
// -- Logout ------------------------------------------------------
@Post('/logout', {
requireAuth: true,
requireUserActor: true,
allowUnconfirmed: true,
antiCsrf: true,
})
@@ -928,6 +1035,507 @@ export class AuthController extends PuterController {
res.json({ email_confirmed: true, original_client_socket_id });
}
// -- Phone verification (SMS via Prelude) ------------------------
@Post('/send-confirm-phone', {
subdomain: ['api', ''],
requireUserActor: true,
allowUnconfirmed: true,
rateLimit: {
scope: 'send-confirm-phone',
limit: 10,
window: 60 * 60_000,
key: 'user',
},
})
async handleSendConfirmPhone(req: Request, res: Response): Promise<void> {
const user = await this.stores.user.getById(req.actor!.user.id!, {
force: true,
});
if (!user)
throw new HttpError(404, 'User not found.', {
legacyCode: 'user_not_found' as never,
});
if (user.suspended)
throw new HttpError(403, 'Account suspended.', {
legacyCode: 'account_suspended',
});
if (!this.clients.prelude?.isConfigured())
throw new HttpError(503, 'Phone verification is unavailable.', {
legacyCode: 'service_unavailable' as never,
});
// Parse to E.164 (Prelude's required form + the stored form) and the
// country, so we can apply the per-country cost cap.
const parsed = parsePhone(
req.body?.phone,
this.clients.prelude.defaultCountry,
);
if (!parsed)
throw new HttpError(400, 'Invalid phone number.', {
legacyCode: 'bad_request',
});
// Cost cap: skip countries with no SMS channel or rates above the cap
// (see PreludeClient / countries.ts). Avoids paying exorbitant per-SMS
// rates in low-revenue, high-fraud geographies.
if (!this.clients.prelude.isCountrySupported(parsed.country))
throw new HttpError(
400,
'Phone verification is not available for this country.',
{ legacyCode: 'phone_country_not_supported' as never },
);
// Abuse caps live ENTIRELY in a listening abuse extension, consulted
// via `puter.phone-verification.check`. The backend ships no thresholds
// or detection of its own (so none of it is readable in the open-source
// repo): it forwards the user / number / ip, and the extension decides
// `allowed` plus an opaque `reason` (per-account + per-number send
// velocity, cross-account reuse, …). With no extension listening
// `allowed` stays true. Fail-open on a hook error — this is abuse/cost
// control, not a security boundary (the route rate limit and the country
// cost cap remain), so a flaky hook must not lock signups out.
const abuseCheck = {
user_id: user.id,
user_uid: user.uuid,
phone: parsed.e164,
device_fingerprint: req.deviceFingerprint ?? null,
allowed: true,
reason: null as string | null,
};
try {
await this.clients.event?.emitAndWait(
'puter.phone-verification.check',
abuseCheck,
{},
);
} catch (e) {
console.warn('[send-confirm-phone] abuse-check hook failed:', e);
}
// Forward the verdict verbatim: a generic 429 plus the opaque reason for
// the client to message on. The backend never interprets the reason —
// its meaning lives in the extension (which sets it) and the GUI (which
// displays it), so no abuse semantics leak into the OSS repo.
if (abuseCheck.allowed === false)
throw new HttpError(
429,
'Phone verification is unavailable for this number right now.',
{
legacyCode: 'phone_verification_unavailable' as never,
fields: abuseCheck.reason
? { reason: abuseCheck.reason }
: {},
},
);
// Stage the parsed number as pending in KV (NOT on the user row) so a
// never-confirmed number is never written to the indexed `phone`
// column. /confirm-phone reads it back and persists it to the row only
// once Prelude confirms the code. ~1h TTL covers the code's lifetime.
// Stored before the send so we never dispatch an SMS we couldn't later
// confirm against.
const pendingPhoneKey = `phone-verify-pending:${user.id}`;
try {
await this.stores.kv.set({
key: pendingPhoneKey,
value: parsed.e164,
expireAt: Math.floor(Date.now() / 1000) + 60 * 60,
});
} catch (e) {
console.warn('[send-confirm-phone] pending-store failed:', e);
throw new HttpError(503, 'Could not start phone verification.', {
legacyCode: 'service_unavailable' as never,
});
}
const ip = req.ip || req.socket?.remoteAddress || undefined;
const userAgent =
typeof req.headers['user-agent'] === 'string'
? req.headers['user-agent']
: undefined;
try {
const result = await this.clients.prelude.createVerification(
parsed.e164,
{
ip,
device_id: req.deviceFingerprint ?? undefined,
user_agent: userAgent,
},
);
// Prelude rejected the attempt as abusive — surface as rate-limit.
if (
result.status === 'blocked' ||
result.status === 'shadow_blocked'
) {
throw new HttpError(
429,
'Phone verification is temporarily unavailable for this number.',
{ legacyCode: 'too_many_requests' as never },
);
}
} catch (e) {
if (e instanceof HttpError) throw e;
console.warn('[send-confirm-phone] createVerification failed:', e);
throw new HttpError(502, 'Could not send verification code.', {
legacyCode: 'upstream_error' as never,
});
}
// Tell the abuse extension a code was actually sent, so it can bump its
// send-velocity counters (per number + per account). Fire-and-forget;
// the backend keeps no send counts of its own. Only reached after a
// successful send, so an upstream error never burns quota.
try {
this.clients.event?.emit(
'puter.phone-verification.sent' as never,
{
user_id: user.id,
user_uid: user.uuid,
phone: parsed.e164,
device_fingerprint: req.deviceFingerprint ?? null,
} as never,
{},
);
} catch {
// ignore — best-effort velocity signal
}
res.json({});
}
@Post('/confirm-phone', {
subdomain: ['api', ''],
requireUserActor: true,
allowUnconfirmed: true,
rateLimit: {
scope: 'confirm-phone',
limit: 10,
window: 10 * 60_000,
key: 'user',
},
})
async handleConfirmPhone(req: Request, res: Response): Promise<void> {
const { code, original_client_socket_id } = req.body ?? {};
if (!code)
throw new HttpError(400, 'Missing `code`.', {
legacyCode: 'bad_request',
});
const user = await this.stores.user.getById(req.actor!.user.id!, {
force: true,
});
if (!user)
throw new HttpError(404, 'User not found.', {
legacyCode: 'not_found',
});
if (!user.requires_phone_verification) {
res.json({ phone_verified: true, original_client_socket_id });
return;
}
// The number being verified is the one staged at send time (KV
// pending), not the user row — we don't persist an unverified number.
// Fall back to a row value for accounts that already have one on file
// (legacy / a number persisted by a prior verified flow).
const pendingPhoneKey = `phone-verify-pending:${user.id}`;
let pendingPhone: string | null = null;
try {
const { res: staged } = await this.stores.kv.get({
key: pendingPhoneKey,
});
if (typeof staged === 'string' && staged) pendingPhone = staged;
} catch (e) {
console.warn('[confirm-phone] pending read failed:', e);
}
if (!pendingPhone) pendingPhone = user.phone ?? null;
if (!pendingPhone)
throw new HttpError(
400,
'No phone number on file. Request a code first.',
{ legacyCode: 'bad_request' },
);
if (!this.clients.prelude?.isConfigured())
throw new HttpError(503, 'Phone verification is unavailable.', {
legacyCode: 'service_unavailable' as never,
});
let status;
try {
({ status } = await this.clients.prelude.checkVerification(
pendingPhone,
String(code),
));
} catch (e) {
console.warn('[confirm-phone] checkVerification failed:', e);
throw new HttpError(502, 'Could not verify code.', {
legacyCode: 'upstream_error' as never,
});
}
if (status !== 'success') {
res.json({ phone_verified: false, original_client_socket_id });
return;
}
// Verified — persist the number now (and only now) and clear the gate.
await this.stores.user.update(user.id, {
requires_phone_verification: 0,
phone: pendingPhone,
});
try {
this.clients.event?.emit(
'user.phone-verified' as never,
{
user_id: user.id,
user_uid: user.uuid,
phone: pendingPhone,
} as never,
{},
);
} catch {
// ignore — event is a side-channel signal, not load-bearing
}
// Notify other tabs/devices for this user so they refresh + drop the gate.
try {
await this.services.socket?.send(
{ room: user.id },
'user.phone_verified',
{ original_client_socket_id },
);
} catch {
// ignore — best-effort
}
res.json({ phone_verified: true, original_client_socket_id });
}
// -- Card verification ($0 auth via a payments extension) --------
/**
* Start card verification for the calling user. Pure mechanism: the
* endpoint emits `puter.card-verification.setup` and a payments
* extension fills in the client credentials — the OSS backend holds
* no provider knowledge or config. Phone verification (when required)
* must be completed first; the ordering is enforced here so a client
* can't skip the cheaper gate.
*/
@Post('/card-verification/setup', {
subdomain: ['api', ''],
requireUserActor: true,
allowUnconfirmed: true,
rateLimit: {
scope: 'card-verification-setup',
limit: 5,
window: 60 * 60_000,
key: 'user',
},
})
async handleCardVerificationSetup(
req: Request,
res: Response,
): Promise<void> {
const user = await this.stores.user.getById(req.actor!.user.id!, {
force: true,
});
if (!user)
throw new HttpError(404, 'User not found.', {
legacyCode: 'user_not_found' as never,
});
if (user.suspended)
throw new HttpError(403, 'Account suspended.', {
legacyCode: 'account_suspended',
});
if (!user.requires_card_verification) {
res.json({ card_verified: true });
return;
}
if (user.requires_phone_verification)
throw new HttpError(
409,
'Phone verification must be completed first.',
{ legacyCode: 'conflict' },
);
// `enabled` stays null when no extension is listening; an installed
// extension always sets it (true/false) before doing any work.
const setupEvent = {
user_id: user.id,
user_uid: user.uuid,
ip: (req.ip || req.socket?.remoteAddress || null) as string | null,
device_fingerprint: req.deviceFingerprint ?? null,
enabled: null as boolean | null,
allowed: true,
reason: null as string | null,
client_secret: null as string | null,
publishable_key: null as string | null,
};
try {
await this.clients.event?.emitAndWait(
'puter.card-verification.setup',
setupEvent,
{},
);
} catch (e) {
console.warn('[card-verification/setup] setup hook failed:', e);
}
// Abuse veto (e.g. per-device setup-velocity cap): the extension refused
// before any Stripe work. Forward the opaque reason verbatim as a 429,
// same as the phone gate — the backend never interprets it.
if (setupEvent.allowed === false)
throw new HttpError(
429,
'Card verification is unavailable right now.',
{
legacyCode: 'too_many_requests' as never,
fields: setupEvent.reason
? { reason: setupEvent.reason }
: {},
},
);
// Kill switch: the extension reports the feature disabled — unstick
// any user still carrying the flag instead of dead-ending them.
if (setupEvent.enabled === false) {
await this.stores.user.update(user.id, {
requires_card_verification: 0,
});
res.json({ card_verified: true, disabled: true });
return;
}
if (!setupEvent.client_secret || !setupEvent.publishable_key)
throw new HttpError(503, 'Card verification is not available.', {
legacyCode: 'service_unavailable' as never,
});
res.json({
client_secret: setupEvent.client_secret,
publishable_key: setupEvent.publishable_key,
});
}
/**
* Complete card verification. The client confirms the setup intent with
* the payment provider directly, then posts the resulting id here; the
* payments extension checks it (and applies its own abuse limits) via
* `puter.card-verification.confirm`. On success the gate clears exactly
* like `/confirm-phone` clears the phone gate.
*/
@Post('/card-verification/confirm', {
subdomain: ['api', ''],
requireUserActor: true,
allowUnconfirmed: true,
rateLimit: {
scope: 'card-verification-confirm',
limit: 10,
window: 10 * 60_000,
key: 'user',
},
})
async handleCardVerificationConfirm(
req: Request,
res: Response,
): Promise<void> {
const { setup_intent_id, original_client_socket_id } = req.body ?? {};
if (
typeof setup_intent_id !== 'string' ||
setup_intent_id.length === 0 ||
setup_intent_id.length > 255
)
throw new HttpError(400, 'Invalid `setup_intent_id`.', {
legacyCode: 'bad_request',
});
const user = await this.stores.user.getById(req.actor!.user.id!, {
force: true,
});
if (!user)
throw new HttpError(404, 'User not found.', {
legacyCode: 'not_found',
});
if (!user.requires_card_verification) {
res.json({ card_verified: true });
return;
}
if (user.requires_phone_verification)
throw new HttpError(
409,
'Phone verification must be completed first.',
{ legacyCode: 'conflict' },
);
const confirmEvent = {
user_id: user.id,
user_uid: user.uuid,
setup_intent_id,
enabled: null as boolean | null,
verified: false,
reason: null as string | null,
fingerprint: null as string | null,
funding: null as string | null,
country: null as string | null,
};
try {
await this.clients.event?.emitAndWait(
'puter.card-verification.confirm',
confirmEvent,
{},
);
} catch (e) {
console.warn('[card-verification/confirm] confirm hook failed:', e);
}
// Kill switch — same semantics as /card-verification/setup.
if (confirmEvent.enabled === false) {
await this.stores.user.update(user.id, {
requires_card_verification: 0,
});
res.json({ card_verified: true, disabled: true });
return;
}
// No extension listening — nothing could have verified anything.
if (confirmEvent.enabled === null)
throw new HttpError(503, 'Card verification is not available.', {
legacyCode: 'service_unavailable' as never,
});
if (confirmEvent.verified !== true) {
res.json({ card_verified: false, reason: confirmEvent.reason });
return;
}
await this.stores.user.update(user.id, {
requires_card_verification: 0,
});
try {
this.clients.event?.emit(
'user.card-verified' as never,
{
user_id: user.id,
user_uid: user.uuid,
fingerprint: confirmEvent.fingerprint,
funding: confirmEvent.funding,
country: confirmEvent.country,
} as never,
{},
);
} catch {
// ignore — event is a side-channel signal, not load-bearing
}
// Notify other tabs/devices for this user so they refresh + drop the gate.
try {
await this.services.socket?.send(
{ room: user.id },
'user.card_verified',
{ original_client_socket_id },
);
} catch {
// ignore — best-effort
}
res.json({ card_verified: true });
}
// -- Password recovery -------------------------------------------
@Post('/send-pass-recovery-email', {
@@ -1129,6 +1737,12 @@ export class AuthController extends PuterController {
}
await this.stores.user.invalidateById(user.id);
// A password reset is the "I think someone else has access" flow —
// evict every interactive session so a hijacked one doesn't survive.
await this.services.auth.revokeInteractiveSessionsForUserId(
user.id as number,
);
res.send('Password successfully updated.');
}
@@ -1165,6 +1779,10 @@ export class AuthController extends PuterController {
change_email_confirm_token: null,
});
// Sign out every other web session (cascading to their derived
// rows); only the session that changed the password survives.
await this.services.auth.revokeAllSessions(req.actor!);
if (this.clients.email && user.email) {
try {
await this.clients.email.send(
@@ -1634,7 +2252,9 @@ export class AuthController extends PuterController {
// -- Anti-CSRF token generation ----------------------------------
@Get('/get-anticsrf-token', {
requireAuth: true,
// Anti-CSRF tokens are only consumed by `requireUserActor` routes,
// so issuance is scoped to the same actor kind for consistency.
requireUserActor: true,
allowUnconfirmed: true,
})
async handleGetAntiCsrfToken(req: Request, res: Response): Promise<void> {
@@ -2138,12 +2758,15 @@ export class AuthController extends PuterController {
// -- Access tokens -----------------------------------------------
async handleMigrateToken(req: Request, res: Response): Promise<void> {
// 1. Origin lock. Reject anything that isn't same-origin to
// `config.origin` or in the explicit per-deployment allowlist.
// No Origin header → reject (this endpoint is browser-only by
// design; server-side callers should re-auth properly).
// 1. Browser-only gate. No Origin header → reject (server-side
// callers should re-auth properly). The exchange itself is open
// to any browser origin: puter.js apps live on arbitrary
// third-party domains, the v1 bearer token is the credential,
// and cookies are never read here — so a hostile page learns
// nothing it doesn't already hold. Origin *trust* only decides
// whether the `puter_token_v2` companion cookie is set below.
const reqOrigin = req.headers.origin;
if (!reqOrigin || !this.#isMigrateTokenOriginAllowed(reqOrigin)) {
if (!reqOrigin) {
throw new HttpError(403, 'Origin not allowed', {
legacyCode: 'forbidden',
});
@@ -2173,12 +2796,19 @@ export class AuthController extends PuterController {
: undefined,
});
// `puter_token_v2` is the cookie companion to v2 app tokens —
// the app runs in-browser (we already gated on Origin above) so
// the GUI's cookie-only middleware can authenticate subsequent
// calls from the same iframe without the client having to
// forward Authorization headers. Access tokens are programmatic
// (no browser cookie surface) so we deliberately skip them here.
if (result.kind === 'app') {
// it lets the GUI's cookie-only middleware authenticate
// subsequent same-origin calls without the client forwarding
// Authorization headers. Issuing it is gated on the trusted
// origin allowlist: an arbitrary attacker page must not be able
// to plant a session cookie on the GUI origin (login CSRF).
// Untrusted origins still get the token in the JSON body — the
// cookie is only consumed same-origin (see authProbe) so they
// lose nothing. Access tokens are programmatic (no browser
// cookie surface) so we deliberately skip them here.
if (
result.kind === 'app' &&
this.#isMigrateTokenOriginAllowed(reqOrigin)
) {
res.cookie('puter_token_v2', result.token, {
...sessionCookieFlags(this.config),
httpOnly: true,
@@ -2210,13 +2840,25 @@ export class AuthController extends PuterController {
requireAuth: true,
})
async handleCreateAccessToken(req: Request, res: Response): Promise<void> {
const { permissions, expiresIn } = req.body;
const { permissions, expiresIn, label } = req.body;
if (!Array.isArray(permissions) || permissions.length === 0) {
throw new HttpError(400, 'Missing or empty `permissions` array', {
legacyCode: 'bad_request',
});
}
// Optional user-facing name for the manage-sessions UI. Trim and clamp
// to the same 64-char limit the rename endpoint enforces.
let normalizedLabel: string | null = null;
if (label !== undefined && label !== null) {
if (typeof label !== 'string') {
throw new HttpError(400, '`label` must be a string', {
legacyCode: 'bad_request',
});
}
normalizedLabel = label.trim().slice(0, 64) || null;
}
// Normalize specs: string → [string], [string] → [string, {}], [string, extra] → as-is
const normalized = permissions.map((spec) => {
if (typeof spec === 'string') return [spec];
@@ -2231,7 +2873,10 @@ export class AuthController extends PuterController {
const token = await this.services.auth.createAccessToken(
req.actor!,
normalized as never,
expiresIn ? { expiresIn } : {},
{
...(expiresIn ? { expiresIn } : {}),
...(normalizedLabel ? { label: normalizedLabel } : {}),
},
);
res.json({ token });
}
@@ -2335,8 +2980,8 @@ export class AuthController extends PuterController {
}
await this.clients.db.write(
'UPDATE `user` SET `otp_enabled` = 1 WHERE `uuid` = ?',
[user.uuid],
'UPDATE `user` SET `otp_enabled` = ? WHERE `uuid` = ?',
[this.clients.db.booleanValue(true), user.uuid],
);
await this.stores.user.invalidateById(user.id);
@@ -2371,8 +3016,8 @@ export class AuthController extends PuterController {
});
await this.clients.db.write(
'UPDATE `user` SET `otp_enabled` = 0, `otp_recovery_codes` = NULL, `otp_secret` = NULL WHERE `uuid` = ?',
[user.uuid],
'UPDATE `user` SET `otp_enabled` = ?, `otp_recovery_codes` = NULL, `otp_secret` = NULL WHERE `uuid` = ?',
[this.clients.db.booleanValue(false), user.uuid],
);
await this.stores.user.invalidateById(user.id);
@@ -2468,6 +3113,9 @@ export class AuthController extends PuterController {
});
await this.stores.group.addUsers(uid, users);
// New members inherit the group's permissions immediately, not
// after the permission-cache TTL.
await this.services.permission.bumpPermissionCacheForUsernames(users);
res.json({});
}
@@ -2497,6 +3145,9 @@ export class AuthController extends PuterController {
});
await this.stores.group.removeUsers(uid, users);
// Removed members must lose the group's permissions immediately,
// not after the permission-cache TTL.
await this.services.permission.bumpPermissionCacheForUsernames(users);
res.json({});
}
@@ -2789,7 +3440,6 @@ export class AuthController extends PuterController {
router.post(
'/auth/migrate-token',
{
subdomain: 'api',
rateLimit: {
scope: 'migrate-token',
limit: 20,
@@ -2804,6 +3454,22 @@ export class AuthController extends PuterController {
// -- Private helpers ----------------------------------------------
async #cascadeDeleteUser(userId: number): Promise<void> {
// Capture the identifiers downstream teardown needs before the row is
// gone — the marketplace extension cancels the user's Stripe
// subscriptions off `user.delete`, keyed by uuid / customer id.
let userUuid: string | undefined;
let stripeCustomerId: string | null = null;
try {
const rows = (await this.clients.db.read(
'SELECT `uuid`, `stripe_customer_id` FROM `user` WHERE `id` = ?',
[userId],
)) as Array<{ uuid?: string; stripe_customer_id?: string | null }>;
userUuid = rows[0]?.uuid;
stripeCustomerId = rows[0]?.stripe_customer_id ?? null;
} catch (e) {
console.warn('[cascade-delete-user] identifier lookup failed:', e);
}
try {
await this.services.fs.removeAllForUser(userId);
} catch (e) {
@@ -2821,6 +3487,24 @@ export class AuthController extends PuterController {
userId,
]);
await this.stores.user.invalidateById(userId);
// Fire-and-forget: let listeners purge external state tied to the
// account (Stripe subscriptions are cancelled immediately, without
// proration). Emitted after the row delete — listeners key off the
// payload, not the DB row.
try {
this.clients.event?.emit(
'user.delete',
{
user_id: userId,
user_uuid: userUuid,
stripe_customer_id: stripeCustomerId,
},
{},
);
} catch {
// ignore — event emission shouldn't block deletion
}
}
async #generateRandomUsername(): Promise<string> {
@@ -2980,6 +3664,9 @@ export class AuthController extends PuterController {
password?: string | null;
email_confirmed?: number | boolean;
requires_email_confirmation?: number | boolean;
phone?: string | null;
requires_phone_verification?: number | boolean;
requires_card_verification?: number | boolean;
},
): Promise<void> {
const meta = {
@@ -3028,6 +3715,9 @@ export class AuthController extends PuterController {
email: user.email,
email_confirmed: user.email_confirmed,
requires_email_confirmation: user.requires_email_confirmation,
phone: user.phone,
requires_phone_verification: user.requires_phone_verification,
requires_card_verification: user.requires_card_verification,
is_temp: user.password === null && user.email === null,
taskbar_items,
},
@@ -51,6 +51,7 @@ export class DesktopController extends PuterController {
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
},
async (req, res) => {
const { url, color, fit } = req.body ?? {};
@@ -105,6 +106,7 @@ export class DesktopController extends PuterController {
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
},
async (req, res) => {
const { items } = req.body ?? {};
@@ -130,6 +132,7 @@ export class DesktopController extends PuterController {
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
},
async (req, res) => {
const { item_uid, item_path, layout } = req.body ?? {};
@@ -156,6 +159,7 @@ export class DesktopController extends PuterController {
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
},
async (req, res) => {
const { item_uid, item_path, sort_by, sort_order } =
@@ -21,7 +21,8 @@ import { Readable } from 'node:stream';
import type { Request, RequestHandler, Response } from 'express';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import type { Actor } from '../../core/actor.js';
import { actorUid, type Actor } from '../../core/actor.js';
import type { DriverMethodLifecycleEvent } from '../../clients/event/types.js';
import { runWithContext } from '../../core/context.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
@@ -33,7 +34,7 @@ import type { DriverController } from './DriverController.js';
// The DriverController under test is the same instance the live request
// pipeline uses, so its iface→driver registry is populated from real
// drivers (puter-kvstore, puter-apps, puter-subdomains, …). The HTTP
// handlers (`#handleCall`, `#handleListInterfaces`, `#handleXd`) are
// handlers (`#handleCall`, `#handleListInterfaces`) are
// private — we exercise the public lookup API (`resolve` / list / get
// default) which the handlers themselves delegate to.
@@ -116,7 +117,7 @@ describe('DriverController.resolve', () => {
});
});
// ── Route handlers (#handleCall, #handleListInterfaces, #handleXd) ─
// ── Route handlers (#handleCall, #handleListInterfaces) ─────────────
// The handlers are private class fields. We capture references to them by
// invoking `registerRoutes` with a fake router whose `post`/`get` save the
@@ -189,10 +190,7 @@ const makeRes = (): MockRes => {
return res;
};
const makeReq = (
body: Record<string, unknown> = {},
actor?: Actor,
): Request =>
const makeReq = (body: Record<string, unknown> = {}, actor?: Actor): Request =>
({
body,
actor,
@@ -338,11 +336,7 @@ describe('DriverController.#handleCall (via captured router)', () => {
actor,
);
await runWithContext({ actor }, () =>
routes['POST /call'](
req,
res as unknown as Response,
() => {},
),
routes['POST /call'](req, res as unknown as Response, () => {}),
);
const body = res.body as {
success: boolean;
@@ -374,11 +368,9 @@ describe('DriverController.#handleCall (via captured router)', () => {
// Register it through the controller's private map by re-running
// its #buildIfaceMap path: easier to just stash it into the
// existing iface map directly via TypeScript-defeating cast.
const internalDrivers = (
controller as unknown as {
['#drivers']: Map<string, Map<string, unknown>>;
}
);
const internalDrivers = controller as unknown as {
['#drivers']: Map<string, Map<string, unknown>>;
};
// Access the actual private slot via the well-known getter
// pattern doesn't work for `#`-private fields; instead, re-run
// registerRoutes after stashing on the bag — but that's already
@@ -413,13 +405,12 @@ describe('DriverController.#handleCall (via captured router)', () => {
>;
const original = kv.set;
kv.set = async function (...args: unknown[]) {
const { Context } = await import(
'../../core/context.js'
);
const { Context } = await import('../../core/context.js');
Context.set('driverMetadata', { providerUsed: 'kv-direct' });
return (
original as (...x: unknown[]) => Promise<unknown>
).apply(this, args);
return (original as (...x: unknown[]) => Promise<unknown>).apply(
this,
args,
);
};
try {
const res = makeRes();
@@ -432,11 +423,7 @@ describe('DriverController.#handleCall (via captured router)', () => {
actor,
);
await runWithContext({ actor }, () =>
routes['POST /call'](
req,
res as unknown as Response,
() => {},
),
routes['POST /call'](req, res as unknown as Response, () => {}),
);
const body = res.body as {
metadata?: Record<string, unknown>;
@@ -474,6 +461,129 @@ describe('DriverController.#handleCall (via captured router)', () => {
});
});
describe('DriverController driver-method lifecycle events', () => {
let routes: Captured;
beforeAll(() => {
routes = captureRoutes(controller);
});
const grantKvPerm = async (actor: Actor) => {
await server.stores.permission.setFlatUserPerm(
actor.user!.id!,
'service:puter-kvstore:ii:puter-kvstore',
{
permission: 'service:puter-kvstore:ii:puter-kvstore',
deleted: false,
issuer_user_id: actor.user!.id!,
} as never,
);
};
it('emits before then after around a successful driver method', async () => {
const actor = await makeUserActor();
await grantKvPerm(actor);
// Correlate on the actor: each test creates a fresh user, so this
// isolates our call from any other traffic on the shared event bus.
const who = actorUid(actor);
const before: DriverMethodLifecycleEvent[] = [];
const after: DriverMethodLifecycleEvent[] = [];
server.clients.event.on(
'driver.puter-kvstore.set.before',
(_k, data) => {
if (data.actorUid === who) before.push(data);
},
);
server.clients.event.on(
'driver.puter-kvstore.set.after',
(_k, data) => {
if (data.actorUid === who) after.push(data);
},
);
const key = `lc-${uuidv4()}`;
const req = makeReq(
{
interface: 'puter-kvstore',
method: 'set',
args: { key, value: 'v' },
},
actor,
);
await runWithContext({ actor }, () =>
routes['POST /call'](
req,
makeRes() as unknown as Response,
() => {},
),
);
expect(before).toHaveLength(1);
expect(before[0]).toMatchObject({
phase: 'before',
iface: 'puter-kvstore',
method: 'set',
actor,
actorUid: who,
});
expect(after).toHaveLength(1);
expect(after[0].phase).toBe('after');
expect(after[0].args).toMatchObject({ key, value: 'v' });
expect(typeof after[0].durationMs).toBe('number');
});
it('emits reject and answers 403 when a before listener vetoes', async () => {
const actor = await makeUserActor();
await grantKvPerm(actor);
// Scope the veto to a sentinel key so it doesn't affect other calls
// sharing this server's event bus.
const key = `veto-${uuidv4()}`;
const reject: DriverMethodLifecycleEvent[] = [];
server.clients.event.on(
'driver.puter-kvstore.set.before',
(_k, data) => {
if ((data.args as { key?: string })?.key === key) {
data.allow = false;
data.rejectReason = 'blocked in test';
}
},
);
server.clients.event.on(
'driver.puter-kvstore.set.reject',
(_k, data) => {
if (data.rejectReason === 'blocked in test') reject.push(data);
},
);
const req = makeReq(
{
interface: 'puter-kvstore',
method: 'set',
args: { key, value: 'v' },
},
actor,
);
await expect(
runWithContext({ actor }, () =>
routes['POST /call'](
req,
makeRes() as unknown as Response,
() => {},
),
),
).rejects.toMatchObject({ statusCode: 403 });
expect(reject).toHaveLength(1);
expect(reject[0]).toMatchObject({
phase: 'reject',
method: 'set',
args: { key, value: 'v' },
rejectReason: 'blocked in test',
});
});
});
describe('DriverController.#handleListInterfaces', () => {
let routes: Captured;
beforeAll(() => {
@@ -497,31 +607,11 @@ describe('DriverController.#handleListInterfaces', () => {
});
});
describe('DriverController.#handleXd', () => {
let routes: Captured;
beforeAll(() => {
routes = captureRoutes(controller);
});
it('serves the iframe-bridge HTML with text/html content-type', () => {
const res = makeRes();
routes['GET /xd'](
makeReq(),
res as unknown as Response,
() => {},
);
expect(res.contentType).toBe('text/html');
expect(res.sentBody).toMatch(/<!DOCTYPE html>/);
expect(res.sentBody).toMatch(/\/drivers\/call/);
});
});
describe('DriverController.registerRoutes', () => {
it('registers POST /call, GET /list-interfaces, and GET /xd', () => {
it('registers POST /call and GET /list-interfaces', () => {
const routes = captureRoutes(controller);
expect(routes['POST /call']).toBeInstanceOf(Function);
expect(routes['GET /list-interfaces']).toBeInstanceOf(Function);
expect(routes['GET /xd']).toBeInstanceOf(Function);
});
});
@@ -18,6 +18,7 @@
*/
import type { Request, Response } from 'express';
import { actorUid } from '../../core/actor.js';
import { Context } from '../../core/context.js';
import { Controller } from '../../core/http/decorators.js';
import { HttpError, isHttpError } from '../../core/http/HttpError.js';
@@ -26,8 +27,6 @@ import {
checkDriverRateLimit,
} from '../../core/http/middleware/rateLimit.js';
import type { PuterRouter } from '../../core/http/PuterRouter.js';
import type { PermissionService } from '../../services/permission/PermissionService.js';
import type { WithLifecycle } from '../../types';
import type { DriverMeta } from '../../drivers/meta.js';
import {
isDriverStreamResult,
@@ -35,120 +34,13 @@ import {
resolveDriverMethodConcurrent,
resolveDriverMethodRateLimit,
} from '../../drivers/meta.js';
import type { PermissionService } from '../../services/permission/PermissionService.js';
import { PermissionUtil } from '../../services/permission/permissionUtil.js';
import type { WithLifecycle } from '../../types';
import { PuterController } from '../types.js';
type DriverInstance = WithLifecycle & Record<string, unknown>;
// -- /drivers/xd payload ---------------------------------------------
//
// Self-contained HTML/JS shipped to the iframe consumer. Listens for
// postMessage events shaped as `{ id, interface, method, params }`,
// forwards to `/drivers/call`, and posts `{ id, result }` back to the
// originating window.
//
// Wire-shape note: the postMessage uses `params` (puter-js's historical
// name) but `/drivers/call` expects `args`; this bridge translates.
const XD_SCRIPT = /* js */ `
(function () {
const call = async ({ interface_name, method_name, params }) => {
const response = await fetch('/drivers/call', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
interface: interface_name,
method: method_name,
args: params,
}),
});
return await response.json();
};
const fcall = async ({ interface_name, method_name, params }) => {
const form = new FormData();
form.append('interface', interface_name);
form.append('method', method_name);
for (const k in params) {
form.append(k, params[k]);
}
const response = await fetch('/drivers/call', {
method: 'POST',
body: form,
});
return await response.json();
};
window.addEventListener('message', async (event) => {
const { id, interface: iface, method, params } = event.data || {};
let has_file = false;
for (const k in params) {
if (params[k] instanceof File) {
has_file = true;
break;
}
}
const result = has_file
? await fcall({ interface_name: iface, method_name: method, params })
: await call({ interface_name: iface, method_name: method, params });
if (event.source) {
event.source.postMessage({ id, result }, event.origin);
}
});
})();
`;
const XD_HTML = `<!DOCTYPE html>
<html>
<head>
<title>Puter Driver API</title>
<script>
document.addEventListener('DOMContentLoaded', function () {
${XD_SCRIPT}
});
</script>
</head>
<body></body>
</html>`;
// -- Controller ------------------------------------------------------
/**
* Routes driver RPC calls through a unified HTTP surface.
*
* - `POST /drivers/call` — invoke `<iface>.<method>(args)` on the
* registered driver after a per-actor permission + rate-limit check.
* Stream-shaped results are piped directly; everything else is
* returned as JSON.
* - `GET /drivers/list-interfaces` — enumerate registered driver
* interfaces with their default + alternate implementations.
* - `GET /drivers/xd` — legacy iframe bridge; serves an HTML page that
* proxies `postMessage` RPCs to `/drivers/call` on the same origin.
*
* Holds an internal iface → driverName → instance map built at
* construction time from `this.drivers`. Extensions that register
* additional drivers end up in that bag before this controller is
* instantiated, so they show up here automatically.
*/
/**
* Catch-all upstream-error translator for the driver boundary.
*
* Drivers that wrap a third-party SDK (OpenAI, Anthropic, etc.) often
* let the SDK's own error class bubble — those carry an HTTP `.status`
* but are plain `Error` subclasses, not `HttpError`s, so they would
* otherwise hit the global error handler as unexpected 500s and page
* PagerDuty. Repackage them with `upstream_*` legacy codes so the
* alarm gate (server.ts) treats them as upstream failures and only
* pages on the two we actually care about (rate-limit / auth).
*
* Status extraction covers the shapes we've seen in the wild:
* - `.status` / `.statusCode` (OpenAI / Anthropic / Together / Google GenAI)
* - `.response.status` (Replicate's `ApiError`)
* - `.$metadata.httpStatusCode` (AWS SDK v3, e.g. Polly)
* - status sniffed from the message string (last resort, for providers
* that re-wrap their SDK error in `new Error(msg)` and lose the field)
*
* `HttpError`s thrown by drivers pass through untouched.
*/
const extractUpstreamStatus = (e: {
status?: number;
statusCode?: number;
@@ -273,11 +165,6 @@ export class DriverController extends PuterController {
{ subdomain: 'api', requireAuth: true },
this.#handleListInterfaces,
);
router.get(
'/xd',
{ subdomain: 'api', requireAuth: true },
this.#handleXd,
);
}
// -- Handlers ----------------------------------------------------
@@ -336,7 +223,16 @@ export class DriverController extends PuterController {
| PermissionService
| undefined;
if (permService) {
const permKey = `service:${resolvedDriverName}:ii:${ifaceName}`;
// Build via PermissionUtil.join so any `:` in a driver or
// interface name is escaped — raw interpolation would let a
// crafted name shift permission-segment boundaries and match
// a broader/narrower parent than intended in the scan logic.
const permKey = PermissionUtil.join(
'service',
String(resolvedDriverName),
'ii',
ifaceName,
);
const hasPermission = await permService.check(
req.actor,
permKey,
@@ -439,14 +335,91 @@ export class DriverController extends PuterController {
// a stale value from a prior call.
Context.set('driverName', requestedDriver);
// Per-method lifecycle events, scoped to `driver.<iface>.<method>`.
// Subscribers can listen on `driver.*`, `driver.<iface>.*`, or the
// exact key. `before` is emitted via `emitAndWait` so a listener may
// veto the call by setting `allow = false` (emits `reject`, throws
// 403); otherwise `after`/`error` carry the result/error + duration.
const actor = req.actor ? actorUid(req.actor) : undefined;
const resolved = String(resolvedDriverName);
const beforeEvent = {
phase: 'before' as const,
iface: ifaceName,
method,
driver: resolved,
actor: req.actor,
actorUid: actor,
args,
allow: true as boolean,
rejectReason: undefined as string | undefined,
};
await this.clients.event?.emitAndWait(
`driver.${ifaceName}.${method}.before`,
beforeEvent,
{},
);
if (beforeEvent.allow === false) {
this.clients.event?.emit(
`driver.${ifaceName}.${method}.reject`,
{
phase: 'reject',
iface: ifaceName,
method,
driver: resolved,
actor: req.actor,
actorUid: actor,
args,
rejectReason: beforeEvent.rejectReason,
},
{},
);
throw new HttpError(
403,
beforeEvent.rejectReason ??
`Blocked by policy: ${ifaceName}:${method}`,
{ legacyCode: 'forbidden' },
);
}
// Drivers read actor/context via the Context API — no drilled args.
const startedAt = Date.now();
let result;
try {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
result = await (fn as (...x: unknown[]) => any).call(driver, args);
} catch (e) {
this.clients.event?.emit(
`driver.${ifaceName}.${method}.error`,
{
phase: 'error',
iface: ifaceName,
method,
driver: resolved,
actor: req.actor,
actorUid: actor,
args,
error: e,
durationMs: Date.now() - startedAt,
},
{},
);
throw translateProviderError(e);
}
this.clients.event?.emit(
`driver.${ifaceName}.${method}.after`,
{
phase: 'after',
iface: ifaceName,
method,
driver: resolved,
actor: req.actor,
actorUid: actor,
args,
result,
durationMs: Date.now() - startedAt,
},
{},
);
if (isDriverStreamResult(result)) {
res.setHeader('Content-Type', result.content_type);
@@ -489,11 +462,6 @@ export class DriverController extends PuterController {
res.json(out);
};
#handleXd = (_req: Request, res: Response): void => {
res.type('text/html');
res.send(XD_HTML);
};
// -- Internals ---------------------------------------------------
#buildIfaceMap(): void {
+309 -15
View File
@@ -18,6 +18,7 @@
*/
import type { Request, Response } from 'express';
import type { Readable } from 'node:stream';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import type { Actor } from '../../core/actor.js';
@@ -124,6 +125,14 @@ const makeRes = () => {
const withActor = async <T>(actor: Actor, fn: () => Promise<T>): Promise<T> =>
runWithContext({ actor }, fn);
const streamToString = async (stream: Readable): Promise<string> => {
const chunks: Buffer[] = [];
for await (const chunk of stream) {
chunks.push(Buffer.from(chunk as Buffer));
}
return Buffer.concat(chunks).toString('utf8');
};
// ── /startBatchWrite ────────────────────────────────────────────────
describe('FSController.startBatchWrites', () => {
@@ -434,6 +443,65 @@ describe('FSController.completeBatchWrites', () => {
expect(finalized?.wasOverwrite).toBe(true);
});
// Regression: a signed (direct-to-S3) upload could declare a tiny size
// and PUT far more — the presigned URL doesn't bound the body. The
// completion path must reconcile the recorded size against the object's
// true size, or storage accounting is permanently understated and the
// quota is bypassable.
it('reconciles the recorded size to the real uploaded bytes (ignores under-declared size)', async () => {
const { actor, userId } = await makeUser();
const username = actor.user!.username!;
const target = `/${username}/Documents/under-declared.bin`;
// Declare 1 byte.
const start = makeRes();
await withActor(actor, () =>
controller.startBatchWrites(
makeReq<SignedWriteRequest[]>({
body: [{ fileMetadata: { path: target, size: 1 } }],
actor,
}),
start.res,
),
);
const [started] = start.captured.body as SignedWriteResponse[];
// Actually upload 4096 bytes to the session's object key (simulating
// a client that PUTs more than it declared via the signed URL).
const realBytes = Buffer.alloc(4096, 0x41);
await server.stores.s3Object.uploadFromServer(
{
bucket: started!.bucket,
objectKey: started!.objectKey,
contentType: 'application/octet-stream',
body: realBytes,
contentLength: realBytes.byteLength,
},
started!.bucketRegion,
);
const complete = makeRes();
await withActor(actor, () =>
controller.completeBatchWrites(
makeReq<CompleteWriteRequest[]>({
body: [{ uploadId: started!.sessionId }],
actor,
}),
complete.res,
),
);
const entry = await server.stores.fsEntry.getEntryByPath(target);
expect(entry).not.toBeNull();
// Recorded size is the true 4096 bytes, not the declared 1.
expect(entry?.size).toBe(4096);
// And the user's accounted usage reflects the real bytes.
const allowance =
await server.stores.fsEntry.getUserStorageAllowance(userId);
expect(allowance.curr).toBeGreaterThanOrEqual(4096);
});
it('emits updated events with GUI metadata when overwriting via batch completion', async () => {
const { actor, userId } = await makeUser();
const username = actor.user!.username!;
@@ -490,9 +558,7 @@ describe('FSController.completeBatchWrites', () => {
.body as SignedWriteResponse[];
const emitSpy = vi.spyOn(server.clients.event, 'emit');
let updatedCall:
| (typeof emitSpy.mock.calls)[number]
| undefined;
let updatedCall: (typeof emitSpy.mock.calls)[number] | undefined;
try {
await withActor(actor, () =>
controller.completeBatchWrites(
@@ -667,9 +733,9 @@ describe('FSController.statEntry', () => {
...makeReq({ body: { path: '/x' }, actor }),
actor: undefined,
} as unknown as Request;
await expect(
controller.statEntry(req, res),
).rejects.toMatchObject({ statusCode: 401 });
await expect(controller.statEntry(req, res)).rejects.toMatchObject({
statusCode: 401,
});
});
});
@@ -839,9 +905,7 @@ describe('FSController.searchEntries', () => {
).toBe(true);
}
expect(
results.some(
(r) => r.path === `/${username}/Documents/${needle}`,
),
results.some((r) => r.path === `/${username}/Documents/${needle}`),
).toBe(false);
});
@@ -953,6 +1017,82 @@ describe('FSController.mkdirEntry', () => {
expect(body.isDir).toBe(true);
});
it('dedupes an existing directory when dedupe_name is true', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const target = `/${username}/Documents/hello`;
await withActor(actor, () =>
controller.mkdirEntry(
makeReq({
body: { path: target },
actor,
}),
makeRes().res,
),
);
const { res, captured } = makeRes();
await withActor(actor, () =>
controller.mkdirEntry(
makeReq({
body: { path: target, dedupe_name: true },
actor,
}),
res,
),
);
const body = captured.body as {
path: string;
name: string;
isDir: boolean;
};
expect(body.path).toBe(`/${username}/Documents/hello (1)`);
expect(body.name).toBe('hello (1)');
expect(body.isDir).toBe(true);
expect(
await server.stores.fsEntry.getEntryByPath(
`/${username}/Documents/hello (1)`,
),
).toMatchObject({ isDir: true });
});
it('requires parent write when deduping an existing directory', async () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appUid = `app-mkdir-${uuidv4()}`;
const appActor: Actor = { ...userActor, app: { uid: appUid } };
const target = `/${username}/AppData/${appUid}`;
await withActor(userActor, () =>
controller.mkdirEntry(
makeReq({
body: { path: target },
actor: userActor,
}),
makeRes().res,
),
);
await expect(
withActor(appActor, () =>
controller.mkdirEntry(
makeReq({
body: { path: target, dedupe_name: true },
actor: appActor,
}),
makeRes().res,
),
),
).rejects.toMatchObject({ statusCode: 404 });
expect(
await server.stores.fsEntry.getEntryByPath(
`/${username}/AppData/${appUid} (1)`,
),
).toBeNull();
});
it('expands ~/ in the path to the user home', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
@@ -1183,10 +1323,6 @@ describe('FSController.copyEntry', () => {
// ── /read (readEntry, full read) ────────────────────────────────────
describe('FSController.readEntry (file streaming)', () => {
// makeRes here adds a real Writable surface so that
// `pipeline(download.body, res)` inside the controller can pipe
// the in-memory S3 stream into the test's response and capture
// bytes for assertions.
const makeStreamingRes = () => {
const captured = {
statusCode: 200,
@@ -1194,7 +1330,8 @@ describe('FSController.readEntry (file streaming)', () => {
bodyChunks: [] as Buffer[],
};
// eslint-disable-next-line @typescript-eslint/no-require-imports
const { Writable } = require('node:stream') as typeof import('node:stream');
const { Writable } =
require('node:stream') as typeof import('node:stream');
const writable = new Writable({
write(chunk: Buffer, _enc, cb) {
captured.bodyChunks.push(chunk);
@@ -1254,7 +1391,9 @@ describe('FSController.readEntry (file streaming)', () => {
// Pipeline awaits the stream-end on success.
expect(captured.statusCode).toBe(200);
expect(captured.headers['Content-Type']).toMatch(/text\/plain/);
expect(captured.headers['Content-Length']).toBe(String(body.byteLength));
expect(captured.headers['Content-Length']).toBe(
String(body.byteLength),
);
expect(captured.headers['Content-Disposition']).toMatch(
/inline; filename=/,
);
@@ -1698,3 +1837,158 @@ describe('FSController.mkshortcutEntry', () => {
expect(body.isShortcut).toBe(true);
});
});
describe('FSController metadata.objectKey injection', () => {
const writeFile = async (
actor: Actor,
path: string,
content: string,
metadata?: Record<string, unknown>,
) => {
await withActor(actor, () =>
controller.write(
makeReq({
body: {
fileMetadata: {
path,
size: Buffer.byteLength(content),
contentType: 'text/plain',
overwrite: true,
...(metadata ? { metadata } : {}),
},
fileContent: content,
encoding: 'utf8',
},
actor,
}) as unknown as Request<
Record<string, never>,
null,
import('./requestTypes.js').WriteRequest
>,
makeRes().res,
),
);
const entry = await server.stores.fsEntry.getEntryByPath(path, {
skipCache: true,
});
if (!entry) throw new Error(`entry not found after write: ${path}`);
return entry;
};
it("does not stream another user's file when a client injects metadata.objectKey on write", async () => {
const victim = await makeUser();
const attacker = await makeUser();
const victimSecret = 'VICTIM-TOP-SECRET-PAYLOAD';
const attackerDecoy = 'attacker-own-decoy-bytes';
const victimEntry = await writeFile(
victim.actor,
`/${victim.actor.user!.username}/Documents/secret.txt`,
victimSecret,
);
const victimRead = await server.services.fs.readContent(victimEntry);
expect(await streamToString(victimRead.body)).toBe(victimSecret);
const attackerEntry = await writeFile(
attacker.actor,
`/${attacker.actor.user!.username}/Documents/loot.txt`,
attackerDecoy,
{ objectKey: victimEntry.uuid },
);
const persisted = attackerEntry.metadata
? (JSON.parse(attackerEntry.metadata) as Record<string, unknown>)
: {};
expect(persisted.objectKey).toBeUndefined();
const attackerRead =
await server.services.fs.readContent(attackerEntry);
const got = await streamToString(attackerRead.body);
expect(got).toBe(attackerDecoy);
expect(got).not.toBe(victimSecret);
});
it('read path ignores a divergent metadata.objectKey on an already-poisoned row', async () => {
const victim = await makeUser();
const attacker = await makeUser();
const victimSecret = 'VICTIM-SECRET-FOR-POISON-TEST';
const attackerDecoy = 'attacker-decoy-for-poison-test';
const victimEntry = await writeFile(
victim.actor,
`/${victim.actor.user!.username}/Documents/secret2.txt`,
victimSecret,
);
const attackerEntry = await writeFile(
attacker.actor,
`/${attacker.actor.user!.username}/Documents/loot2.txt`,
attackerDecoy,
);
await server.stores.fsEntry.updateEntry(attackerEntry.uuid, {
metadata: JSON.stringify({ objectKey: victimEntry.uuid }),
});
const poisoned = await server.stores.fsEntry.getEntryByPath(
attackerEntry.path,
{ skipCache: true },
);
if (!poisoned) throw new Error('poisoned entry not found');
expect(
(JSON.parse(poisoned.metadata!) as { objectKey: string }).objectKey,
).toBe(victimEntry.uuid);
const read = await server.services.fs.readContent(poisoned);
expect(await streamToString(read.body)).toBe(attackerDecoy);
});
it('scrubs objectKey from move newMetadata while preserving legit trash metadata', async () => {
const victim = await makeUser();
const attacker = await makeUser();
const victimSecret = 'VICTIM-SECRET-FOR-MOVE-TEST';
const attackerDecoy = 'attacker-decoy-for-move-test';
const username = attacker.actor.user!.username!;
const victimEntry = await writeFile(
victim.actor,
`/${victim.actor.user!.username}/Documents/secret3.txt`,
victimSecret,
);
const attackerEntry = await writeFile(
attacker.actor,
`/${username}/Documents/loot3.txt`,
attackerDecoy,
);
const documents = await server.stores.fsEntry.getEntryByPath(
`/${username}/Documents`,
{ skipCache: true },
);
if (!documents) throw new Error('Documents dir not found');
const moved = await withActor(attacker.actor, () =>
server.services.fs.move(attacker.userId, {
source: attackerEntry,
destinationParent: documents,
newName: 'loot3-moved.txt',
newMetadata: {
original_path: `/${username}/Documents/loot3.txt`,
trashed_ts: 1700000000,
objectKey: victimEntry.uuid,
},
}),
);
const persisted = JSON.parse(moved.metadata!) as Record<
string,
unknown
>;
expect(persisted.objectKey).toBeUndefined();
expect(persisted.original_path).toBe(
`/${username}/Documents/loot3.txt`,
);
expect(persisted.trashed_ts).toBe(1700000000);
const read = await server.services.fs.readContent(moved);
expect(await streamToString(read.body)).toBe(attackerDecoy);
});
});
+22 -3
View File
@@ -37,6 +37,7 @@ import {
runWithConcurrencyLimit,
runWithConcurrencyLimitSettled,
} from '../../util/concurrency.js';
import { applyInlineContentSecurity } from '../../util/inlineContentSecurity.js';
import { PuterController } from '../types.js';
import { FS_COSTS } from './costs.js';
import type {
@@ -964,8 +965,13 @@ export class FSController extends PuterController {
range,
});
if (download.contentType)
if (download.contentType) {
res.setHeader('Content-Type', download.contentType);
// Stored type is uploader-controlled and served inline on the
// api origin — sandbox active-document types (HTML/SVG/XML) so
// embedded scripts can't run here. Inert types are untouched.
applyInlineContentSecurity(res, download.contentType);
}
if (download.contentLength !== null)
res.setHeader('Content-Length', String(download.contentLength));
if (download.contentRange)
@@ -1039,13 +1045,15 @@ export class FSController extends PuterController {
legacyCode: 'bad_request',
});
const dedupeName =
this.#toBoolean(body.dedupe_name ?? body.dedupeName) ?? false;
await this.#assertCanCreate(actor, path);
if (dedupeName) await this.#assertCanDedupeCreate(actor, path);
const entry = await this.services.fs.mkdir(userId, {
path,
overwrite: this.#toBoolean(body.overwrite) ?? false,
dedupeName:
this.#toBoolean(body.dedupe_name ?? body.dedupeName) ?? false,
dedupeName,
createMissingParents:
this.#toBoolean(
body.create_missing_parents ??
@@ -1310,6 +1318,17 @@ export class FSController extends PuterController {
await this.#assertAccess(actor, parentForCheck, 'write');
}
async #assertCanDedupeCreate(actor: Actor, targetPath: string) {
const existing = await this.stores.fsEntry.getEntryByPath(targetPath);
if (!existing) return;
const parent = pathPosix.dirname(targetPath);
await this.#assertAccess(
actor,
parent === '/' ? targetPath : parent,
'write',
);
}
async #assertAccess(
actor: Actor,
path: string,
@@ -206,6 +206,78 @@ describe('LegacyFSController.mkdir', () => {
expect(fetched?.isDir).toBe(true);
});
it('dedupes an existing directory when dedupe_name is true', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const parent = `/${username}/Documents`;
await withActor(actor, () =>
controller.mkdir(
makeReq({
body: { parent, path: 'hello' },
actor,
}),
makeRes().res,
),
);
const { res, captured } = makeRes();
await withActor(actor, () =>
controller.mkdir(
makeReq({
body: { parent, path: 'hello', dedupe_name: true },
actor,
}),
res,
),
);
const body = captured.body as Record<string, unknown>;
expect(body).toMatchObject({
path: `${parent}/hello (1)`,
name: 'hello (1)',
is_dir: true,
});
expect(
await server.stores.fsEntry.getEntryByPath(`${parent}/hello (1)`),
).toMatchObject({ isDir: true });
});
it('requires parent write when deduping an existing directory', async () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appUid = `app-legacy-mkdir-${uuidv4()}`;
const appActor: Actor = { ...userActor, app: { uid: appUid } };
const parent = `/${username}/AppData`;
await withActor(userActor, () =>
controller.mkdir(
makeReq({
body: { parent, path: appUid },
actor: userActor,
}),
makeRes().res,
),
);
await expect(
withActor(appActor, () =>
controller.mkdir(
makeReq({
body: { parent, path: appUid, dedupe_name: true },
actor: appActor,
}),
makeRes().res,
),
),
).rejects.toMatchObject({ statusCode: 404 });
expect(
await server.stores.fsEntry.getEntryByPath(
`${parent}/${appUid} (1)`,
),
).toBeNull();
});
it("rejects writing into another user's home with a 4xx", async () => {
const a = await makeUser();
const b = await makeUser();
@@ -1010,6 +1082,76 @@ describe('LegacyFSController.sign', () => {
),
).rejects.toMatchObject({ statusCode: 404 });
});
it('refuses an app actor signing for a different app (403)', async () => {
// An app-under-user actor may only mint a token for its own app;
// requesting a different app's UID is rejected.
const { actor: userActor } = await makeUser();
const targetApp = await (
server.stores.app.create as unknown as (
fields: Record<string, unknown>,
opts: { ownerUserId: number },
) => Promise<{ uid: string; id: number }>
)(
{
name: `victim-${uuidv4()}`,
title: 'Victim app',
index_url: 'https://example.test/victim.html',
},
{ ownerUserId: userActor.user!.id! },
);
const attackerActor: Actor = {
...userActor,
app: { uid: `attacker-${uuidv4()}` },
};
const { res } = makeRes();
await expect(
withActor(attackerActor, () =>
controller.sign(
makeReq({
body: {
items: [{}],
app_uid: targetApp.uid,
},
actor: attackerActor,
}),
res,
),
),
).rejects.toMatchObject({ statusCode: 403, legacyCode: 'forbidden' });
});
it('lets an app actor sign for its own app', async () => {
const { actor: userActor } = await makeUser();
const ownApp = await (
server.stores.app.create as unknown as (
fields: Record<string, unknown>,
opts: { ownerUserId: number },
) => Promise<{ uid: string; id: number }>
)(
{
name: `self-${uuidv4()}`,
title: 'Self app',
index_url: 'https://example.test/self.html',
},
{ ownerUserId: userActor.user!.id! },
);
const appActor: Actor = { ...userActor, app: { uid: ownApp.uid } };
const { res, captured } = makeRes();
await withActor(appActor, () =>
controller.sign(
makeReq({
body: { items: [{}], app_uid: ownApp.uid },
actor: appActor,
}),
res,
),
);
const body = captured.body as { token?: string };
expect(typeof body.token).toBe('string');
});
});
// ── writeFile (validation paths) ────────────────────────────────────
@@ -1089,6 +1231,85 @@ describe('LegacyFSController.openItem', () => {
});
});
// ── openItem: write_url stripping for read-only callers ─────────────
//
// Mirrors `/sign` and `/readdir`, which already strip `write_url` when the
// caller only proved read. Without these gates, /open_item handed out a
// valid write signature to read-only sharees — `/writeFile`'s own ACL
// re-check would still reject the write, but the leak shape (a signed
// write URL escaping the access boundary) is the same one those other
// endpoints already defend against.
describe('LegacyFSController.openItem (write_url stripping)', () => {
beforeAll(() => {
(
controller as unknown as { config: { api_base_url?: string } }
).config.api_base_url = 'http://api.test.local';
});
it('returns write_url for the owner (who has write)', async () => {
const { actor } = await makeUser();
const target = `/${actor.user!.username}/Documents/owned.txt`;
await withActor(actor, () =>
controller.touch(
makeReq({ body: { path: target }, actor }),
makeRes().res,
),
);
const { res, captured } = makeRes();
await withActor(actor, () =>
controller.openItem(
makeReq({ body: { path: target }, actor }),
res,
),
);
const body = captured.body as {
signature: { read_url?: string; write_url?: string };
};
expect(body.signature.read_url).toBeDefined();
expect(body.signature.write_url).toBeDefined();
expect(body.signature.write_url).toContain('writeFile');
});
it('strips write_url for a read-only sharee (the fix)', async () => {
const victim = await makeUser();
const attacker = await makeUser();
const target = `/${victim.actor.user!.username}/Documents/shared-ro.txt`;
await withActor(victim.actor, () =>
controller.touch(
makeReq({ body: { path: target }, actor: victim.actor }),
makeRes().res,
),
);
const entry = await server.stores.fsEntry.getEntryByPath(target);
await server.services.permission.grantUserUserPermission(
victim.actor,
attacker.actor.user!.username!,
`fs:${entry!.uuid}:read`,
{},
);
const { res, captured } = makeRes();
await withActor(attacker.actor, () =>
controller.openItem(
makeReq({
body: { uid: entry!.uuid },
actor: attacker.actor,
}),
res,
),
);
const body = captured.body as {
signature: { read_url?: string; write_url?: string };
};
// Sharee still gets read_url (they have read).
expect(body.signature.read_url).toBeDefined();
// …but write_url is stripped (they don't have write).
expect(body.signature.write_url).toBeUndefined();
});
});
// ── requestAppRootDir ───────────────────────────────────────────────
describe('LegacyFSController.requestAppRootDir', () => {
@@ -29,6 +29,7 @@ import type { PuterRouter } from '../../core/http/PuterRouter.js';
import type { ACLService } from '../../services/acl/ACLService.js';
import type { SignedFile } from '../../util/fileSigning.js';
import { verifySignature } from '../../util/fileSigning.js';
import { applyInlineContentSecurity } from '../../util/inlineContentSecurity.js';
import { PuterController } from '../types.js';
import { FS_COSTS } from './costs.js';
import {
@@ -45,14 +46,6 @@ import {
} from './legacyFsHelpers.js';
import { RouteOptions } from '../../core/http/index.js';
/**
* Legacy FS routes, implemented as thin shims over `FSService`.
*
* Each shim parses the request shape (FSNodeParam-style `{ path, uid, id }`
* or `{ parent, name }`), invokes the service method, and returns the
* snake_case response clients expect.
*/
type RouterCache = Map<string, RequestHandler | null>;
const additionalRoutePaths: Record<string, string> = {};
@@ -146,6 +139,11 @@ export class LegacyFSController extends PuterController {
{
subdomain: ['api', ''],
requireUserActor: true,
// The user's own credential may download their files; apps and
// scoped tokens stay blocked. antiCsrf still protects the
// cookie-authed GUI path — a bearer-token PAT is exempt below
// (CSRF can't forge a header-credentialed request).
allowFullAccessToken: true,
requireVerified: true,
antiCsrf: true,
},
@@ -427,17 +425,33 @@ export class LegacyFSController extends PuterController {
const normalizedTarget = targetPath.startsWith('/')
? targetPath
: `/${targetPath}`;
const dedupeName =
getBoolean(body, 'dedupe_name', 'change_name') ?? false;
await assertCanCreate(
this.services.acl,
this.services.fs,
actor,
normalizedTarget,
);
if (dedupeName) {
const existing =
await this.stores.fsEntry.getEntryByPath(normalizedTarget);
if (existing) {
const parent = pathPosix.dirname(normalizedTarget);
await assertAccess(
this.services.acl,
this.services.fs,
actor,
parent === '/' ? normalizedTarget : parent,
'write',
);
}
}
const entry = await this.services.fs.mkdir(userId, {
path: targetPath,
overwrite: getBoolean(body, 'overwrite') ?? false,
dedupeName: getBoolean(body, 'dedupe_name', 'change_name') ?? false,
dedupeName,
createMissingParents:
getBoolean(
body,
@@ -1336,8 +1350,15 @@ export class LegacyFSController extends PuterController {
query.download === '1' ||
query.download === true;
if (download.contentType)
if (download.contentType) {
res.setHeader('Content-Type', download.contentType);
// Uploader-controlled type served inline on the file origin —
// sandbox active-document types so an uploaded HTML/SVG can't
// execute scripts here. Mirrors FSController /fs/read.
if (!wantsAttachment) {
applyInlineContentSecurity(res, download.contentType);
}
}
if (download.contentLength !== null)
res.setHeader('Content-Length', String(download.contentLength));
if (download.contentRange)
@@ -1391,6 +1412,20 @@ export class LegacyFSController extends PuterController {
'read',
);
// Downgrade the envelope when the caller only proved read.
// `/writeFile`'s ACL re-check would still block the write, but
// returning `write_url` to a read-only caller is the same
// privilege-leak shape that `/sign` and `/readdir` strip.
const writeOk = await this.services.acl.check(
actor,
{
path: entry.path,
resolveAncestors: () =>
this.services.fs.getAncestorChain(entry.path),
},
'write',
);
const suggested =
(await this.services.suggestedApps?.getSuggestedApps({
name: entry.name,
@@ -1417,7 +1452,13 @@ export class LegacyFSController extends PuterController {
}
const signingCfg = signingConfigFromAppConfig(this.config);
const signature = { ...signEntry(entry, signingCfg), path: entry.path };
const signed = signEntry(entry, signingCfg);
const signature = writeOk
? { ...signed, path: entry.path }
: (() => {
const { write_url: _, ...rest } = signed;
return { ...rest, path: entry.path };
})();
res.json({
signature,
token,
@@ -1582,17 +1623,6 @@ export class LegacyFSController extends PuterController {
download.body.pipe(res);
};
// Helpers for writeFile
// -- GUI event emission -------------------------------------------
//
// Fire-and-forget `outer.gui.item.*` events so SocketService,
// BroadcastService, WorkerDriver (hot-reload), and cache-invalidation
// listeners pick up mutations made through the legacy (bare-path) routes.
// FSController (v2-native /fs/* routes) emits these from its own handlers;
// LegacyFSController delegates to the same FSService but needs its
// own emissions because the service layer deliberately doesn't emit GUI
// events (that's a controller concern).
async #emitGuiEvent(
eventName:
| 'outer.gui.item.added'
@@ -44,6 +44,7 @@ export class HostingController extends PuterController {
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
requireVerified: true,
},
async (req, res) => {
@@ -38,7 +38,11 @@ export class NotificationController extends PuterController {
* POST /notif/mark-ack — user dismissed a notification.
* Sets `acknowledged` timestamp; pushes ack event to sockets.
*/
@Post('/mark-ack', { subdomain: 'api', requireUserActor: true })
@Post('/mark-ack', {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
})
async markAck(req: Request, res: Response): Promise<void> {
const uid = req.body?.uid;
if (typeof uid !== 'string' || uid.length === 0) {
@@ -77,7 +81,11 @@ export class NotificationController extends PuterController {
* POST /notif/mark-read — user saw a notification.
* Sets `shown` timestamp; pushes ack event to sockets.
*/
@Post('/mark-read', { subdomain: 'api', requireUserActor: true })
@Post('/mark-read', {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
})
async markRead(req: Request, res: Response): Promise<void> {
const uid = req.body?.uid;
if (typeof uid !== 'string' || uid.length === 0) {
+21 -1
View File
@@ -17,6 +17,7 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { createHmac, randomBytes, timingSafeEqual } from 'node:crypto';
import type { Request, Response } from 'express';
import type { Actor } from '../../core/actor.js';
import { HttpError } from '../../core/http/HttpError.js';
@@ -24,6 +25,21 @@ import type { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterController } from '../types.js';
import { PEER_COSTS } from './costs.js';
/**
* Constant-time secret comparison for the internal-auth header. HMAC both
* sides under a random per-process key to a fixed 32-byte digest first:
* this avoids leaking length via an early-return and sidesteps
* `timingSafeEqual`'s equal-length requirement for arbitrary-length
* inputs. The key need not persist — it only has to be unknown to the
* attacker for the duration of the comparison.
*/
const COMPARE_KEY = randomBytes(32);
const secretsEqual = (a: string, b: string): boolean => {
const ha = createHmac('sha256', COMPARE_KEY).update(a).digest();
const hb = createHmac('sha256', COMPARE_KEY).update(b).digest();
return timingSafeEqual(ha, hb);
};
/**
* Encode a UUID (or `app-<uuid>` UID) as base64url with no padding.
* Strips an `app-` prefix and dashes, then reinterprets the hex bytes.
@@ -170,7 +186,11 @@ export class PeerController extends PuterController {
}
const expectedSecret = cfg.internal_auth_secret;
const header = req.headers['x-puter-internal-auth'];
if (!expectedSecret || header !== expectedSecret) {
if (
!expectedSecret ||
typeof header !== 'string' ||
!secretsEqual(header, expectedSecret)
) {
throw new HttpError(403, 'Forbidden', { legacyCode: 'forbidden' });
}
@@ -188,10 +188,11 @@ describe('PuterAIController.registerRoutes', () => {
]),
);
// The four upstream-proxy routes are user-only: app actors must
// call puter-chat-completion directly. The `requireUserActor`
// route option wires up the gate that enforces this, so each
// proxy route gets the assertion (not just chat/completions).
// The four upstream-proxy routes reject third-party apps (they must
// call puter-chat-completion directly), but admit the user's own
// full-access personal access token: `requireUserActor` keeps apps out
// and `allowFullAccessToken` opens the gate to a full-access PAT. Each
// proxy route carries both flags.
const userOnlyPaths = [
'/puterai/openai/v1/chat/completions',
'/puterai/openai/v1/completions',
@@ -203,6 +204,7 @@ describe('PuterAIController.registerRoutes', () => {
expect(route?.opts).toEqual({
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
});
}
const modelsRoute = calls.find(
@@ -1504,6 +1506,212 @@ describe('PuterAIController.anthropicMessages streaming + helpers', () => {
});
});
// ── Inline compaction ───────────────────────────────────────────────
describe('PuterAIController inline compaction', () => {
// Extract the single `event: compaction\ndata: {...}` SSE frame.
const compactionFrame = (out: string): string | null => {
const m = out.match(/event: compaction\ndata: [^\n]*\n\n/);
return m ? m[0] : null;
};
it('forwards the compaction opt-in to the driver (/responses)', async () => {
const spy = stubChatComplete({
message: { role: 'assistant', content: 'ok' },
finish_reason: 'stop',
});
await controller.openaiResponses(
makeReq({
body: { model: 'gpt-test', input: 'hi', compaction: true },
actor: makeUserActor(),
}),
makeRes().res,
);
expect(spy.mock.calls[0]![0].compaction).toBe(true);
});
it('round-trips a compaction `input` item into a messages compaction item', async () => {
const spy = stubChatComplete({
message: { role: 'assistant', content: 'ack' },
finish_reason: 'stop',
});
await controller.openaiResponses(
makeReq({
body: {
model: 'gpt-test',
input: [
{
type: 'compaction',
id: 'cmpct_1',
encrypted_content: 'ENC',
},
],
},
actor: makeUserActor(),
}),
makeRes().res,
);
expect(spy.mock.calls[0]![0].messages).toContainEqual({
type: 'compaction',
id: 'cmpct_1',
encrypted_content: 'ENC',
});
});
it('emits a canonical compaction SSE event and output item (/responses stream)', async () => {
stubChatComplete({
dataType: 'stream',
content_type: 'application/x-ndjson',
stream: ndjsonStreamFrom([
{ type: 'text', text: 'hi' },
{ type: 'compaction', id: 'cmpct_9', encrypted_content: 'ENC9' },
{ type: 'usage', usage: { prompt_tokens: 1, completion_tokens: 1 } },
]),
});
const { res, captured } = makeRes();
await controller.openaiResponses(
makeReq({
body: {
model: 'gpt-test',
input: 'hi',
stream: true,
compaction: true,
},
actor: makeUserActor(),
}),
res,
);
await new Promise<void>((resolve) => setImmediate(resolve));
const out = captured.written.join('');
expect(compactionFrame(out)).toBe(
'event: compaction\ndata: {"type":"compaction","id":"cmpct_9","encrypted_content":"ENC9"}\n\n',
);
// Native shape also lands in the final response.completed output[].
const completed = out
.split('event: response.completed\n')[1]
?.split('\n\n')[0];
expect(completed).toContain('"type":"compaction"');
expect(completed).toContain('"encrypted_content":"ENC9"');
});
it('emits the compaction item in non-streaming /responses output', async () => {
stubChatComplete({
message: { role: 'assistant', content: 'done' },
finish_reason: 'stop',
compaction: { id: 'cmpct_n', encrypted_content: 'ENCN' },
});
const { res, captured } = makeRes();
await controller.openaiResponses(
makeReq({
body: { model: 'gpt-test', input: 'hi', compaction: true },
actor: makeUserActor(),
}),
res,
);
const body = captured.body as { output: Array<Record<string, unknown>> };
expect(body.output).toContainEqual(
expect.objectContaining({
type: 'compaction',
encrypted_content: 'ENCN',
}),
);
});
it('emits an identical canonical compaction SSE event on the Anthropic surface', async () => {
// /responses frame
stubChatComplete({
dataType: 'stream',
content_type: 'application/x-ndjson',
stream: ndjsonStreamFrom([
{ type: 'compaction', id: 'cmpct_x', encrypted_content: 'ENCX' },
]),
});
const r1 = makeRes();
await controller.openaiResponses(
makeReq({
body: { model: 'gpt-test', input: 'hi', stream: true },
actor: makeUserActor(),
}),
r1.res,
);
await new Promise<void>((resolve) => setImmediate(resolve));
// /anthropic/v1/messages frame
stubChatComplete({
dataType: 'stream',
content_type: 'application/x-ndjson',
stream: ndjsonStreamFrom([
{ type: 'compaction', id: 'cmpct_x', encrypted_content: 'ENCX' },
]),
});
const r2 = makeRes();
await controller.anthropicMessages(
makeReq({
body: {
model: 'claude-test',
messages: [{ role: 'user', content: 'hi' }],
stream: true,
},
actor: makeUserActor(),
}),
r2.res,
);
await new Promise<void>((resolve) => setImmediate(resolve));
const a = compactionFrame(r1.captured.written.join(''));
const b = compactionFrame(r2.captured.written.join(''));
expect(a).not.toBeNull();
expect(a).toBe(b); // byte-identical streaming shape across providers
});
it('renders a native compaction content block in non-streaming /messages', async () => {
stubChatComplete({
message: { role: 'assistant', content: 'done' },
finish_reason: 'stop',
compaction: { id: 'cmpct_m', encrypted_content: 'ENCM' },
});
const { res, captured } = makeRes();
await controller.anthropicMessages(
makeReq({
body: {
model: 'claude-test',
messages: [{ role: 'user', content: 'hi' }],
},
actor: makeUserActor(),
}),
res,
);
const body = captured.body as { content: Array<Record<string, unknown>> };
expect(body.content).toContainEqual({
type: 'compaction',
id: 'cmpct_m',
encrypted_content: 'ENCM',
});
});
it('does not emit compaction frames for a normal stream (regression)', async () => {
stubChatComplete({
dataType: 'stream',
content_type: 'application/x-ndjson',
stream: ndjsonStreamFrom([
{ type: 'text', text: 'hello' },
{ type: 'usage', usage: { prompt_tokens: 1, completion_tokens: 1 } },
]),
});
const { res, captured } = makeRes();
await controller.openaiResponses(
makeReq({
body: { model: 'gpt-test', input: 'hi', stream: true },
actor: makeUserActor(),
}),
res,
);
await new Promise<void>((resolve) => setImmediate(resolve));
expect(captured.written.join('')).not.toContain('event: compaction');
});
});
// ── Model details listing ───────────────────────────────────────────
describe('PuterAIController model details', () => {
@@ -52,6 +52,12 @@ export class PuterAIController extends PuterController {
const apiAuthOpts = {
subdomain: 'api',
requireUserActor: true,
// These are the user's own programmatic AI surface (OpenAI/Anthropic
// wire). `requireUserActor` here exists to keep third-party apps
// out, not to block the user's own credential — so admit full-access
// personal access tokens (CLI/MCP/scripts). Apps and scoped tokens
// stay blocked.
allowFullAccessToken: true,
} as RouteOptions;
const publicOpts = { subdomain: 'api', requireAuth: false } as const;
@@ -581,6 +587,15 @@ export class PuterAIController extends PuterController {
? { metadata: body.metadata as Record<string, string> }
: {}),
...(body.conversation ? { conversation: body.conversation } : {}),
...(body.context_management !== undefined
? { context_management: body.context_management }
: {}),
...(body.compaction !== undefined
? {
compaction:
body.compaction as ICompleteArguments['compaction'],
}
: {}),
...(body.previous_response_id
? { previous_response_id: String(body.previous_response_id) }
: {}),
@@ -744,6 +759,30 @@ export class PuterAIController extends PuterController {
output_index: outputIndex,
item,
});
} else if (ev.type === 'compaction') {
// Native shape in the final `output[]`, plus the
// canonical SSE event shared with the Anthropic surface.
const item = {
type: 'compaction',
...(ev.id !== undefined ? { id: ev.id } : {}),
encrypted_content: ev.encrypted_content,
};
const outputIndex = output.length;
output.push(item);
sendEvent({
type: 'response.output_item.added',
output_index: outputIndex,
item,
});
sendEvent({
type: 'response.output_item.done',
output_index: outputIndex,
item,
});
writeCompactionEvent(res, {
id: ev.id,
encrypted_content: ev.encrypted_content,
});
} else if (ev.type === 'usage') {
usage = buildResponsesUsage(
ev.usage as Record<string, unknown>,
@@ -863,6 +902,15 @@ export class PuterAIController extends PuterController {
...(body.max_tokens !== undefined
? { max_tokens: Number(body.max_tokens) }
: {}),
...(body.context_management !== undefined
? { context_management: body.context_management }
: {}),
...(body.compaction !== undefined
? {
compaction:
body.compaction as ICompleteArguments['compaction'],
}
: {}),
...(body.provider
? { provider: toStringOrEmpty(body.provider) }
: { provider: DEFAULTS.anthropic }),
@@ -962,6 +1010,14 @@ export class PuterAIController extends PuterController {
},
});
closeBlock();
} else if (ev.type === 'compaction') {
// Close any open content block, then emit the canonical
// compaction SSE event — byte-identical to /responses.
closeBlock();
writeCompactionEvent(res, {
id: ev.id,
encrypted_content: ev.encrypted_content,
});
} else if (ev.type === 'usage') {
usage = ev.usage as Record<string, unknown>;
}
@@ -1020,6 +1076,18 @@ export class PuterAIController extends PuterController {
if (textContent)
contentBlocks.push({ type: 'text', text: textContent });
contentBlocks.push(...toolUseBlocks);
// Native Anthropic-shaped compaction block (non-streaming bodies stay
// provider-native, unlike the unified streaming event).
const compaction = (
messageResult as { compaction?: Record<string, unknown> }
).compaction;
if (compaction) {
contentBlocks.push({
type: 'compaction',
...(compaction.id !== undefined ? { id: compaction.id } : {}),
encrypted_content: compaction.encrypted_content,
});
}
if (contentBlocks.length === 0)
contentBlocks.push({ type: 'text', text: '' });
@@ -1076,6 +1144,25 @@ const setSseHeaders = (res: Response): void => {
res.setHeader('Connection', 'keep-alive');
};
/**
* Inline-compaction is emitted in a single canonical SSE shape that is
* byte-identical across the `/responses` and `/anthropic/v1/messages` streaming
* surfaces, so a streaming client parses compaction the same way regardless of
* which upstream served the request. (Non-streaming bodies stay provider-native.)
*/
const writeCompactionEvent = (
res: Response,
compaction: { id?: unknown; encrypted_content?: unknown },
): void => {
const payload = {
type: 'compaction',
...(compaction.id !== undefined ? { id: compaction.id } : {}),
encrypted_content: compaction.encrypted_content,
};
res.write(`event: compaction\n`);
res.write(`data: ${JSON.stringify(payload)}\n\n`);
};
/**
* The chat driver returns either a stream-result envelope or a plain
* message result. Proxy routes invoked with `stream: true` expect the
@@ -1319,6 +1406,17 @@ const responseInputToMessages = (input: unknown): unknown[] => {
if (!item || typeof item !== 'object') continue;
const it = item as Record<string, unknown>;
if (it.type === 'compaction') {
// Round-tripped compaction artifact: preserve it as a bare item so
// `normalize_single_message` wraps it into an internal compaction
// content block (providers map it back to their native input shape).
messages.push({
type: 'compaction',
...(it.id !== undefined ? { id: it.id } : {}),
encrypted_content: it.encrypted_content,
});
continue;
}
if (it.type === 'function_call_output') {
messages.push({
role: 'tool',
@@ -1410,6 +1508,16 @@ const responseOutputFromResult = (
});
}
const compaction = (result as { compaction?: Record<string, unknown> })
.compaction;
if (compaction) {
output.push({
id: (compaction.id as string | undefined) || generateId('cmpct'),
type: 'compaction',
encrypted_content: compaction.encrypted_content,
});
}
return output;
};
@@ -165,7 +165,7 @@ export class StaticPagesController extends PuterController {
const domain = this.config.domain ?? req.hostname;
const origin = `${req.protocol}://${domain}`;
const apps = (await this.clients.db.read(
'SELECT `name` FROM `apps` WHERE `approved_for_listing` = 1',
`SELECT \`name\` FROM \`apps\` WHERE \`approved_for_listing\` = ${this.clients.db.booleanLiteral(true)}`,
)) as Array<{ name: string }>;
const urls = [
`<url><loc>${req.protocol}://docs.${domain}/</loc></url>`,
@@ -252,7 +252,7 @@ export class StaticPagesController extends PuterController {
const [dupe] = (await this.clients.db.read(
`SELECT EXISTS(
SELECT 1 FROM \`user\` WHERE (\`email\` = ? OR \`clean_email\` = ?)
AND \`email_confirmed\` = 1
AND \`email_confirmed\` = ${this.clients.db.booleanLiteral(true)}
AND \`password\` IS NOT NULL
) AS email_exists`,
[user.email, cleanEmail],
@@ -77,6 +77,7 @@ export class SystemController extends PuterController {
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: {
scope: 'contact-us',
limit: 10,
@@ -4,6 +4,7 @@ import type { Request, Response } from 'express';
import { Readable } from 'node:stream';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import { hash as bcryptHash } from 'bcrypt';
import { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
@@ -228,6 +229,112 @@ describe('WebDAVController', () => {
});
});
describe('pending-verification gate', () => {
// WebDAV must enforce the same gate every other authenticated route
// gets from requireVerifiedAccount — it dispatches off a single use()
// with no route options, so the middleware is never wired in and it
// has to call assertVerifiedAccount itself. Without it, an account
// still pending email/phone/card verification could read/write its
// whole filesystem over the `dav` subdomain.
const gatedActor = (flags: Record<string, unknown>) => ({
user: {
id: 1,
uuid: 'gated-uuid',
username: 'gated',
...flags,
},
});
it('rejects a session actor pending phone verification with 403', async () => {
const { res, captured } = makeRes();
await dispatchMiddleware(
makeReq({
method: 'PROPFIND',
actor: gatedActor({ requires_phone_verification: true }),
}),
res,
noop,
);
expect(captured.statusCode).toBe(403);
});
it('rejects a session actor pending card verification with 403', async () => {
const { res, captured } = makeRes();
await dispatchMiddleware(
makeReq({
method: 'GET',
actor: gatedActor({ requires_card_verification: true }),
}),
res,
noop,
);
expect(captured.statusCode).toBe(403);
});
it('rejects a session actor with an unconfirmed email with 403', async () => {
const { res, captured } = makeRes();
await dispatchMiddleware(
makeReq({
method: 'PROPFIND',
actor: gatedActor({
requires_email_confirmation: true,
email_confirmed: false,
}),
}),
res,
noop,
);
expect(captured.statusCode).toBe(403);
});
it('enforces the gate on the Basic-auth path (flags carried onto the built actor)', async () => {
const username = `webdav-gated-${Math.random()
.toString(36)
.slice(2, 10)}`;
const created = await server.stores.user.create({
username,
uuid: uuidv4(),
password: await bcryptHash('correct-horse', 4),
email: `${username}@test.local`,
free_storage: 100 * 1024 * 1024,
requires_email_confirmation: false,
});
await server.stores.user.update(created.id, {
requires_phone_verification: 1,
});
const { res, captured } = makeRes();
await dispatchMiddleware(
makeReq({
method: 'PROPFIND',
headers: {
authorization: basicAuth(username, 'correct-horse'),
},
}),
res,
noop,
);
expect(captured.statusCode).toBe(403);
});
it('lets a fully-verified session actor through the gate', async () => {
const { res, captured } = makeRes();
await dispatchMiddleware(
makeReq({
method: 'OPTIONS',
actor: gatedActor({
requires_phone_verification: false,
requires_card_verification: false,
requires_email_confirmation: false,
}),
}),
res,
noop,
);
expect(captured.statusCode).toBe(200);
});
});
describe('unsupported methods', () => {
it('returns 405 for unknown HTTP methods', async () => {
const { res, captured } = makeRes();
@@ -23,6 +23,7 @@ import { posix as pathPosix } from 'node:path';
import { EventMap } from '../../clients/event/types.js';
import type { Actor } from '../../core/actor.js';
import { HttpError } from '../../core/http/HttpError.js';
import { assertVerifiedAccount } from '../../core/http/middleware/gates.js';
import type { PuterRouter } from '../../core/http/PuterRouter.js';
import { verify as verifyOtp } from '../../services/auth/OTPUtil.js';
import { expandTildePath } from '../../services/fs/resolveNode.js';
@@ -89,6 +90,16 @@ export class WebDAVController extends PuterController {
const actor = await this.#resolveActor(req, res);
if (!actor) return; // 401 already sent
// Apply the same pending-verification gate every other authenticated
// route gets from `requireVerifiedAccount`. WebDAV dispatches every
// method off a single `router.use` with no route options, so that
// middleware is never inserted into its chain — without this call an
// account still pending email / phone / card verification could read,
// write, and delete its entire filesystem over the `dav` subdomain,
// bypassing the gate. Throws a 403 HttpError, surfaced by the catch in
// registerRoutes.
assertVerifiedAccount(actor.user);
// Expand `~`/`~/...` against the authenticated actor's username.
// WebDAV doesn't standardize `~`, but some clients do — and the
// pre-existing behaviour silently expanded it via the FS store.
@@ -233,6 +244,15 @@ export class WebDAVController extends PuterController {
email_confirmed: user.email_confirmed ?? false,
requires_email_confirmation:
user.requires_email_confirmation ?? false,
// Carry the signup-time verification flags so the
// assertVerifiedAccount() gate in #dispatch can see them on
// the Basic-auth path too (the cookie / `-token` paths get
// them from AuthService#actorUserFromRow). Omitting these is
// what let phone/card-gated accounts through WebDAV.
requires_phone_verification:
user.requires_phone_verification ?? false,
requires_card_verification:
user.requires_card_verification ?? false,
},
};
}
+9
View File
@@ -33,6 +33,15 @@ export interface ActorAccessToken {
uid: string;
issuer: Actor;
authorized?: Actor | null;
/**
* Full-API-access ("personal access token") flag, set from the signed
* `full_access` JWT claim. Such a token may exercise everything its issuing
* user can do via the API and is admitted past `requireNonAccessTokenGate`
* — but is still rejected by `requireUserActor` / web-session gates, so it
* can never manage the account. Normal (scoped) access tokens leave this
* false and remain blocked from non-`allowAccessToken` routes.
*/
fullAccess?: boolean;
}
export interface Actor {
@@ -66,6 +66,21 @@ declare global {
/** True when the request's Host is a custom domain (not one of the configured Puter domains). */
is_custom_domain?: boolean;
/**
* Coarse server-derived request fingerprint (IP + UA + accept
* headers), populated by the global fingerprint middleware. Always
* present; the same value the rate limiter keys on.
*/
networkFingerprint?: string;
/**
* Client-supplied device fingerprint (ThumbmarkJS hash) from the
* body or `x-puter-device-fingerprint` header, populated by the
* global fingerprint middleware. Present only when the client sent a
* well-shaped value; spoofable but stable per device across IPs.
*/
deviceFingerprint?: string;
/** Parsed cookies, populated by the global `cookie-parser` middleware. */
cookies?: Record<string, string>;
}
@@ -73,6 +73,19 @@ export const antiCsrf = {
export function requireAntiCsrf() {
return async (req, _res, next) => {
try {
// CSRF only applies to ambient credentials the browser attaches
// automatically (the session cookie). A full-access personal access
// token travels in the Authorization header — it can't be forged
// cross-origin — so it needs no anti-CSRF token, which is what makes
// routes like `/fs/down` reachable by a PAT. We deliberately scope
// this to full-access tokens (the only access tokens routed onto
// antiCsrf-protected endpoints): anything else — cookie-authed user
// actors, app actors, or a scoped token that somehow reaches here —
// still goes through the check below and fails closed.
if (req.actor?.accessToken?.fullAccess) {
return next();
}
const sessionId = req.actor?.user?.uuid;
if (!sessionId) {
return next(
@@ -118,6 +118,51 @@ describe('requireAntiCsrf middleware', () => {
expect(arg).toBeUndefined();
});
it('exempts a FULL-ACCESS access token (no anti_csrf token needed)', async () => {
// A full-access PAT is header-credentialed (bearer), so it can't be
// CSRF-forged — the middleware lets it through without a token. This is
// what makes antiCsrf routes like /fs/down reachable by a PAT.
const arg = await runMiddleware({
actor: {
user: { uuid: 'user-uuid-1' },
accessToken: {
uid: 'tok-1',
issuer: { user: { uuid: 'user-uuid-1' } },
fullAccess: true,
},
},
body: {},
});
expect(arg).toBeUndefined();
});
it('does NOT exempt a scoped (non-full-access) access token — fails closed', async () => {
// Scoped tokens are never deliberately routed onto antiCsrf endpoints;
// if one reaches here it must still present a token (which it can't get)
// rather than getting a free pass.
const arg = await runMiddleware({
actor: {
user: { uuid: 'user-uuid-1' },
accessToken: {
uid: 'tok-1',
issuer: { user: { uuid: 'user-uuid-1' } },
},
},
body: {},
});
expect(isHttpError(arg)).toBe(true);
expect(arg.statusCode).toBe(400);
});
it('still enforces the token for cookie-authed user actors (no accessToken)', async () => {
const arg = await runMiddleware({
actor: { user: { uuid: 'user-uuid-1' } },
body: {},
});
expect(isHttpError(arg)).toBe(true);
expect(arg.statusCode).toBe(400);
});
it('returns 401 unauthorized when no actor is attached', async () => {
const arg = await runMiddleware({ body: { anti_csrf: 'whatever' } });
expect(isHttpError(arg)).toBe(true);
@@ -0,0 +1,93 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, Response } from 'express';
import { describe, expect, it, vi } from 'vitest';
import { createFingerprintMiddleware } from './fingerprint';
const VALID_FP = 'dev-fingerprint-abc123';
const run = (reqPartial: Partial<Request> & Record<string, unknown>) => {
const middleware = createFingerprintMiddleware();
const req = {
headers: {},
...reqPartial,
} as unknown as Request;
const next = vi.fn();
middleware(req, {} as Response, next);
return { req, next };
};
describe('createFingerprintMiddleware', () => {
it('always sets a well-formed networkFingerprint and calls next', () => {
const { req, next } = run({ ip: '203.0.113.1' });
expect(req.networkFingerprint).toMatch(/^[A-Za-z0-9_-]{16}$/);
expect(next).toHaveBeenCalledOnce();
});
it('networkFingerprint is stable for identical inputs and varies by UA', () => {
const headers = { 'user-agent': 'UA/1.0' };
const a = run({ ip: '203.0.113.1', headers }).req.networkFingerprint;
const b = run({ ip: '203.0.113.1', headers }).req.networkFingerprint;
const c = run({
ip: '203.0.113.1',
headers: { 'user-agent': 'UA/2.0' },
}).req.networkFingerprint;
expect(a).toBe(b);
expect(a).not.toBe(c);
});
it('reads a well-shaped device fingerprint from the body', () => {
const { req } = run({ body: { fingerprint: VALID_FP } });
expect(req.deviceFingerprint).toBe(VALID_FP);
});
it('falls back to the x-puter-device-fingerprint header', () => {
const { req } = run({
headers: { 'x-puter-device-fingerprint': VALID_FP },
});
expect(req.deviceFingerprint).toBe(VALID_FP);
});
it('prefers the body fingerprint over the header', () => {
const { req } = run({
body: { fingerprint: VALID_FP },
headers: { 'x-puter-device-fingerprint': 'other-fingerprint-xyz' },
});
expect(req.deviceFingerprint).toBe(VALID_FP);
});
it('drops a malformed fingerprint to undefined', () => {
expect(run({ body: { fingerprint: 'bad fp!' } }).req.deviceFingerprint)
.toBeUndefined();
expect(run({ body: { fingerprint: 'short' } }).req.deviceFingerprint)
.toBeUndefined();
expect(
run({ body: { fingerprint: 123 as unknown as string } }).req
.deviceFingerprint,
).toBeUndefined();
});
it('leaves deviceFingerprint undefined when nothing was supplied', () => {
const { req } = run({ ip: '203.0.113.1' });
expect(req.deviceFingerprint).toBeUndefined();
});
});
@@ -0,0 +1,76 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { RequestHandler } from 'express';
import { computeNetworkFingerprint } from './rateLimit.js';
import '../expressAugmentation';
/**
* Conservative charset/length for the client-supplied device fingerprint. It
* becomes part of KV keys downstream (abuse extension), so anything outside
* this shape is dropped to `undefined` rather than trusted. Kept in sync with
* the abuse extension's `DEVICE_FINGERPRINT_SHAPE`; the core only enforces the
* shape, never any abuse policy built on the value.
*/
const DEVICE_FINGERPRINT_SHAPE = /^[A-Za-z0-9._-]{8,128}$/;
/** Header the GUI may send the device fingerprint on for non-signup requests. */
const DEVICE_FINGERPRINT_HEADER = 'x-puter-device-fingerprint';
function readDeviceFingerprint(req: {
body?: unknown;
headers?: Record<string, unknown>;
}): string | undefined {
// Body first (what /signup already sends), then a header fallback so
// authenticated, bodyless-or-different-shape requests can still carry it.
const body = req.body as { fingerprint?: unknown } | undefined;
const candidate =
(typeof body?.fingerprint === 'string'
? body.fingerprint
: undefined) ??
(typeof req.headers?.[DEVICE_FINGERPRINT_HEADER] === 'string'
? (req.headers[DEVICE_FINGERPRINT_HEADER] as string)
: undefined);
if (candidate && DEVICE_FINGERPRINT_SHAPE.test(candidate)) return candidate;
return undefined;
}
/**
* Stamp request-scoped fingerprints onto `req` so any downstream gate, handler,
* or service (via the ALS `Context.get('req')`) can read them without recomputing:
*
* - `req.networkFingerprint` — always set; a coarse IP+headers hash (the same
* value the rate limiter keys on). Server-derived, so it can't be forged
* away, but it's coarse (shared behind NAT/VPN, rotates with UA).
* - `req.deviceFingerprint` — set only when the client supplied a well-shaped
* device fingerprint (ThumbmarkJS hash) in the body or the
* `x-puter-device-fingerprint` header; `undefined` otherwise. Client-supplied
* and spoofable, but stable per real device across IP rotation.
*
* Install AFTER the body parsers (so the body fingerprint is readable) and
* before `requestContext` (so the snapshot into ALS already carries them).
* Never rejects — a missing/invalid device fingerprint is simply absent.
*/
export const createFingerprintMiddleware = (): RequestHandler => {
return (req, _res, next) => {
req.networkFingerprint = computeNetworkFingerprint(req);
req.deviceFingerprint = readDeviceFingerprint(req);
next();
};
};
+155 -7
View File
@@ -25,7 +25,8 @@ import {
adminOnlyGate,
allowedAppIdsGate,
requireAuthGate,
requireEmailConfirmedGate,
requireVerifiedAccount,
requireNonAccessTokenGate,
requireUserActorGate,
requireVerifiedGate,
subdomainGate,
@@ -224,6 +225,110 @@ describe('requireUserActorGate', () => {
const got = runGate(requireUserActorGate(), {});
expectHttpError(got, 401, 'token_missing');
});
it('rejects a FULL-ACCESS access token too — account routes stay closed', () => {
// The account wall is actor-type based: even a full-access PAT (which
// the resource wall lets through) is rejected here, so it can never
// reach change-password/email/2FA/token-minting/etc.
const got = runGate(requireUserActorGate(), {
actor: {
user: { uuid: 'u-1' },
accessToken: {
uid: 'tok-1',
issuer: { user: { uuid: 'u-1' } },
fullAccess: true,
},
},
});
expectHttpError(got, 403, 'forbidden');
});
// allowFullAccess opt-in: relaxes ONLY the access-token half, for
// user-resource/inference routes (e.g. the AI proxy).
it('admits a full-access PAT when allowFullAccess is set', () => {
const got = runGate(requireUserActorGate({ allowFullAccess: true }), {
actor: {
user: { uuid: 'u-1' },
accessToken: {
uid: 'tok-1',
issuer: { user: { uuid: 'u-1' } },
fullAccess: true,
},
},
});
expect(got).toBeUndefined();
});
it('still rejects a SCOPED access token even when allowFullAccess is set', () => {
const got = runGate(requireUserActorGate({ allowFullAccess: true }), {
actor: {
user: { uuid: 'u-1' },
accessToken: {
uid: 'tok-1',
issuer: { user: { uuid: 'u-1' } },
// no fullAccess
},
},
});
expectHttpError(got, 403, 'forbidden');
});
it('still rejects a third-party app even when allowFullAccess is set', () => {
const got = runGate(requireUserActorGate({ allowFullAccess: true }), {
actor: { user: { uuid: 'u-1' }, app: { uid: 'app-1' } },
});
expectHttpError(got, 403, 'forbidden');
});
});
// -- requireNonAccessTokenGate --
describe('requireNonAccessTokenGate', () => {
it('passes through for plain user actors', () => {
const got = runGate(requireNonAccessTokenGate(), {
actor: { user: { uuid: 'u-1' } },
});
expect(got).toBeUndefined();
});
it('passes through for app-under-user actors (only access tokens are gated)', () => {
const got = runGate(requireNonAccessTokenGate(), {
actor: { user: { uuid: 'u-1' }, app: { uid: 'app-1' } },
});
expect(got).toBeUndefined();
});
it('rejects a normal (scoped) access token with 403', () => {
const got = runGate(requireNonAccessTokenGate(), {
actor: {
user: { uuid: 'u-1' },
accessToken: {
uid: 'tok-1',
issuer: { user: { uuid: 'u-1' } },
},
},
});
expectHttpError(got, 403, 'forbidden');
});
it('admits a FULL-ACCESS access token (the resource-wall carve-out)', () => {
const got = runGate(requireNonAccessTokenGate(), {
actor: {
user: { uuid: 'u-1' },
accessToken: {
uid: 'tok-1',
issuer: { user: { uuid: 'u-1' } },
fullAccess: true,
},
},
});
expect(got).toBeUndefined();
});
it("falls back to 401 if there's no actor at all (defensive)", () => {
const got = runGate(requireNonAccessTokenGate(), {});
expectHttpError(got, 401, 'token_missing');
});
});
// ── adminOnlyGate ───────────────────────────────────────────────────
@@ -335,11 +440,11 @@ describe('requireVerifiedGate', () => {
});
});
// ── requireEmailConfirmedGate ───────────────────────────────────────
// ── requireVerifiedAccount ──────────────────────────────────────────
describe('requireEmailConfirmedGate', () => {
describe('requireVerifiedAccount', () => {
it('passes through users that do not require confirmation (e.g. legacy/temp)', () => {
const got = runGate(requireEmailConfirmedGate(), {
const got = runGate(requireVerifiedAccount(), {
actor: {
user: {
uuid: 'u-1',
@@ -352,7 +457,7 @@ describe('requireEmailConfirmedGate', () => {
});
it('passes through confirmed users even when confirmation is required', () => {
const got = runGate(requireEmailConfirmedGate(), {
const got = runGate(requireVerifiedAccount(), {
actor: {
user: {
uuid: 'u-1',
@@ -365,7 +470,7 @@ describe('requireEmailConfirmedGate', () => {
});
it('returns 403 email_confirmation_required for pending-confirmation users', () => {
const got = runGate(requireEmailConfirmedGate(), {
const got = runGate(requireVerifiedAccount(), {
actor: {
user: {
uuid: 'u-1',
@@ -377,8 +482,51 @@ describe('requireEmailConfirmedGate', () => {
expectHttpError(got, 403, 'email_confirmation_required');
});
it('returns 403 phone_verification_required while the phone gate is set', () => {
const got = runGate(requireVerifiedAccount(), {
actor: {
user: {
uuid: 'u-1',
requires_email_confirmation: false,
email_confirmed: true,
requires_phone_verification: true,
},
},
});
expectHttpError(got, 403, 'phone_verification_required');
});
it('returns 403 card_verification_required while the card gate is set', () => {
const got = runGate(requireVerifiedAccount(), {
actor: {
user: {
uuid: 'u-1',
requires_email_confirmation: false,
email_confirmed: true,
requires_card_verification: true,
},
},
});
expectHttpError(got, 403, 'card_verification_required');
});
it('passes through once every gate is cleared', () => {
const got = runGate(requireVerifiedAccount(), {
actor: {
user: {
uuid: 'u-1',
requires_email_confirmation: true,
email_confirmed: true,
requires_phone_verification: false,
requires_card_verification: false,
},
},
});
expect(got).toBeUndefined();
});
it("passes through when there's no actor (auth gate handled it)", () => {
const got = runGate(requireEmailConfirmedGate(), {});
const got = runGate(requireVerifiedAccount(), {});
expect(got).toBeUndefined();
});
});
+95 -17
View File
@@ -99,8 +99,18 @@ export const requireAuthGate = (): RequestHandler => {
* Reject app-under-user and access-token actors with 403. Use on endpoints
* that should only be exercised by a human session — settings changes,
* admin-style actions on the user's own account.
*
* `allowFullAccess` (set per-route via the `allowFullAccessToken` route option)
* relaxes ONLY the access-token half: a full-access ("personal access token")
* actor is admitted, because it represents the user's own full API reach.
* Third-party apps are ALWAYS rejected, and scoped access tokens are always
* rejected. This opt-in is for user-resource / inference endpoints (AI proxy,
* etc.) that use this gate purely to keep apps out — NEVER for account or
* security management, which must stay closed to every access token.
*/
export const requireUserActorGate = (): RequestHandler => {
export const requireUserActorGate = (
opts: { allowFullAccess?: boolean } = {},
): RequestHandler => {
return (req, _res, next) => {
const actor = req.actor;
// requireAuth runs first; this gate just narrows the actor type.
@@ -108,7 +118,14 @@ export const requireUserActorGate = (): RequestHandler => {
next(rejectAuth(req));
return;
}
if (actor.app || actor.accessToken) {
// Third-party apps are never allowed through this gate.
const appBlocked = !!actor.app;
// Access tokens are blocked unless the route opted in AND this is a
// full-access PAT (the user's own credential). Scoped tokens: blocked.
const tokenBlocked =
!!actor.accessToken &&
!(opts.allowFullAccess && actor.accessToken.fullAccess);
if (appBlocked || tokenBlocked) {
next(
new HttpError(
403,
@@ -129,7 +146,13 @@ export const requireNonAccessTokenGate = (): RequestHandler => {
next(rejectAuth(req));
return;
}
if (actor.accessToken) {
// Full-access ("personal access token") access tokens are admitted here:
// they carry the user's full API reach by design. They remain blocked
// from account management because those routes also use
// `requireUserActorGate`, which rejects ALL access tokens. Normal
// (scoped) access tokens stay blocked from non-`allowAccessToken`
// routes.
if (actor.accessToken && !actor.accessToken.fullAccess) {
next(
new HttpError(
403,
@@ -201,29 +224,84 @@ export const requireVerifiedGate = (strictFlag: boolean): RequestHandler => {
};
/**
* Reject authenticated users whose account is pending email confirmation
* (`requires_email_confirmation && !email_confirmed`). Runs on every
* authenticated route by default; routes that set `allowUnconfirmed: true`
* skip this gate.
* Reject authenticated users whose account is still pending any signup-time
* verification — email confirmation, SMS phone verification, or credit-card
* verification. The abuse harness sets the phone/card flags on low-reputation
* signups (in place of a hard block), and this gate is what actually keeps
* those accounts out of the product until the flag clears: the flags live on
* `req.actor.user`, so every authenticated route enforces them, not just the
* GUI modal.
*
* Returns 403 with `email_confirmation_required` so clients can show the
* confirmation prompt instead of a generic error.
* Runs on every authenticated route by default; routes that set
* `allowUnconfirmed: true` opt out (the verification endpoints themselves,
* plus essential flows like whoami / logout / save-account so a pending
* account can still reach the screens that clear the gate).
*
* Returns 403 with a per-gate legacy code (`email_confirmation_required` /
* `phone_verification_required` / `card_verification_required`) so clients can
* show the right prompt instead of a generic error. There is no state where a
* user should be allowed in with one verification pending, so any pending gate
* rejects.
*/
export const requireEmailConfirmedGate = (): RequestHandler => {
export const requireVerifiedAccount = (): RequestHandler => {
return (req, _res, next) => {
const user = req.actor?.user;
if (user?.requires_email_confirmation && !user?.email_confirmed) {
next(
new HttpError(403, 'Please confirm your email to continue', {
legacyCode: 'email_confirmation_required',
}),
);
try {
assertVerifiedAccount(req.actor?.user);
} catch (err) {
next(err);
return;
}
next();
};
};
/**
* The pending-verification check, factored out of {@link requireVerifiedAccount}
* so auth paths that build their own actor outside the route-option machinery
* can enforce the exact same gate. The WebDAV controller is the motivating
* case: it dispatches every method off a single `router.use`, so
* `requireVerifiedAccount` is never wired into its chain — it has to call this
* directly. Keeping one implementation is the point: a verification gate added
* here is picked up by every caller, so the paths can't drift (which is how
* WebDAV came to bypass the phone/card gate to begin with).
*
* Throws 403 with a per-gate legacy code (`email_confirmation_required` /
* `phone_verification_required` / `card_verification_required`) so clients can
* show the right prompt instead of a generic error. There is no state where a
* user should be let in with any verification pending, so the first pending
* gate rejects.
*/
export const assertVerifiedAccount = (
user:
| {
requires_email_confirmation?: unknown;
email_confirmed?: unknown;
requires_phone_verification?: unknown;
requires_card_verification?: unknown;
}
| undefined,
): void => {
if (user?.requires_email_confirmation && !user?.email_confirmed) {
throw new HttpError(403, 'Please confirm your email to continue', {
legacyCode: 'email_confirmation_required',
});
}
if (user?.requires_phone_verification) {
throw new HttpError(
403,
'Please verify your phone number to continue',
{
legacyCode: 'phone_verification_required' as never,
},
);
}
if (user?.requires_card_verification) {
throw new HttpError(403, 'Please verify your card to continue', {
legacyCode: 'card_verification_required' as never,
});
}
};
/**
* Reject unless the actor is acting through one of the named apps.
* App-under-user actors are permitted iff `actor.app.uid` is in the allowList;
@@ -264,6 +264,79 @@ describe('createUserSubdomainRedirect', () => {
]);
});
const selfHosted = {
domain: 'puter.localhost',
static_hosting_domain: 'site.puter.localhost',
static_hosting_domain_alt: 'host.puter.localhost',
private_app_hosting_domain: 'app.puter.localhost',
private_app_hosting_domain_alt: 'dev.puter.localhost',
} as IConfig;
it('still redirects a bare subdomain on the main domain to the hosting domain (self-hosted)', () => {
const { out, next } = run(
createUserSubdomainRedirect(selfHosted),
makeReq({
subdomains: ['localhost', 'puter', 'foo'],
host: 'foo.puter.localhost',
originalUrl: '/bar?x=1',
protocol: 'http',
}),
);
expect(out.redirectArgs).toEqual([
302,
'http://foo.site.puter.localhost/bar?x=1',
]);
expect(next).not.toHaveBeenCalled();
});
it('passes through hosts already on the static hosting domain (no redirect loop)', () => {
const { out, next } = run(
createUserSubdomainRedirect(selfHosted),
makeReq({
subdomains: ['localhost', 'puter', 'site', 'foo'],
host: 'foo.site.puter.localhost',
originalUrl: '/',
}),
);
expect(out.redirectArgs).toBeUndefined();
expect(next).toHaveBeenCalledTimes(1);
});
it('passes through hosts on the alt / private-app hosting domains too', () => {
for (const host of [
'foo.host.puter.localhost',
'foo.app.puter.localhost',
'foo.dev.puter.localhost',
]) {
const { out, next } = run(
createUserSubdomainRedirect(selfHosted),
makeReq({
subdomains: [
'localhost',
'puter',
host.split('.')[1],
'foo',
],
host,
}),
);
expect(out.redirectArgs).toBeUndefined();
expect(next).toHaveBeenCalledTimes(1);
}
});
it('passes through the hosting-domain root itself (exact match, no loop)', () => {
const { out, next } = run(
createUserSubdomainRedirect(selfHosted),
makeReq({
subdomains: ['localhost', 'puter', 'site'],
host: 'site.puter.localhost',
}),
);
expect(out.redirectArgs).toBeUndefined();
expect(next).toHaveBeenCalledTimes(1);
});
it("passes through when the request host has a port the configured domain doesn't", () => {
// Edge case worth pinning: the suffix check is exact-`endsWith`,
// so a port mismatch silently bypasses the redirect. Documenting
@@ -88,6 +88,15 @@ export const createUserSubdomainRedirect = (
if (!domain || !target) {
return (_req, _res, next) => next();
}
const hostingDomains = [
config.static_hosting_domain,
config.static_hosting_domain_alt,
config.private_app_hosting_domain,
config.private_app_hosting_domain_alt,
]
.map((d) => (d ?? '').toLowerCase().split(':')[0])
.filter((d) => d.length > 0);
return (req, res, next) => {
const active = (
req.subdomains?.[req.subdomains.length - 1] ?? ''
@@ -95,6 +104,14 @@ export const createUserSubdomainRedirect = (
if (active === '' || RESERVED_SUBDOMAINS.has(active)) return next();
const host = (req.headers.host ?? '').toLowerCase();
const hostName = host.split(':')[0];
if (
hostingDomains.some(
(d) => hostName === d || hostName.endsWith(`.${d}`),
)
) {
return next();
}
if (!host.endsWith(domain)) return next();
// host ends in domain — swap the domain suffix for the hosting one,
@@ -243,7 +243,9 @@ export async function resolveOwnedAppForHostedSite(opts: {
const uniqueCandidates = [...new Set(urlCandidates)];
if (uniqueCandidates.length === 0) return null;
const privateFilter = opts.requirePrivate ? 'AND `is_private` = 1 ' : '';
const privateFilter = opts.requirePrivate
? `AND \`is_private\` = ${opts.db.booleanLiteral(true)} `
: '';
const placeholders = uniqueCandidates.map(() => '?').join(', ');
const rows = await opts.db.read(
`SELECT * FROM apps WHERE owner_user_id = ? ${privateFilter}AND index_url IN (${placeholders}) LIMIT 2`,
@@ -534,6 +534,51 @@ describe('createPuterSiteMiddleware — file serving', () => {
expect(piped!.equals(body)).toBe(true);
});
it('emits site.htmlServed with the original URL target including query string', async () => {
const owner = await makeUserWithHome();
const homePath = `/${owner.username}`;
const homeEntry = await server.stores.fsEntry.getEntryByPath(homePath);
const sub = `serveq-${Math.random().toString(36).slice(2, 8)}`;
await server.stores.subdomain.create({
userId: owner.id,
subdomain: sub,
rootDirId: homeEntry!.id,
});
await writeFile(
owner.id,
`${homePath}/index.html`,
Buffer.from('<html>hi</html>'),
'text/html',
);
const emitSpy = vi.spyOn(server.clients.event, 'emit');
try {
const mw = buildMiddleware();
const { res } = makeRes();
await mw(
makeReq({
hostname: `${sub}.site.puter.localhost`,
path: '/index.html',
originalUrl: '/index.html?brand=paypal',
}),
res,
vi.fn(),
);
const htmlServedCall = emitSpy.mock.calls.find(
(call) => call[0] === 'site.htmlServed',
);
expect(htmlServedCall?.[1]).toMatchObject({
subdomain: sub,
requestPath: '/index.html',
requestUrl: '/index.html?brand=paypal',
mime: 'text/html',
});
} finally {
emitSpy.mockRestore();
}
});
it('returns 206 status when a Range header is supplied', async () => {
const owner = await makeUserWithHome();
const homePath = `/${owner.username}`;
@@ -555,8 +555,12 @@ export const createPuterSiteMiddleware = (
res.setHeader('Content-Type', mime);
// Fire-and-forget signal for downstream extensions
if (mime === 'text/html' || mime === 'application/xhtml+xml') {
const mimeBase = mime.split(';', 1)[0].trim().toLowerCase();
if (mimeBase === 'text/html' || mimeBase === 'application/xhtml+xml') {
try {
const requestUrl = (req.originalUrl || '/').startsWith('/')
? req.originalUrl || '/'
: `/${req.originalUrl}`;
layers.clients.event.emit(
'site.htmlServed',
{
@@ -564,7 +568,8 @@ export const createPuterSiteMiddleware = (
entry,
host,
requestPath: req.path,
mime,
requestUrl,
mime: mimeBase,
},
{},
);
+12 -1
View File
@@ -392,7 +392,14 @@ function ip(req) {
return req.ip || req.socket?.remoteAddress || 'unknown';
}
function fingerprint(req) {
/**
* A coarse network fingerprint for a request: a short hash of the (proxy-aware)
* IP plus the headers a client can't trivially vary per-request without also
* changing how the request looks. Used as the default rate-limit key here, and
* exported so the global fingerprint middleware can stamp the identical value
* on `req.networkFingerprint` (one key space shared by both).
*/
export function computeNetworkFingerprint(req) {
const parts = [
ip(req),
req.headers?.['user-agent'] || '',
@@ -406,6 +413,10 @@ function fingerprint(req) {
.slice(0, 16);
}
function fingerprint(req) {
return computeNetworkFingerprint(req);
}
// -- Route middleware ------------------------------------------------
/**
@@ -140,6 +140,67 @@ describe('rateLimitGate — memory backend (default)', () => {
expect(isHttpError(re)).toBe(true);
});
// Stacked gates, mirroring the materializer's handling of a
// `rateLimit: [...]` array (each entry is its own gate; a rejection
// short-circuits the chain). This is the credential-endpoint pattern:
// per-fingerprint budget for fairness on shared IPs, per-IP backstop
// against header rotation.
const runStack = async (stack, req) => {
for (const opts of stack) {
const err = await runGate(opts, req);
if (err) return err;
}
return undefined;
};
it('stacked gates: per-IP backstop catches header rotation that escapes the fingerprint key', async () => {
const stack = [
{ limit: 3, window: 60_000, scope: 'stack-fp' },
{ limit: 6, window: 60_000, key: 'ip', scope: 'stack-ip' },
];
// Rotating the User-Agent mints a fresh fingerprint bucket every
// request, so the fingerprint gate admits all of these — the IP
// backstop must cap the total anyway.
for (let i = 0; i < 6; i++) {
const arg = await runStack(
stack,
makeReq({ headers: { 'user-agent': `rotated-ua-${i}` } }),
);
expect(arg).toBeUndefined();
}
const rejected = await runStack(
stack,
makeReq({ headers: { 'user-agent': 'rotated-ua-final' } }),
);
expect(isHttpError(rejected)).toBe(true);
expect(rejected.statusCode).toBe(429);
});
it('stacked gates: clients sharing an IP each get their own fingerprint budget under the IP cap', async () => {
const stack = [
{ limit: 2, window: 60_000, scope: 'stack2-fp' },
{ limit: 10, window: 60_000, key: 'ip', scope: 'stack2-ip' },
];
// Two clients (distinct UAs) behind one NAT IP: each gets the
// full per-fingerprint budget, and each is cut off by its own
// fingerprint bucket — not starved by the other's traffic.
for (const ua of ['alice-browser', 'bob-browser']) {
for (let i = 0; i < 2; i++) {
expect(
await runStack(
stack,
makeReq({ headers: { 'user-agent': ua } }),
),
).toBeUndefined();
}
const rejected = await runStack(
stack,
makeReq({ headers: { 'user-agent': ua } }),
);
expect(isHttpError(rejected)).toBe(true);
}
});
it("falls back to fingerprint when 'user' strategy is selected but no actor present", async () => {
// The fallback prevents anonymous traffic from sharing a single
// bucket (which would invite trivial DoS via global rate-limit).
@@ -0,0 +1,251 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { EventEmitter } from 'node:events';
import type { Request, Response } from 'express';
import { describe, expect, it } from 'vitest';
import { EventClient } from '../../clients/event/EventClient';
import type {
EventMap,
RouteLifecycleEvent,
} from '../../clients/event/types';
import type { IConfig } from '../../types';
import {
createRouteLifecycleMiddleware,
pathKeySegment,
routeEventKeyBase,
} from './routeLifecycle';
// -- Fakes --
//
// EventClient is a pure in-memory bus (no external deps), so we use the real
// one and register real listeners. The request/response are the boundary we
// fake: a minimal `res` that is an EventEmitter (for `finish`/`close`) and
// records status/body.
const makeEvents = () => new EventClient({} as IConfig);
class FakeRes extends EventEmitter {
statusCode = 200;
writableFinished = false;
headersSent = false;
body: unknown;
status(code: number) {
this.statusCode = code;
return this;
}
json(b: unknown) {
this.body = b;
return this;
}
}
const makeReq = (actorUuid?: string): Request =>
({
actor: actorUuid ? { user: { uuid: actorUuid } } : undefined,
}) as unknown as Request;
// Collect every emit on a given key into an array for assertions.
const record = (events: EventClient, key: keyof EventMap) => {
const seen: RouteLifecycleEvent[] = [];
events.on(key as never, (_k, data) =>
seen.push(data as RouteLifecycleEvent),
);
return seen;
};
const POST = 'post' as const;
const PATH = '/fs/completeBatchWrite';
const BASE = 'route.post.fs.completeBatchWrite';
describe('pathKeySegment', () => {
it('joins path parts with dots and drops slashes', () => {
expect(pathKeySegment('/fs/completeBatchWrite')).toBe(
'fs.completeBatchWrite',
);
});
it('strips the leading colon from path params', () => {
expect(pathKeySegment('/foo/:id')).toBe('foo.id');
});
it('maps the root path and non-string paths to placeholders', () => {
expect(pathKeySegment('/')).toBe('root');
expect(pathKeySegment(/^\/x/)).toBe('_');
});
});
describe('routeEventKeyBase', () => {
it('scopes the key to method + normalized path', () => {
expect(routeEventKeyBase(POST, PATH)).toBe(BASE);
});
});
describe('createRouteLifecycleMiddleware', () => {
it('emits before, then after on a clean finish', async () => {
const events = makeEvents();
const before = record(events, `${BASE}.before` as keyof EventMap);
const after = record(events, `${BASE}.after` as keyof EventMap);
const error = record(events, `${BASE}.error` as keyof EventMap);
const mw = createRouteLifecycleMiddleware(events, POST, PATH);
const req = makeReq('u-1');
const res = new FakeRes();
let nextCalled = false;
await mw(req, res as unknown as Response, () => {
nextCalled = true;
});
expect(nextCalled).toBe(true);
expect(before).toHaveLength(1);
expect(before[0]).toMatchObject({
phase: 'before',
method: 'post',
path: PATH,
actor: { user: { uuid: 'u-1' } },
actorUid: 'user:u-1',
});
// The live req/res are exposed so listeners can read the body or
// respond themselves.
expect(before[0].req).toBe(req);
expect(before[0].res).toBe(res);
res.statusCode = 200;
res.writableFinished = true;
res.emit('finish');
expect(after).toHaveLength(1);
expect(after[0]).toMatchObject({ phase: 'after', statusCode: 200 });
expect(typeof after[0].durationMs).toBe('number');
expect(error).toHaveLength(0);
});
it('emits error (not after) when the response is a 5xx', async () => {
const events = makeEvents();
const after = record(events, `${BASE}.after` as keyof EventMap);
const error = record(events, `${BASE}.error` as keyof EventMap);
const mw = createRouteLifecycleMiddleware(events, POST, PATH);
const res = new FakeRes();
await mw(makeReq(), res as unknown as Response, () => {});
res.statusCode = 500;
res.writableFinished = true;
res.emit('finish');
expect(after).toHaveLength(0);
expect(error).toHaveLength(1);
expect(error[0]).toMatchObject({ phase: 'error', statusCode: 500 });
});
it('emits error when the connection closes without finishing (abort)', async () => {
const events = makeEvents();
const error = record(events, `${BASE}.error` as keyof EventMap);
const mw = createRouteLifecycleMiddleware(events, POST, PATH);
const res = new FakeRes();
await mw(makeReq(), res as unknown as Response, () => {});
res.writableFinished = false;
res.emit('close');
expect(error).toHaveLength(1);
expect(error[0].phase).toBe('error');
expect(error[0].error).toBeInstanceOf(Error);
});
it('emits a terminal event only once across finish + close', async () => {
const events = makeEvents();
const after = record(events, `${BASE}.after` as keyof EventMap);
const error = record(events, `${BASE}.error` as keyof EventMap);
const mw = createRouteLifecycleMiddleware(events, POST, PATH);
const res = new FakeRes();
await mw(makeReq(), res as unknown as Response, () => {});
res.statusCode = 200;
res.writableFinished = true;
res.emit('finish');
res.emit('close');
expect(after).toHaveLength(1);
expect(error).toHaveLength(0);
});
it('vetoes the request when a before listener sets allow=false', async () => {
const events = makeEvents();
events.on(`${BASE}.before` as never, (_k, data) => {
(data as RouteLifecycleEvent).allow = false;
(data as RouteLifecycleEvent).rejectReason = 'quota exceeded';
});
const reject = record(events, `${BASE}.reject` as keyof EventMap);
const after = record(events, `${BASE}.after` as keyof EventMap);
const mw = createRouteLifecycleMiddleware(events, POST, PATH);
const res = new FakeRes();
let nextCalled = false;
await mw(makeReq('u-9'), res as unknown as Response, () => {
nextCalled = true;
});
expect(nextCalled).toBe(false);
expect(res.statusCode).toBe(403);
expect(res.body).toMatchObject({
error: { code: 'forbidden', message: 'quota exceeded' },
});
expect(reject).toHaveLength(1);
expect(reject[0]).toMatchObject({
phase: 'reject',
statusCode: 403,
rejectReason: 'quota exceeded',
});
// A vetoed request never runs, so no terminal after fires.
res.emit('finish');
expect(after).toHaveLength(0);
});
it('treats a listener-sent response as a terminal after (not a reject) with the real status', async () => {
const events = makeEvents();
events.on(`${BASE}.before` as never, (_k, data) => {
// Listener answers the request itself without vetoing.
const e = data as RouteLifecycleEvent;
(e.res as unknown as FakeRes).headersSent = true;
e.res.status(418).json({ teapot: true });
});
const reject = record(events, `${BASE}.reject` as keyof EventMap);
const after = record(events, `${BASE}.after` as keyof EventMap);
const mw = createRouteLifecycleMiddleware(events, POST, PATH);
const res = new FakeRes();
let nextCalled = false;
await mw(makeReq(), res as unknown as Response, () => {
nextCalled = true;
});
expect(nextCalled).toBe(false);
// Listener's own 418 stands; the middleware doesn't clobber it with 403.
expect(res.statusCode).toBe(418);
expect(res.body).toEqual({ teapot: true });
// Not a veto, so no reject — a terminal `after` keyed off the real 418.
expect(reject).toHaveLength(0);
expect(after).toHaveLength(1);
expect(after[0]).toMatchObject({ phase: 'after', statusCode: 418 });
});
});
+162
View File
@@ -0,0 +1,162 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, RequestHandler } from 'express';
import { actorUid } from '../actor';
import type { EventClient } from '../../clients/event/EventClient';
import type { RouteMethod } from './types';
type RoutePathValue = string | RegExp | Array<string | RegExp> | undefined;
/**
* Normalize a route path into a dot-separated key segment so lifecycle events
* can be scoped per endpoint. Strips slashes and param markers so the key is
* stable (`/drivers/call` -> `drivers.call`, `/foo/:id` -> `foo.id`).
* Non-string paths (RegExp / arrays) collapse to a single `_` placeholder.
*/
export const pathKeySegment = (fullPath: RoutePathValue): string => {
if (typeof fullPath !== 'string') return '_';
const seg = fullPath
.split('/')
.filter(Boolean)
.map((p) => p.replace(/^:/, '').replace(/[^A-Za-z0-9_-]/g, '_'))
.join('.');
return seg || 'root';
};
/**
* Scoped event-key base for a route: `route.<method>.<normalized-path>`.
* Callers append `.before` / `.after` / `.error` / `.reject`.
*/
export const routeEventKeyBase = (
method: RouteMethod,
fullPath: RoutePathValue,
): `route.${string}` => `route.${method}.${pathKeySegment(fullPath)}`;
/**
* Build the per-endpoint lifecycle middleware for one route.
*
* On each request it: emits `before` (awaited, so a listener may veto by
* setting `allow = false`), times the handler, and emits a single terminal
* event when the response finishes or the connection closes — `after` on a
* clean response, `error` on an abort or a >=500, `reject` only when a
* listener vetoes (it then answers 403 and the handler never runs). A listener
* that writes its own response without vetoing terminates as `after`/`error`
* keyed off the real status, not `reject`.
*
* Subscribers can listen on `route.*`, `route.<method>.*`, or the exact key.
*/
export const createRouteLifecycleMiddleware = (
events: EventClient,
method: RouteMethod,
fullPath: RoutePathValue,
): RequestHandler => {
const pathLabel =
typeof fullPath === 'string' ? fullPath : String(fullPath);
const keyBase = routeEventKeyBase(method, fullPath);
return async (req: Request, res, next) => {
const actor = req.actor ? actorUid(req.actor) : undefined;
const startedAt = Date.now();
const base = {
method,
path: pathLabel,
req,
res,
actor: req.actor,
actorUid: actor,
};
const beforeEvent = {
phase: 'before' as const,
...base,
allow: true as boolean,
rejectReason: undefined as string | undefined,
};
await events.emitAndWait(`${keyBase}.before`, beforeEvent, {});
// Explicit veto: a listener set `allow = false`. Emit `reject` and
// answer 403 — unless the listener already wrote its own response, in
// which case we report that status rather than a misleading 403.
if (beforeEvent.allow === false) {
events.emit(
`${keyBase}.reject`,
{
phase: 'reject',
...base,
statusCode: res.headersSent ? res.statusCode : 403,
durationMs: Date.now() - startedAt,
rejectReason: beforeEvent.rejectReason,
},
{},
);
if (!res.headersSent) {
res.status(403).json({
error: {
code: 'forbidden',
message:
beforeEvent.rejectReason ?? 'Blocked by policy',
},
});
}
return;
}
// A listener answered the request itself without vetoing (it wrote a
// response in the `before` hook). That's a normal terminal outcome,
// not a reject — emit `after`/`error` keyed off the real status.
if (res.headersSent) {
const statusCode = res.statusCode;
const isError = statusCode >= 500;
events.emit(
`${keyBase}.${isError ? 'error' : 'after'}`,
{
phase: isError ? 'error' : 'after',
...base,
statusCode,
durationMs: Date.now() - startedAt,
},
{},
);
return;
}
let settled = false;
const settle = (aborted: boolean) => {
if (settled) return;
settled = true;
const statusCode = res.statusCode;
const isError = aborted || statusCode >= 500;
events.emit(
`${keyBase}.${isError ? 'error' : 'after'}`,
{
phase: isError ? 'error' : 'after',
...base,
statusCode,
durationMs: Date.now() - startedAt,
...(aborted ? { error: new Error('request aborted') } : {}),
},
{},
);
};
res.once('finish', () => settle(false));
res.once('close', () => settle(!res.writableFinished));
next();
};
};
+35 -14
View File
@@ -66,6 +66,22 @@ export type RoutePath = string | RegExp | Array<string | RegExp>;
* in the chain. Commented-out slots are reserved for the next chunks
* (body parsing, post-auth gates, timing).
*/
/** One rate-limit window. See `RouteOptions.rateLimit` for semantics. */
export interface RouteRateLimit {
limit: number;
window: number;
/**
* Per-subscription overrides for `limit` (`SubscriptionPolicy.id`
* → cap). Same mechanic as `concurrent.bySubscription`: resolved
* via the metering service per request; falls back to the base
* `limit` when there's no actor / no metering / no match.
*/
bySubscription?: Record<string, number>;
key?: 'fingerprint' | 'ip' | 'user' | ((req: Request) => string);
scope?: string;
backend?: 'memory' | 'redis' | 'kv';
}
export interface RouteOptions {
/** Extra per-route middleware. Applied after built-in gates, before the handler. */
middleware?: RequestHandler[];
@@ -86,6 +102,17 @@ export interface RouteOptions {
/** Reject app/access-token actors. Implies `requireAuth`. */
requireUserActor?: boolean;
/**
* Only meaningful alongside `requireUserActor`. Relaxes the access-token
* half of that gate so a FULL-ACCESS personal access token (the user's own
* credential) is admitted — third-party apps and scoped tokens stay
* blocked. Use ONLY on user-resource / inference endpoints that gate with
* `requireUserActor` purely to keep apps out (e.g. the AI proxy). NEVER set
* on account/security/credential routes — those must stay closed to every
* access token. Default-deny: omitting this keeps PATs blocked.
*/
allowFullAccessToken?: boolean;
/** Allows access-tokens */
allowAccessToken?: boolean;
@@ -198,21 +225,15 @@ export interface RouteOptions {
* All registered backends stay co-resident at runtime, so different
* routes can pick whichever fits their access pattern. Omitting
* `backend` uses the server-wide default (`config.rate_limit.backend`).
*
* An array applies every limit independently (a request must pass all
* of them). Use this to pair a per-client budget with a coarser
* backstop on a different key — e.g. per-fingerprint for fairness on
* shared IPs, plus per-IP so rotating headers can't mint fresh
* fingerprint buckets indefinitely. Give each entry its own `scope`
* so the counters don't collide.
*/
rateLimit?: {
limit: number;
window: number;
/**
* Per-subscription overrides for `limit` (`SubscriptionPolicy.id`
* → cap). Same mechanic as `concurrent.bySubscription`: resolved
* via the metering service per request; falls back to the base
* `limit` when there's no actor / no metering / no match.
*/
bySubscription?: Record<string, number>;
key?: 'fingerprint' | 'ip' | 'user' | ((req: Request) => string);
scope?: string;
backend?: 'memory' | 'redis' | 'kv';
};
rateLimit?: RouteRateLimit | RouteRateLimit[];
/**
* Concurrent in-flight limiting. Caps how many requests for this
@@ -547,6 +547,46 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
expect(passed.max_tokens!).toBeGreaterThan(0);
});
it('throws 402 instead of leaving `max_tokens` unset when credits cannot afford one output token', async () => {
// Regression: previously a sub-1 cap set max_tokens to `undefined`,
// which let the provider run to the model's full output limit and
// overdraw the account. It must reject instead.
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValueOnce([
{
id: 'capme',
aliases: [],
costs_currency: 'usd-cents',
costs: { input_tokens: 1000, output_tokens: 2000 },
max_tokens: 8192,
},
]);
const d = await makeDriver();
// Pass the cheap pre-flight gate but leave a balance too small to
// afford a single 2000-microcent output token.
vi.spyOn(server.services.metering, 'hasEnoughCredits').mockResolvedValue(
true,
);
vi.spyOn(server.services.metering, 'getRemainingUsage').mockResolvedValue(
100,
);
const completeSpy = vi.spyOn(FakeChatProvider.prototype, 'complete');
await expect(
withTestActor(() =>
d.complete({
model: 'capme',
messages: [{ role: 'user', content: 'hi' }],
}),
),
).rejects.toMatchObject({
statusCode: 402,
legacyCode: 'insufficient_funds',
});
expect(completeSpy).not.toHaveBeenCalled();
});
it('rejects subscriber-only models for the default free subscription', async () => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValueOnce([
{
@@ -17,8 +17,9 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { PassThrough } from 'node:stream';
import crypto from 'node:crypto';
import { PassThrough } from 'node:stream';
import { EventMap } from '../../clients/event/types.js';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import {
@@ -28,12 +29,17 @@ import {
import type { DriverStreamResult } from '../meta.js';
import { PuterDriver } from '../types.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../util/aiLimits.js';
import { AlibabaProvider } from './providers/alibaba/AlibabaProvider.js';
import { AzureChatProvider } from './providers/azure/AzureChatProvider.js';
import { AzureResponsesProvider } from './providers/azure/AzureResponsesProvider.js';
import { ClaudeProvider } from './providers/claude/ClaudeProvider.js';
import { DeepSeekProvider } from './providers/deepseek/DeepSeekProvider.js';
import { FakeChatProvider } from './providers/FakeChatProvider.js';
import { GeminiChatProvider } from './providers/gemini/GeminiChatProvider.js';
import { GroqAIProvider } from './providers/groq/GroqAIProvider.js';
import { MiniMaxProvider } from './providers/minimax/MiniMaxProvider.js';
import { MistralAIProvider } from './providers/mistral/MistralAiProvider.js';
import { MoonshotProvider } from './providers/moonshot/MoonshotProvider.js';
import { OllamaChatProvider } from './providers/ollama/OllamaProvider.js';
import { OpenAiChatProvider } from './providers/openai/OpenAiChatCompletionsProvider.js';
import { OpenAiResponsesChatProvider } from './providers/openai/OpenAiChatResponsesProvider.js';
@@ -41,8 +47,6 @@ import { OpenRouterProvider } from './providers/openrouter/OpenRouterProvider.js
import { TogetherAIProvider } from './providers/together/TogetherAIProvider.js';
import { XAIProvider } from './providers/xai/XAIProvider.js';
import { ZAIProvider } from './providers/zai/ZAIProvider.js';
import { AlibabaProvider } from './providers/alibaba/AlibabaProvider.js';
import { MoonshotProvider } from './providers/moonshot/MoonshotProvider.js';
import type {
IChatCompleteResult,
IChatModel,
@@ -56,7 +60,6 @@ import {
normalize_single_message,
} from './utils/Messages.js';
import { AIChatStream } from './utils/Streaming.js';
import { EventMap } from '../../clients/event/types.js';
const MAX_FALLBACKS = 4; // includes first attempt
@@ -276,7 +279,7 @@ export class ChatCompletionDriver extends PuterDriver {
let intendedProvider = args.provider || '';
if (!args.model && !intendedProvider) {
intendedProvider = 'claude'; // default provider
intendedProvider = 'azure-openai'; // default provider
}
if (
!args.model &&
@@ -397,16 +400,25 @@ export class ChatCompletionDriver extends PuterDriver {
remainingCredits - approximateInputCost;
const maxAllowedOutputTokens =
maxAllowedOutputUcents / outputTokenCost;
if (maxAllowedOutputTokens) {
const cap = Math.floor(
Math.min(
args.max_tokens ?? Number.POSITIVE_INFINITY,
maxAllowedOutputTokens,
model.max_tokens - approximateTokenCount,
),
);
args.max_tokens = cap < 1 ? undefined : cap;
const cap = Math.floor(
Math.min(
args.max_tokens ?? Number.POSITIVE_INFINITY,
maxAllowedOutputTokens,
model.max_tokens - approximateTokenCount,
),
);
// `cap` is the credit-bounded ceiling on output tokens. When
// it drops below 1 the user can't afford even a single output
// token, so reject the request. Crucially we must NOT leave
// `max_tokens` unset here: an undefined max_tokens lets the
// provider run to the model's full output limit (e.g. 128k for
// Claude), billing far past the user's remaining balance.
if (cap < 1) {
throw new HttpError(402, 'No usage left for request.', {
legacyCode: 'insufficient_funds',
});
}
args.max_tokens = cap;
}
}
@@ -795,6 +807,43 @@ export class ChatCompletionDriver extends PuterDriver {
);
}
// Azure AI Foundry (OpenAI + xAI Grok). Registered before the regular
// OpenAI/xAI providers so that since its costs mirror theirs but
// Azure is preferred for us, it takes precedence in the per-model
// bucket
const azureOpenai = providers['azure-openai'];
const azureOpenaiKey = readKey(azureOpenai);
const azureOpenaiURL = azureOpenai?.apiURL as string | undefined;
if (azureOpenaiKey && azureOpenaiURL) {
const azureStores = {
fsEntry: this.stores.fsEntry,
s3Object: this.stores.s3Object,
};
const azureConfig = {
apiKey: azureOpenaiKey,
apiURL: azureOpenaiURL,
};
const azureCompletions = new AzureChatProvider(
metering,
azureStores,
this.services.fs,
azureConfig,
);
// Codex / Responses-API-only models can't use Chat Completions, so
// they route through a sibling Responses provider pointed at the
// same Azure endpoint. web_search (also Responses-only) delegates
// here too.
const azureResponses = new AzureResponsesProvider(
metering,
azureStores,
this.services.fs,
azureConfig,
);
azureCompletions.setResponsesProvider(azureResponses);
this.#providers['azure-openai'] = azureCompletions;
this.#providers['azure-openai-responses'] = azureResponses;
}
const openaiKey = readKey(providers['openai-completion']);
if (openaiKey) {
const openaiStores = {
@@ -869,6 +918,18 @@ export class ChatCompletionDriver extends PuterDriver {
);
}
const minimax = providers['minimax'];
const minimaxKey = readKey(minimax);
if (minimaxKey) {
this.#providers['minimax'] = new MiniMaxProvider(
{
apiKey: minimaxKey,
apiBaseUrl: minimax?.apiBaseUrl as string | undefined,
},
metering,
);
}
const zai = providers['zai'];
const zaiKey = readKey(zai);
if (zaiKey) {
@@ -0,0 +1,101 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Integration test for the Azure AI Foundry chat-completions provider.
*
* Hits the real Azure endpoint. Exercises both flavours of model the
* provider fronts:
* - an OpenAI model (`gpt-4o`, non-reasoning so `max_tokens=16` returns
* visible text), and
* - an xAI Grok model (`grok-4-20-non-reasoning`), which regression-tests
* the `safety_identifier` param being stripped for Grok (Azure's Grok
* deployments 400 on that OpenAI-only argument).
*
* Skipped unless both `PUTER_TEST_AI_AZURE_OPENAI_API_KEY` and
* `PUTER_TEST_AI_AZURE_OPENAI_API_URL` are set.
*/
import { describe, expect, it } from 'vitest';
import {
INTEGRATION_TEST_TIMEOUT_MS,
makeMeteringStub,
optionalEnv,
skipUnlessEnv,
withTestActor,
} from '../../../integrationTestUtil.js';
import { AzureChatProvider } from './AzureChatProvider.js';
const KEY_ENV = 'PUTER_TEST_AI_AZURE_OPENAI_API_KEY';
const URL_ENV = 'PUTER_TEST_AI_AZURE_OPENAI_API_URL';
describe.skipIf(skipUnlessEnv(KEY_ENV) || skipUnlessEnv(URL_ENV))(
'AzureChatProvider (integration)',
() => {
const buildProvider = () =>
new AzureChatProvider(
makeMeteringStub(),
{ fsEntry: undefined as never, s3Object: undefined as never },
undefined as never,
{ apiKey: optionalEnv(KEY_ENV)!, apiURL: optionalEnv(URL_ENV)! },
);
const expectNonEmptyText = (result: unknown) => {
const text = (result as { message?: { content?: string } }).message
?.content;
expect(typeof text === 'string' && text.length > 0).toBe(true);
};
it(
'returns a non-empty completion from gpt-4o',
{ timeout: INTEGRATION_TEST_TIMEOUT_MS },
async () => {
const provider = buildProvider();
const result = await withTestActor(() =>
provider.complete({
model: 'gpt-4o',
messages: [
{ role: 'user', content: 'Say hi in one word.' },
],
max_tokens: 16,
}),
);
expectNonEmptyText(result);
},
);
it(
'returns a non-empty completion from grok-4-20-non-reasoning (no safety_identifier 400)',
{ timeout: INTEGRATION_TEST_TIMEOUT_MS },
async () => {
const provider = buildProvider();
const result = await withTestActor(() =>
provider.complete({
model: 'grok-4-20-non-reasoning',
messages: [
{ role: 'user', content: 'Say hi in one word.' },
],
max_tokens: 16,
}),
);
expectNonEmptyText(result);
},
);
},
);
@@ -0,0 +1,277 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { HttpError } from '@heyputer/backend/src/core/http/HttpError.js';
import { OpenAI } from 'openai';
import { ChatCompletionCreateParams } from 'openai/resources/index.js';
import { Context } from '../../../../core/context.js';
import type { FSService } from '../../../../services/fs/FSService.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import type { FSEntryStore } from '../../../../stores/fs/FSEntryStore.js';
import type { S3ObjectStore } from '../../../../stores/fs/S3ObjectStore.js';
import type { IChatProvider, ICompleteArguments } from '../../types.js';
import {
messagesHaveCompaction,
wantsCompaction,
} from '../../utils/compaction.js';
import * as OpenAiUtil from '../../utils/OpenAIUtil.js';
import { processPuterPathUploads } from '../openai/fileUpload.js';
import { AZURE_MODELS } from './models.js';
/**
* AzureChatProvider exposes the models we serve through Azure AI Foundry.
* Despite the name, this is not OpenAI-only — Azure AI also fronts xAI's Grok
* models — so it carries its own {@link AZURE_MODELS} list instead of reusing
* the OpenAI one. It speaks the OpenAI-compatible Chat Completions API,
* pointing the client at a configurable Azure endpoint authenticated with an
* Azure-issued API key.
*
* Billing note: the model `costs` are the standard public OpenAI / xAI list
* prices, NOT Azure's. Azure is subsidised for us, so routing through it is
* cheaper while we still bill users at the normal model price.
*
* Implements the puter-chat-completion interface and handles usage tracking,
* spending records, and content moderation.
*/
export class AzureChatProvider implements IChatProvider {
/**
* @type {import('openai').OpenAI}
*/
#openAi: OpenAI;
#defaultModel = 'gpt-5.4-nano';
#meteringService: MeteringService;
#stores: { fsEntry: FSEntryStore; s3Object: S3ObjectStore };
#fsService: FSService;
// Sibling Responses-API provider (Azure or OpenAI) used to handle
// Responses-only features like web_search. Typed loosely since we only
// ever forward `complete()` to it.
#responsesProvider: IChatProvider | null = null;
constructor(
meteringService: MeteringService,
stores: { fsEntry: FSEntryStore; s3Object: S3ObjectStore },
fsService: FSService,
config: { apiKey: string; apiURL: string },
) {
this.#meteringService = meteringService;
this.#stores = stores;
this.#fsService = fsService;
this.#openAi = new OpenAI({
apiKey: config.apiKey,
baseURL: config.apiURL,
});
}
checkModeration(_text: string): { flagged: boolean; categories: string[] } {
throw new Error('Method not implemented.');
}
// Wired up by the driver after the OpenAI providers are built, so the
// Chat Completions path can delegate `web_search` tool calls (Responses-only)
// to the OpenAI Responses provider without a circular constructor dependency.
setResponsesProvider(provider: IChatProvider): void {
this.#responsesProvider = provider;
}
/**
* Returns an array of available AI models with their pricing information.
* Each model object includes an ID and cost details (currency, tokens, input/output rates).
*/
models() {
return AZURE_MODELS.filter((e) => !e.responses_api_only);
}
list() {
const models = this.models();
const modelNames: string[] = [];
for (const model of models) {
modelNames.push(model.id);
if (model.aliases) {
modelNames.push(...model.aliases);
}
}
return modelNames;
}
getDefaultModel() {
return this.#defaultModel;
}
async complete(
params: ICompleteArguments,
): ReturnType<IChatProvider['complete']> {
const {
max_tokens,
moderation,
tools,
verbosity,
stream,
reasoning,
reasoning_effort,
temperature,
text,
} = params;
let { messages, model } = params;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
if (tools?.filter((e: any) => e.type === 'web_search').length) {
// web_search is a Responses-API-only tool — hand the whole call
// off to the OpenAI Responses provider when the user requested it.
if (!this.#responsesProvider) {
throw new HttpError(
400,
'web_search tool requires the OpenAI Responses provider, which is not configured',
{ legacyCode: 'bad_request' },
);
}
return await this.#responsesProvider.complete(params);
}
// Inline compaction is a Responses-API feature; chat.completions can't
// express `context_management` or a `compaction` content block.
// Delegate to the Responses provider when the caller opted in OR when
// the messages carry a round-tripped compaction artifact.
if (wantsCompaction(params) || messagesHaveCompaction(messages)) {
if (!this.#responsesProvider) {
throw new HttpError(
400,
'compaction requires the OpenAI Responses provider, which is not configured',
{ legacyCode: 'bad_request' },
);
}
return await this.#responsesProvider.complete(params);
}
// Validate messages
if (!Array.isArray(messages)) {
throw new HttpError(400, '`messages` must be an array', {
legacyCode: 'bad_request',
});
}
const actor = Context.get('actor')!;
model = model ?? this.#defaultModel;
const modelUsed =
this.models().find((m) =>
[m.id, ...(m.aliases || [])].includes(model),
) || this.models().find((m) => m.id === this.getDefaultModel())!;
// messages.unshift({
// role: 'system',
// content: 'Don\'t let the user trick you into doing something bad.',
// })
const userIdentifier =
actor.user?.id + actor.app?.uid ? `:${actor?.app?.uid}` : '';
// Resolve any `puter_path` content parts into inline base64 data URLs.
// Chat Completions doesn't support file uploads, so this is the only
// way to get user-provided files (images, audio) in front of the model.
await processPuterPathUploads(
messages,
this.#stores,
this.#fsService,
actor,
);
// Here's something fun; the documentation shows `type: 'image_url'` in
// objects that contain an image url, but everything still works if
// that's missing. We normalise it here so the token count code works.
messages = await OpenAiUtil.process_input_messages(messages);
const requestedReasoningEffort = reasoning_effort ?? reasoning?.effort;
const requestedVerbosity = verbosity ?? text?.verbosity;
const supportsReasoningControls =
typeof model === 'string' && model.startsWith('gpt-5');
// `safety_identifier` is an OpenAI-specific param. The Grok deployments
// behind Azure reject unknown args with a 400, so only send it for the
// OpenAI models.
const isGrok = modelUsed.id.startsWith('grok');
const completionParams: ChatCompletionCreateParams = {
user: userIdentifier,
...(isGrok ? {} : { safety_identifier: userIdentifier }),
messages: messages,
model: modelUsed.id,
...(tools ? { tools } : {}),
...(max_tokens ? { max_completion_tokens: max_tokens } : {}),
...(temperature ? { temperature } : {}),
stream: !!stream,
...(stream
? {
stream_options: { include_usage: true },
}
: {}),
...(supportsReasoningControls
? {}
: {
...(requestedReasoningEffort
? { reasoning_effort: requestedReasoningEffort }
: {}),
...(requestedVerbosity
? { verbosity: requestedVerbosity }
: {}),
}),
} as ChatCompletionCreateParams;
const completion =
await this.#openAi.chat.completions.create(completionParams);
return OpenAiUtil.handle_completion_output({
usage_calculator: ({ usage }) => {
const cachedTokens =
usage.prompt_tokens_details?.cached_tokens ?? 0;
const trackedUsage = {
// OpenAI includes cached tokens in `prompt_tokens`, so we
// subtract them out to meter the non-cached remainder. Grok
// reports `prompt_tokens` already excluding cached tokens
// (they're additive, not a subset) — subtracting there
// underflows to a negative count, which bills a negative
// (crediting) cost. Match xAI's `extractMeteredUsage` and
// take Grok's `prompt_tokens` as-is.
prompt_tokens: isGrok
? (usage.prompt_tokens ?? 0)
: (usage.prompt_tokens ?? 0) - cachedTokens,
completion_tokens: usage.completion_tokens ?? 0,
cached_tokens: cachedTokens,
};
const costsOverrideFromModel = Object.fromEntries(
Object.entries(trackedUsage).map(([k, v]) => {
return [k, v * modelUsed.costs[k]];
}),
);
this.#meteringService.utilRecordUsageObject(
trackedUsage,
actor!,
`azure-openai:${modelUsed?.id}`,
costsOverrideFromModel,
);
return trackedUsage;
},
stream,
completion,
moderate: moderation ? this.checkModeration.bind(this) : undefined,
});
}
}
@@ -0,0 +1,83 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Integration test for the Azure AI Foundry Responses provider.
*
* Hits the real Azure endpoint with `gpt-5-codex` — a Responses-API-only
* (Codex) model that the Chat Completions endpoint rejects. This verifies
* the completions/responses split actually routes Codex correctly. Codex is
* a reasoning model, so we give it a generous `max_tokens` and low reasoning
* effort to leave room for visible output.
*
* Skipped unless both `PUTER_TEST_AI_AZURE_OPENAI_API_KEY` and
* `PUTER_TEST_AI_AZURE_OPENAI_API_URL` are set.
*/
import { describe, expect, it } from 'vitest';
import {
INTEGRATION_TEST_TIMEOUT_MS,
makeMeteringStub,
optionalEnv,
skipUnlessEnv,
withTestActor,
} from '../../../integrationTestUtil.js';
import { AzureResponsesProvider } from './AzureResponsesProvider.js';
const KEY_ENV = 'PUTER_TEST_AI_AZURE_OPENAI_API_KEY';
const URL_ENV = 'PUTER_TEST_AI_AZURE_OPENAI_API_URL';
describe.skipIf(skipUnlessEnv(KEY_ENV) || skipUnlessEnv(URL_ENV))(
'AzureResponsesProvider (integration)',
() => {
it(
'returns a non-empty completion from gpt-5-codex',
{ timeout: INTEGRATION_TEST_TIMEOUT_MS },
async () => {
const provider = new AzureResponsesProvider(
makeMeteringStub(),
{
fsEntry: undefined as never,
s3Object: undefined as never,
},
undefined as never,
{
apiKey: optionalEnv(KEY_ENV)!,
apiURL: optionalEnv(URL_ENV)!,
},
);
const result = await withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
messages: [
{ role: 'user', content: 'Say hi in one word.' },
],
max_tokens: 2048,
reasoning: { effort: 'low' },
}),
);
const text = (result as { message?: { content?: string } })
.message?.content;
expect(typeof text === 'string' && text.length > 0).toBe(true);
},
);
},
);
@@ -0,0 +1,307 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { OpenAI } from 'openai';
import { ResponseCreateParams } from 'openai/resources/responses/responses.mjs';
import { Context } from '../../../../core/context.js';
import type { FSService } from '../../../../services/fs/FSService.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import type { FSEntryStore } from '../../../../stores/fs/FSEntryStore.js';
import type { S3ObjectStore } from '../../../../stores/fs/S3ObjectStore.js';
import type { IChatProvider, ICompleteArguments } from '../../types.js';
import { toOpenAiContextManagement } from '../../utils/compaction.js';
import * as OpenAiUtil from '../../utils/OpenAIUtil.js';
import { processPuterPathUploads } from '../openai/fileUpload.js';
import { AZURE_MODELS } from './models.js';
import { HttpError } from '@heyputer/backend/src/core/http/HttpError.js';
/**
* AzureResponsesProvider serves the Responses-API-only models we expose through
* Azure AI Foundry (the Codex family and similar). It mirrors
* {@link OpenAiResponsesChatProvider}, but points the OpenAI client at the
* configurable Azure endpoint and draws from {@link AZURE_MODELS}.
*
* Codex / `responses_api_only` models reject the Chat Completions endpoint, so
* the sibling {@link AzureChatProvider} (Chat Completions) filters them out and
* the driver routes them here instead.
*
* Billing note: the model `costs` are the standard public OpenAI list prices,
* NOT Azure's — Azure is subsidised for us.
*/
export class AzureResponsesProvider implements IChatProvider {
/**
* @type {import('openai').OpenAI}
*/
#openAi: OpenAI;
#defaultModel = 'gpt-5-codex';
#meteringService: MeteringService;
#stores: { fsEntry: FSEntryStore; s3Object: S3ObjectStore };
#fsService: FSService;
constructor(
meteringService: MeteringService,
stores: { fsEntry: FSEntryStore; s3Object: S3ObjectStore },
fsService: FSService,
config: { apiKey: string; apiURL: string },
) {
this.#meteringService = meteringService;
this.#stores = stores;
this.#fsService = fsService;
this.#openAi = new OpenAI({
apiKey: config.apiKey,
baseURL: config.apiURL,
});
}
/**
* Returns an array of available AI models with their pricing information.
* Each model object includes an ID and cost details (currency, tokens, input/output rates).
*/
models(extra_params?: { no_restrictions?: boolean }) {
if (extra_params?.no_restrictions) {
return AZURE_MODELS;
}
return AZURE_MODELS.filter((e) => e.responses_api_only === true);
}
list() {
const models = this.models({ no_restrictions: false });
const modelNames: string[] = [];
for (const model of models) {
modelNames.push(model.id);
if (model.aliases) {
modelNames.push(...model.aliases);
}
}
return modelNames;
}
getDefaultModel() {
return this.#defaultModel;
}
async complete({
messages,
model,
max_tokens,
moderation,
tools,
tool_choice,
parallel_tool_calls,
include,
conversation,
compaction,
context_management,
previous_response_id,
instructions,
metadata,
prompt,
prompt_cache_key,
prompt_cache_retention,
store,
top_p,
truncation,
background,
service_tier,
verbosity,
stream,
reasoning,
reasoning_effort,
temperature,
text,
}: ICompleteArguments): ReturnType<IChatProvider['complete']> {
// Validate messages
if (!Array.isArray(messages)) {
throw new HttpError(400, '`messages` must be an array', {
legacyCode: 'bad_request',
});
}
const actor = Context.get('actor');
model = model ?? this.#defaultModel;
const modelUsed =
this.models({ no_restrictions: true }).find((m) =>
[m.id, ...(m.aliases || [])].includes(model),
) ||
this.models({ no_restrictions: true }).find(
(m) => m.id === this.getDefaultModel(),
)!;
const userIdentifier =
actor?.user.id + actor?.app?.uid ? `:${actor?.app?.uid}` : '';
// Resolve any `puter_path` content parts into inline base64 data URLs
// before the Responses API sees them.
await processPuterPathUploads(
messages,
this.#stores,
this.#fsService,
actor,
);
if (tools) {
// Unravel tools to OpenAI Responses API format
// eslint-disable-next-line @typescript-eslint/no-explicit-any
tools = (tools as any).map((e) => {
if (e.type === 'function') {
const tool = e.function;
tool.type = 'function';
return tool;
} else {
return e;
}
});
}
// Here's something fun; the documentation shows `type: 'image_url'` in
// objects that contain an image url, but everything still works if
// that's missing. We normalise it here so the token count code works.
messages =
await OpenAiUtil.process_input_messages_responses_api(messages);
const requestedReasoningEffort = reasoning_effort ?? reasoning?.effort;
const requestedVerbosity = verbosity ?? text?.verbosity;
const supportsReasoningControls =
typeof model === 'string' && model.startsWith('gpt-5');
// Translate the neutral compaction opt-in (or pass a raw
// `context_management` payload through) to OpenAI's Responses shape.
const contextManagement = toOpenAiContextManagement({
compaction,
context_management,
});
const completionParams: ResponseCreateParams = {
user: userIdentifier,
safety_identifier: userIdentifier,
input: messages,
model: modelUsed.id,
...(tools ? { tools } : {}),
...(tool_choice !== undefined ? { tool_choice } : {}),
...(parallel_tool_calls !== undefined
? { parallel_tool_calls }
: {}),
...(include !== undefined ? { include } : {}),
...(contextManagement !== undefined
? { context_management: contextManagement }
: {}),
...(conversation !== undefined ? { conversation } : {}),
...(previous_response_id !== undefined
? { previous_response_id }
: {}),
...(instructions !== undefined ? { instructions } : {}),
...(metadata !== undefined ? { metadata } : {}),
...(prompt !== undefined ? { prompt } : {}),
...(prompt_cache_key !== undefined ? { prompt_cache_key } : {}),
...(prompt_cache_retention !== undefined
? { prompt_cache_retention }
: {}),
...(store !== undefined ? { store } : {}),
...(max_tokens !== undefined
? { max_output_tokens: max_tokens }
: {}),
...(temperature !== undefined ? { temperature } : {}),
...(top_p !== undefined ? { top_p } : {}),
...(truncation !== undefined ? { truncation } : {}),
...(background !== undefined ? { background } : {}),
...(service_tier !== undefined ? { service_tier } : {}),
...(stream !== undefined ? { stream: !!stream } : {}),
...(text !== undefined ? { text } : {}),
...(supportsReasoningControls
? {}
: {
...(requestedReasoningEffort
? { reasoning_effort: requestedReasoningEffort }
: {}),
...(requestedVerbosity
? { verbosity: requestedVerbosity }
: {}),
}),
...(supportsReasoningControls && reasoning ? { reasoning } : {}),
} as ResponseCreateParams;
const completion =
await this.#openAi.responses.create(completionParams);
return OpenAiUtil.handle_completion_output_responses_api({
usage_calculator: ({ usage }) => {
const trackedUsage = {
prompt_tokens:
// eslint-disable-next-line @typescript-eslint/no-explicit-any
((usage as any).input_tokens ?? 0) -
// eslint-disable-next-line @typescript-eslint/no-explicit-any
((usage as any).input_tokens_details?.cached_tokens ??
0),
// eslint-disable-next-line @typescript-eslint/no-explicit-any
completion_tokens: (usage as any).output_tokens ?? 0,
cached_tokens:
// eslint-disable-next-line @typescript-eslint/no-explicit-any
(usage as any).input_tokens_details?.cached_tokens ?? 0,
};
const costsOverrideFromModel = Object.fromEntries(
Object.entries(trackedUsage).map(([k, v]) => {
return [k, v * modelUsed.costs[k]];
}),
);
this.#meteringService.utilRecordUsageObject(
trackedUsage,
actor,
`azure-openai:${modelUsed?.id}`,
costsOverrideFromModel,
);
return trackedUsage;
},
stream,
completion,
moderate: moderation ? this.checkModeration.bind(this) : undefined,
});
}
async checkModeration(text: string) {
// create moderation
const results = await this.#openAi.moderations.create({
model: 'omni-moderation-latest',
input: text,
});
let flagged = false;
for (const result of results?.results ?? []) {
// OpenAI does a crazy amount of false positives. We filter by their 80% interval
const veryFlaggedEntries = Object.entries(
result.category_scores,
).filter((e) => e[1] > 0.8);
if (veryFlaggedEntries.length > 0) {
flagged = true;
break;
}
}
return {
flagged,
results,
};
}
}
@@ -0,0 +1,464 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { IChatModel } from '../../types.js';
// Models served through our Azure AI Foundry deployment. This is NOT just
// OpenAI — Azure AI also fronts xAI's Grok models — so the list lives in its
// own provider folder rather than sharing the OpenAI list.
//
// IMPORTANT: the `costs` below intentionally mirror the public list prices of
// the equivalent OpenAI / xAI models (see `../openai/models.ts` and
// `../xai/models.ts`). Azure is subsidised for us, so our actual spend is
// lower — but we bill users at the standard model price, which is the whole
// reason we route through Azure. Do NOT replace these with Azure's own rates.
//
// `id` is the Azure deployment name and is what we send upstream.
export const AZURE_MODELS: IChatModel[] = [
// -- xAI Grok (via Azure AI Foundry) -----------------------------------
{
// Costs mirror xai grok-4-1-fast-non-reasoning.
puterId: 'azure:x-ai/grok-4-1-fast-non-reasoning',
id: 'grok-4-1-fast-non-reasoning',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-07',
release_date: '2025-11-19',
name: 'Grok 4.1 Fast (Non-Reasoning)',
aliases: ['x-ai/grok-4-1-fast-non-reasoning'],
context: 2_000_000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 20,
completion_tokens: 50,
cached_tokens: 5,
},
max_tokens: 2_000_000,
},
{
// Costs mirror xai grok-4-1-fast (alias grok-4-1-fast-reasoning).
puterId: 'azure:x-ai/grok-4-1-fast-reasoning',
id: 'grok-4-1-fast-reasoning',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-07',
release_date: '2025-11-19',
name: 'Grok 4.1 Fast (Reasoning)',
aliases: ['x-ai/grok-4-1-fast-reasoning'],
context: 2_000_000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 20,
completion_tokens: 50,
cached_tokens: 5,
},
max_tokens: 2_000_000,
},
{
// Costs mirror xai grok-4.3.
puterId: 'azure:x-ai/grok-4.3',
id: 'grok-4.3',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
release_date: '2026-05-01',
name: 'Grok 4.3',
aliases: ['x-ai/grok-4.3'],
context: 1_000_000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 125,
completion_tokens: 250,
cached_tokens: 20,
},
max_tokens: 30_000,
},
{
// Costs mirror xai grok-4.20 (grok-4.20-0309-non-reasoning).
puterId: 'azure:x-ai/grok-4-20-non-reasoning',
id: 'grok-4-20-non-reasoning',
modalities: { input: ['text', 'image', 'pdf'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-07',
release_date: '2026-03-09',
name: 'Grok 4.20 (Non-Reasoning)',
aliases: ['x-ai/grok-4-20-non-reasoning'],
context: 2_000_000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 125,
completion_tokens: 250,
cached_tokens: 20,
},
max_tokens: 30_000,
},
{
// Costs mirror xai grok-4.20 (grok-4.20-0309-reasoning).
puterId: 'azure:x-ai/grok-4-20-reasoning',
id: 'grok-4-20-reasoning',
modalities: { input: ['text', 'image', 'pdf'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-07',
release_date: '2026-03-09',
name: 'Grok 4.20 (Reasoning)',
aliases: ['x-ai/grok-4-20-reasoning'],
context: 2_000_000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 125,
completion_tokens: 250,
cached_tokens: 20,
},
max_tokens: 30_000,
},
// -- OpenAI (via Azure AI Foundry) -------------------------------------
{
// Costs mirror openai gpt-5.
puterId: 'azure:openai/gpt-5',
id: 'gpt-5',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2024-09-30',
release_date: '2025-08-07',
aliases: ['openai/gpt-5'],
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 125,
cached_tokens: 13,
completion_tokens: 1000,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5-codex (same list price as gpt-5).
puterId: 'azure:openai/gpt-5-codex',
id: 'gpt-5-codex',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2024-09-30',
release_date: '2025-08-07',
aliases: ['openai/gpt-5-codex'],
// Codex models are Responses-API only on Azure, same as OpenAI.
responses_api_only: true,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 125,
cached_tokens: 13,
completion_tokens: 1000,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5-nano.
puterId: 'azure:openai/gpt-5-nano',
id: 'gpt-5-nano',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2024-05-30',
release_date: '2025-08-07',
aliases: ['openai/gpt-5-nano'],
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 5,
cached_tokens: 1,
completion_tokens: 40,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5-mini.
puterId: 'azure:openai/gpt-5-mini',
id: 'gpt-5-mini',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2024-05-30',
release_date: '2025-08-07',
aliases: ['openai/gpt-5-mini'],
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 25,
cached_tokens: 3,
completion_tokens: 200,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-4o.
puterId: 'azure:openai/gpt-4o',
id: 'gpt-4o',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2023-09',
release_date: '2024-05-13',
aliases: ['openai/gpt-4o'],
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 250,
cached_tokens: 125,
completion_tokens: 1000,
},
context: 128_000,
max_tokens: 16384,
},
{
// Costs mirror openai gpt-5.1-codex-mini.
puterId: 'azure:openai/gpt-5.1-codex-mini',
id: 'gpt-5.1-codex-mini',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2024-09-30',
release_date: '2025-11-13',
aliases: ['openai/gpt-5.1-codex-mini'],
responses_api_only: true,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 25,
cached_tokens: 3,
completion_tokens: 200,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5.1.
puterId: 'azure:openai/gpt-5.1',
id: 'gpt-5.1',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2024-09-30',
release_date: '2025-11-13',
aliases: ['openai/gpt-5.1'],
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 125,
cached_tokens: 13,
completion_tokens: 1000,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5.1-codex.
puterId: 'azure:openai/gpt-5.1-codex',
id: 'gpt-5.1-codex',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2024-09-30',
release_date: '2025-11-13',
aliases: ['openai/gpt-5.1-codex'],
responses_api_only: true,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 125,
cached_tokens: 13,
completion_tokens: 1000,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5.2.
puterId: 'azure:openai/gpt-5.2',
id: 'gpt-5.2',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-08-31',
release_date: '2025-12-11',
aliases: ['openai/gpt-5.2'],
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 175,
cached_tokens: 17.5,
completion_tokens: 1400,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5.2-codex.
puterId: 'azure:openai/gpt-5.2-codex',
id: 'gpt-5.2-codex',
modalities: { input: ['text', 'image', 'pdf'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-08-31',
release_date: '2025-12-11',
aliases: ['openai/gpt-5.2-codex'],
responses_api_only: true,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 175,
cached_tokens: 18,
completion_tokens: 1400,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5.4-nano.
puterId: 'azure:openai/gpt-5.4-nano',
id: 'gpt-5.4-nano',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-08-31',
release_date: '2026-03-19',
aliases: ['openai/gpt-5.4-nano'],
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 20,
cached_tokens: 2,
completion_tokens: 125,
},
context: 400_000,
max_tokens: 128_000,
},
{
// Costs mirror openai gpt-5.4-mini.
puterId: 'azure:openai/gpt-5.4-mini',
id: 'gpt-5.4-mini',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-08-31',
aliases: ['openai/gpt-5.4-mini'],
responses_api_only: true,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 75,
cached_tokens: 7.5,
completion_tokens: 450,
},
context: 400_000,
max_tokens: 128_000,
},
{
// Costs mirror openai gpt-5.3-codex.
puterId: 'azure:openai/gpt-5.3-codex',
id: 'gpt-5.3-codex',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-08-31',
aliases: ['openai/gpt-5.3-codex'],
responses_api_only: true,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 175,
cached_tokens: 17.5,
completion_tokens: 1400,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5.4.
puterId: 'azure:openai/gpt-5.4',
id: 'gpt-5.4',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-08-31',
release_date: '2026-03-05',
aliases: ['openai/gpt-5.4'],
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 250,
cached_tokens: 25,
completion_tokens: 1500,
},
context: 1_050_000,
max_tokens: 1_050_000,
},
];
@@ -486,6 +486,52 @@ describe('ClaudeProvider.complete non-stream output', () => {
10 * Number(haiku.costs.cache_read_input_tokens),
);
});
it('bills the compaction pass by summing usage.iterations', async () => {
const { provider } = makeProvider();
// Per Anthropic: top-level input/output reflect only the message pass;
// the compaction pass lives in `iterations` and must be summed to bill.
const msg = {
content: [{ type: 'text', text: 'done' }],
usage: {
input_tokens: 23000, // message pass only (NOT the total)
output_tokens: 1000,
iterations: [
{
type: 'compaction',
input_tokens: 180000,
output_tokens: 3500,
},
{ type: 'message', input_tokens: 23000, output_tokens: 1000 },
],
},
};
messagesCreateMock.mockResolvedValueOnce(msg);
const result = (await withTestActor(() =>
provider.complete({
model: 'claude-haiku-4-5-20251001',
messages: [{ role: 'user', content: 'hi' }],
compaction: true,
}),
)) as { usage: Record<string, number> };
// Totals are the SUM across iterations, not the top-level fields.
expect(result.usage.input_tokens).toBe(203000); // 180000 + 23000
expect(result.usage.output_tokens).toBe(4500); // 3500 + 1000
const haiku = CLAUDE_MODELS.find(
(m) => m.id === 'claude-haiku-4-5-20251001',
)!;
const [usage, , , overrides] = recordSpy.mock.calls[0]!;
expect(usage.input_tokens).toBe(203000);
expect(overrides.input_tokens).toBe(
203000 * Number(haiku.costs.input_tokens),
);
expect(overrides.output_tokens).toBe(
4500 * Number(haiku.costs.output_tokens),
);
});
});
// ── Streaming deltas ────────────────────────────────────────────────
@@ -618,6 +664,220 @@ describe('ClaudeProvider.complete streaming', () => {
});
});
// ── Inline compaction ───────────────────────────────────────────────
describe('ClaudeProvider.complete compaction', () => {
it('translates the neutral opt-in to context_management + the compaction beta', async () => {
const { provider } = makeProvider();
messagesCreateMock.mockResolvedValueOnce({
content: [{ type: 'text', text: 'hi' }],
usage: { input_tokens: 1, output_tokens: 1 },
});
await withTestActor(() =>
provider.complete({
model: 'claude-haiku-4-5-20251001',
messages: [{ role: 'user', content: 'hi' }],
compaction: { trigger_tokens: 50000 },
}),
);
const [args] = messagesCreateMock.mock.calls[0]!;
expect(args.context_management).toEqual({
edits: [
{
type: 'compact_20260112',
trigger: { type: 'input_tokens', value: 50000 },
},
],
});
expect(args.betas).toContain('compact-2026-01-12');
});
it('emits a canonical compaction event from a streamed compaction block', async () => {
const { provider } = makeProvider();
messagesStreamMock.mockReturnValueOnce(
makeStreamLike(
[
{ type: 'message_start' },
{
type: 'content_block_start',
content_block: {
type: 'compaction',
id: 'cmpct_1',
content: 'ENC', // Anthropic carries the summary here
},
},
{ type: 'content_block_stop' },
{
type: 'message_delta',
usage: { input_tokens: 1, output_tokens: 1 },
},
{ type: 'message_stop' },
],
{ input_tokens: 1, output_tokens: 1 },
),
);
const result = await withTestActor(() =>
provider.complete({
model: 'claude-haiku-4-5-20251001',
messages: [{ role: 'user', content: 'hi' }],
stream: true,
compaction: true,
}),
);
const harness = makeCapturingChatStream();
await (
result as {
init_chat_stream: (p: { chatStream: unknown }) => Promise<void>;
}
).init_chat_stream({ chatStream: harness.chatStream });
const compaction = harness
.events()
.find((e) => e.type === 'compaction');
expect(compaction).toEqual({
type: 'compaction',
id: 'cmpct_1',
encrypted_content: 'ENC',
});
});
it('enables the compaction beta when a round-tripped compaction block is resent (no opt-in)', async () => {
const { provider } = makeProvider();
messagesCreateMock.mockResolvedValueOnce({
content: [{ type: 'text', text: 'ok' }],
usage: { input_tokens: 1, output_tokens: 1 },
});
await withTestActor(() =>
provider.complete({
model: 'claude-haiku-4-5-20251001',
messages: [
{ role: 'user', content: 'continue' },
{
role: 'assistant',
content: [
{
type: 'compaction',
id: 'cmpct_1',
encrypted_content: 'ENC',
},
],
},
],
// note: no `compaction`/`context_management` opt-in
}),
);
const [args] = messagesCreateMock.mock.calls[0]!;
expect(args.betas).toContain('compact-2026-01-12');
// No new compaction was requested, so no context_management is sent.
expect(args.context_management).toBeUndefined();
});
it('surfaces a compaction block from a non-streaming response as result.compaction', async () => {
const { provider } = makeProvider();
messagesCreateMock.mockResolvedValueOnce({
content: [
{ type: 'text', text: 'done' },
{
type: 'compaction',
id: 'cmpct_2',
content: 'ENC2', // Anthropic carries the summary here
},
],
usage: { input_tokens: 1, output_tokens: 1 },
});
const result = (await withTestActor(() =>
provider.complete({
model: 'claude-haiku-4-5-20251001',
messages: [{ role: 'user', content: 'hi' }],
compaction: true,
}),
)) as { compaction?: { id?: string; encrypted_content: string } };
// Anthropic's `content` is surfaced under the unified `encrypted_content`.
expect(result.compaction).toEqual({
type: 'compaction',
id: 'cmpct_2',
encrypted_content: 'ENC2',
});
});
it('maps a round-tripped compaction block back to Anthropic `content` on input', async () => {
const { provider } = makeProvider();
messagesCreateMock.mockResolvedValueOnce({
content: [{ type: 'text', text: 'ok' }],
usage: { input_tokens: 1, output_tokens: 1 },
});
await withTestActor(() =>
provider.complete({
model: 'claude-haiku-4-5-20251001',
messages: [
{ role: 'user', content: 'continue' },
{
role: 'assistant',
content: [
{ type: 'compaction', encrypted_content: 'SUMMARY' },
],
},
],
}),
);
const [args] = messagesCreateMock.mock.calls[0]!;
const compactionBlock = args.messages
.flatMap((m: { content?: unknown[] }) =>
Array.isArray(m.content) ? m.content : [],
)
.find((c: { type?: string }) => c?.type === 'compaction');
// Internal `encrypted_content` carrier → Anthropic native `content`.
expect(compactionBlock).toEqual({
type: 'compaction',
content: 'SUMMARY',
});
});
it('preserves the reply text when an assistant turn carries compaction + text', async () => {
const { provider } = makeProvider();
messagesCreateMock.mockResolvedValueOnce({
content: [{ type: 'text', text: 'ok' }],
usage: { input_tokens: 1, output_tokens: 1 },
});
await withTestActor(() =>
provider.complete({
model: 'claude-haiku-4-5-20251001',
messages: [
{
role: 'assistant',
content: [
{ type: 'compaction', encrypted_content: 'SUMMARY' },
{ type: 'text', text: 'earlier reply' },
],
},
{ role: 'user', content: 'continue' },
],
}),
);
const [args] = messagesCreateMock.mock.calls[0]!;
const blocks = args.messages.flatMap((m: { content?: unknown[] }) =>
Array.isArray(m.content) ? m.content : [],
);
// Compaction mapped to Anthropic `content`, AND the reply text kept.
expect(blocks).toContainEqual({
type: 'compaction',
content: 'SUMMARY',
});
expect(blocks).toContainEqual({ type: 'text', text: 'earlier reply' });
});
});
// ── Moderation ──────────────────────────────────────────────────────
describe('ClaudeProvider.checkModeration', () => {
@@ -34,6 +34,10 @@ import type {
ICompleteArguments,
IChatCompleteResult,
} from '../../types.js';
import {
messagesHaveCompaction,
toAnthropicContextManagement,
} from '../../utils/compaction.js';
import { make_claude_tools } from '../../utils/FunctionCalling.js';
import { extract_and_remove_system_messages } from '../../utils/Messages.js';
import type {
@@ -44,6 +48,11 @@ import type {
import { FILES_API_BETA, processPuterPathUploads } from './fileUpload.js';
import { CLAUDE_MODELS } from './models.js';
// Anthropic inline-compaction beta. The vendored SDK (0.68.0) doesn't type the
// `compact_20260112` edit or the `compaction` content block, so the request
// params and streamed/returned blocks are handled with `as any` casts.
const COMPACTION_BETA = 'compact-2026-01-12';
export class ClaudeProvider implements IChatProvider {
anthropic: Anthropic;
@@ -97,9 +106,18 @@ export class ClaudeProvider implements IChatProvider {
temperature,
reasoning,
reasoning_effort,
compaction,
context_management,
}: ICompleteArguments): Promise<IChatCompleteResult> {
tools = make_claude_tools(tools);
// Translate the neutral compaction opt-in (or pass a raw
// `context_management` payload through) to Anthropic's beta shape.
const contextManagement = toAnthropicContextManagement({
compaction,
context_management,
});
// eslint-disable-next-line @typescript-eslint/no-explicit-any
let system_prompts: string | any[];
[system_prompts, messages] =
@@ -218,6 +236,23 @@ export class ClaudeProvider implements IChatProvider {
return message;
});
// Map round-tripped compaction blocks back to Anthropic's native shape.
// The internal/unified carrier field is `encrypted_content`; Anthropic's
// compaction block uses `content` (a plaintext summary).
// eslint-disable-next-line @typescript-eslint/no-explicit-any
messages = messages.map((message: any) => {
if (!Array.isArray(message.content)) return message;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
message.content = message.content.map((part: any) => {
if (part?.type !== 'compaction') return part;
return {
type: 'compaction',
content: part.content ?? part.encrypted_content ?? '',
};
});
return message;
});
const modelUsed =
this.models().find((m) =>
[m.id, ...(m.aliases || [])].includes(model),
@@ -229,15 +264,21 @@ export class ClaudeProvider implements IChatProvider {
reasoningEffort: requestedReasoningEffort,
maxTokens: max_tokens,
});
// Opus 4.7 errors on non-default sampling params; omit temperature entirely.
// Fable 5 and Opus 4.7/4.8 error on non-default sampling params; omit temperature entirely.
// Other models require temperature=1 when thinking is enabled.
const isOpus47 = modelUsed.id === 'claude-opus-4-7';
const resolvedTemperature = isOpus47
const omitsTemperature = [
'claude-fable-5',
'claude-opus-4-7',
'claude-opus-4-8',
].includes(modelUsed.id);
const resolvedTemperature = omitsTemperature
? undefined
: thinkingConfig
? 1
: (temperature ?? 0);
const supportsEffort = [
'claude-fable-5',
'claude-opus-4-8',
'claude-opus-4-7',
'claude-opus-4-6',
'claude-sonnet-4-6',
@@ -256,6 +297,18 @@ export class ClaudeProvider implements IChatProvider {
actor,
);
const usesBetaFiles = uploadedFileIds.length > 0;
// The compaction beta is needed both to *request* compaction
// (contextManagement) and to *accept a round-tripped* compaction block
// back as input (messagesHaveCompaction).
const usesCompaction =
!!contextManagement || messagesHaveCompaction(messages);
// Compaction and Files API both require the beta endpoint; combine their
// beta headers and route through `beta.messages.*` if either is active.
const betas = [
...(usesBetaFiles ? [FILES_API_BETA] : []),
...(usesCompaction ? [COMPACTION_BETA] : []),
];
const usesBeta = betas.length > 0;
const sdkParams: MessageCreateParams & {
betas?: string[];
@@ -285,7 +338,11 @@ export class ClaudeProvider implements IChatProvider {
...(supportsEffort && requestedReasoningEffort
? { output_config: { effort: requestedReasoningEffort } }
: {}),
...(usesBetaFiles ? { betas: [FILES_API_BETA] } : {}),
// Cast: `context_management` compaction edits aren't typed in SDK 0.68.0.
...(contextManagement
? { context_management: contextManagement as any }
: {}),
...(usesBeta ? { betas } : {}),
} as MessageCreateParams & { betas?: string[] };
const cleanupUploads = async () => {
@@ -309,13 +366,23 @@ export class ClaudeProvider implements IChatProvider {
}: {
chatStream: AIChatStream;
}) => {
const completion = usesBetaFiles
const completion = usesBeta
? this.anthropic.beta.messages.stream(sdkParams)
: this.anthropic.messages.stream(sdkParams);
const usageSum: Record<string, number> = {};
let message, contentBlock;
let currentContentBlockType: string | null = null;
// Inline-compaction block is an untyped beta block; capture its
// artifact across start/delta and emit on stop. Anthropic carries
// the summary in `content` (plaintext) — unlike OpenAI's
// `encrypted_content` — so read `content` first.
let compactionData: {
id?: string;
payload: string;
buffer: string;
} | null = null;
let emittedCompaction = false;
for await (const event of completion) {
if (event.type === 'message_delta') {
const meteredData = this.#usageFormatterUtil(
@@ -339,6 +406,24 @@ export class ClaudeProvider implements IChatProvider {
}
if (event.type === 'content_block_start') {
currentContentBlockType = event.content_block.type;
if (
(event.content_block.type as string) ===
'compaction'
) {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const block = event.content_block as any;
compactionData = {
id: block.id,
// Anthropic uses `content`; fall back to
// `encrypted_content` in case the field varies.
payload:
block.content ??
block.encrypted_content ??
'',
buffer: '',
};
continue;
}
if (event.content_block.type === 'tool_use') {
contentBlock = message!.contentBlock({
type: event.content_block.type,
@@ -357,12 +442,42 @@ export class ClaudeProvider implements IChatProvider {
continue;
}
if (event.type === 'content_block_stop') {
if (currentContentBlockType === 'compaction') {
const encrypted_content =
compactionData?.payload ||
compactionData?.buffer ||
'';
// Only emit (and mark done) if we actually captured
// the summary; otherwise let the finalMessage
// fallback recover it from the complete block.
if (encrypted_content) {
chatStream.compaction({
id: compactionData?.id,
encrypted_content,
});
emittedCompaction = true;
}
compactionData = null;
currentContentBlockType = null;
continue;
}
contentBlock!.end();
contentBlock = null;
currentContentBlockType = null;
continue;
}
if (event.type === 'content_block_delta') {
if (currentContentBlockType === 'compaction') {
// Capture any streamed payload for the artifact.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const d = event.delta as any;
const chunk =
d.partial_json ?? d.text ?? d.data ?? '';
if (typeof chunk === 'string' && compactionData) {
compactionData.buffer += chunk;
}
continue;
}
if (event.delta.type === 'input_json_delta') {
(contentBlock as AIChatToolUseStream)!.addPartialJSON(
event.delta.partial_json,
@@ -385,18 +500,37 @@ export class ClaudeProvider implements IChatProvider {
// signature_delta — ignored
}
}
const finalUsage = await completion
const finalMessage = await completion
.finalMessage()
.then((msg) =>
this.#usageFormatterUtil(
msg.usage as Usage | BetaUsage,
),
)
.catch(() => null);
if (finalUsage) {
if (finalMessage) {
const finalUsage = this.#usageFormatterUtil(
finalMessage.usage as Usage | BetaUsage,
);
for (const [key, value] of Object.entries(finalUsage)) {
usageSum[key] = value;
}
// Fallback: some SDK versions surface the compaction block
// only in the final message, not as a streamed block.
if (!emittedCompaction) {
const block = (
(finalMessage.content as unknown[]) ?? []
).find(
// eslint-disable-next-line @typescript-eslint/no-explicit-any
(c: any) => c?.type === 'compaction',
// eslint-disable-next-line @typescript-eslint/no-explicit-any
) as any;
if (block) {
chatStream.compaction({
id: block.id,
encrypted_content:
block.content ??
block.encrypted_content ??
'',
});
emittedCompaction = true;
}
}
}
chatStream.end(usageSum);
const costsOverrideFromModel =
@@ -417,7 +551,7 @@ export class ClaudeProvider implements IChatProvider {
}
try {
const msg = await (usesBetaFiles
const msg = await (usesBeta
? this.anthropic.beta.messages.create(sdkParams)
: this.anthropic.messages.create(sdkParams));
const usage = this.#usageFormatterUtil(
@@ -434,7 +568,37 @@ export class ClaudeProvider implements IChatProvider {
costsOverrideFromModel,
);
return { message: msg, usage, finish_reason: 'stop' };
// Surface any inline-compaction artifact for stateless round-trip.
const compactionBlock = (
((msg as Message).content as unknown[]) ?? []
).find(
// eslint-disable-next-line @typescript-eslint/no-explicit-any
(c: any) => c?.type === 'compaction',
// eslint-disable-next-line @typescript-eslint/no-explicit-any
) as any;
return {
message: msg,
usage,
finish_reason: 'stop',
...(compactionBlock
? {
compaction: {
// `type` makes the artifact a drop-in `messages`
// item for the round-trip (symmetric with the
// streaming compaction chunk).
type: 'compaction' as const,
...(compactionBlock.id !== undefined
? { id: compactionBlock.id }
: {}),
encrypted_content:
compactionBlock.content ??
compactionBlock.encrypted_content ??
'',
},
}
: {}),
};
} finally {
await cleanupUploads();
}
@@ -442,8 +606,31 @@ export class ClaudeProvider implements IChatProvider {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
#usageFormatterUtil(usage: any) {
// Compaction responses report per-pass usage in `usage.iterations` (the
// compaction pass + the message pass). Per Anthropic's docs the
// top-level `input_tokens`/`output_tokens` reflect ONLY the
// non-compaction passes, so to bill the (often large) compaction pass we
// must sum across all iterations. Compaction is billed at normal token
// rates, and iterations don't break out cache fields — those stay
// top-level (the message pass's), which bills the compaction input at
// the full input rate.
const iterations = Array.isArray(usage?.iterations)
? usage.iterations
: null;
const inputTokens = iterations
? iterations.reduce(
(sum: number, it: any) => sum + (it?.input_tokens || 0),
0,
)
: usage?.input_tokens || 0;
const outputTokens = iterations
? iterations.reduce(
(sum: number, it: any) => sum + (it?.output_tokens || 0),
0,
)
: usage?.output_tokens || 0;
return {
input_tokens: usage?.input_tokens || 0,
input_tokens: inputTokens,
ephemeral_5m_input_tokens:
usage?.cache_creation?.ephemeral_5m_input_tokens ||
usage?.cache_creation_input_tokens ||
@@ -451,7 +638,7 @@ export class ClaudeProvider implements IChatProvider {
ephemeral_1h_input_tokens:
usage?.cache_creation?.ephemeral_1h_input_tokens || 0,
cache_read_input_tokens: usage?.cache_read_input_tokens || 0,
output_tokens: usage?.output_tokens || 0,
output_tokens: outputTokens,
thinking_tokens:
usage?.thinking_tokens ||
usage?.output_tokens_details?.thinking_tokens ||
@@ -486,11 +673,16 @@ export class ClaudeProvider implements IChatProvider {
}) {
if (!reasoningEffort) return undefined;
// Opus 4.7, 4.6, and Sonnet 4.6 use adaptive thinking
// (`budget_tokens` is deprecated on 4.6/Sonnet 4.6, removed on 4.7).
// Opus 4.7 omits thinking content by default; `display: 'summarized'`
// restores visible reasoning in the stream.
if (modelId === 'claude-opus-4-7') {
// Fable 5, Opus 4.7/4.8, 4.6, and Sonnet 4.6 use adaptive thinking
// (`budget_tokens` is deprecated on 4.6/Sonnet 4.6, removed on
// Fable 5 and 4.7+). Fable 5 and Opus 4.7/4.8 omit thinking content
// by default; `display: 'summarized'` restores visible reasoning in
// the stream.
if (
modelId === 'claude-fable-5' ||
modelId === 'claude-opus-4-8' ||
modelId === 'claude-opus-4-7'
) {
return {
type: 'adaptive' as const,
display: 'summarized' as const,
@@ -21,6 +21,66 @@ import type { IChatModel } from '../../types.js';
// Hardcoded from https://models.dev/api.json
export const CLAUDE_MODELS: IChatModel[] = [
{
puterId: 'anthropic:anthropic/claude-fable-5',
id: 'claude-fable-5',
modalities: { input: ['text', 'image', 'pdf'], output: ['text'] },
open_weights: false,
tool_call: true,
release_date: '2026-06-09',
aliases: [
'claude-fable',
'claude-fable-latest',
'claude-fable-5-latest',
'claude-fable-5',
'anthropic/claude-fable-5',
],
name: 'Claude Fable 5',
costs_currency: 'usd-cents',
input_cost_key: 'input_tokens',
output_cost_key: 'output_tokens',
costs: {
tokens: 1_000_000,
input_tokens: 1000,
ephemeral_5m_input_tokens: 1000 * 1.25,
ephemeral_1h_input_tokens: 1000 * 2,
cache_read_input_tokens: 1000 * 0.1,
output_tokens: 5000,
},
context: 1000000,
max_tokens: 128000,
},
{
puterId: 'anthropic:anthropic/claude-opus-4-8',
id: 'claude-opus-4-8',
modalities: { input: ['text', 'image', 'pdf'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2026-01',
release_date: '2026-05-28',
aliases: [
'claude-opus',
'claude-opus-latest',
'claude-opus-4-8-latest',
'claude-opus-4.8',
'claude-opus-4-8',
'anthropic/claude-opus-4-8',
],
name: 'Claude Opus 4.8',
costs_currency: 'usd-cents',
input_cost_key: 'input_tokens',
output_cost_key: 'output_tokens',
costs: {
tokens: 1_000_000,
input_tokens: 500,
ephemeral_5m_input_tokens: 500 * 1.25,
ephemeral_1h_input_tokens: 500 * 2,
cache_read_input_tokens: 500 * 0.1,
output_tokens: 2500,
},
context: 1000000,
max_tokens: 128000,
},
{
puterId: 'anthropic:anthropic/claude-opus-4-7',
id: 'claude-opus-4-7',
@@ -30,8 +90,6 @@ export const CLAUDE_MODELS: IChatModel[] = [
knowledge: '2026-01',
release_date: '2026-04-16',
aliases: [
'claude-opus',
'claude-opus-latest',
'claude-opus-4-7-latest',
'claude-opus-4.7',
'claude-opus-4-7',
@@ -355,6 +355,8 @@ describe('GeminiChatProvider.complete non-stream output', () => {
prompt_tokens: 90,
completion_tokens: 50,
cached_tokens: 10,
thinking_tokens: 0,
grounding_requests: 0,
});
// gemini-2.5-flash costs: prompt=30, completion=250, cached=3.
@@ -367,11 +369,15 @@ describe('GeminiChatProvider.complete non-stream output', () => {
prompt_tokens: 90,
completion_tokens: 50,
cached_tokens: 10,
thinking_tokens: 0,
grounding_requests: 0,
});
expect(overrides).toEqual({
prompt_tokens: 90 * Number(flash.costs.prompt_tokens),
completion_tokens: 50 * Number(flash.costs.completion_tokens),
cached_tokens: 10 * Number(flash.costs.cached_tokens ?? 0),
thinking_tokens: 0,
grounding_requests: 0,
});
});
@@ -446,6 +452,8 @@ describe('GeminiChatProvider.complete streaming', () => {
prompt_tokens: 3, // 4 - 1 cached
completion_tokens: 2,
cached_tokens: 1,
thinking_tokens: 0,
grounding_requests: 0,
});
const flash = GEMINI_MODELS.find((m) => m.id === 'gemini-2.5-flash')!;
@@ -456,10 +464,192 @@ describe('GeminiChatProvider.complete streaming', () => {
prompt_tokens: 3 * Number(flash.costs.prompt_tokens),
completion_tokens: 2 * Number(flash.costs.completion_tokens),
cached_tokens: 1 * Number(flash.costs.cached_tokens ?? 0),
thinking_tokens: 0,
grounding_requests: 0,
});
});
});
// ── Thinking-token metering ─────────────────────────────────────────
describe('GeminiChatProvider.complete thinking token metering', () => {
it('splits thinking tokens out of completion_tokens and bills each at its own rate', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'result', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: {
prompt_tokens: 100,
completion_tokens: 60,
completion_tokens_details: { reasoning_tokens: 20 },
prompt_tokens_details: { cached_tokens: 10 },
},
});
const result = await withTestActor(() =>
provider.complete({
model: 'gemini-2.5-flash',
messages: [{ role: 'user', content: 'think hard' }],
}),
);
expect((result as { usage: unknown }).usage).toEqual({
prompt_tokens: 90, // 100 - 10 cached
completion_tokens: 40, // 60 - 20 thinking
cached_tokens: 10,
thinking_tokens: 20,
grounding_requests: 0,
});
const flash = GEMINI_MODELS.find((m) => m.id === 'gemini-2.5-flash')!;
expect(recordSpy).toHaveBeenCalledTimes(1);
const [usage, , , overrides] = recordSpy.mock.calls[0]!;
expect(usage.thinking_tokens).toBe(20);
expect(usage.completion_tokens).toBe(40);
expect(overrides).toMatchObject({
thinking_tokens: 20 * Number(flash.costs.thinking_tokens),
completion_tokens: 40 * Number(flash.costs.completion_tokens),
});
});
it('sets thinking_tokens to 0 when completion_tokens_details is absent', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'ok', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 5, completion_tokens: 3 },
});
await withTestActor(() =>
provider.complete({
model: 'gemini-2.5-flash',
messages: [{ role: 'user', content: 'hi' }],
}),
);
const [usage] = recordSpy.mock.calls[0]!;
expect(usage.thinking_tokens).toBe(0);
expect(usage.completion_tokens).toBe(3);
});
});
// ── Grounding-request metering ───────────────────────────────────────
describe('GeminiChatProvider.complete grounding request metering', () => {
it('charges one grounding request when non-stream response contains grounding_metadata', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: {
content: 'result',
role: 'assistant',
extra_content: {
grounding_metadata: { web_search_queries: ['foo'] },
},
},
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 10, completion_tokens: 5 },
});
await withTestActor(() =>
provider.complete({
model: 'gemini-2.5-flash',
messages: [{ role: 'user', content: 'search for foo' }],
}),
);
const flash = GEMINI_MODELS.find((m) => m.id === 'gemini-2.5-flash')!;
const [usage, , , overrides] = recordSpy.mock.calls[0]!;
expect(usage.grounding_requests).toBe(1);
expect(overrides!.grounding_requests).toBe(
1 * Number(flash.costs.grounding_requests),
);
});
it('does not charge a grounding request when no grounding_metadata is present', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'ok', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 5, completion_tokens: 3 },
});
await withTestActor(() =>
provider.complete({
model: 'gemini-2.5-flash',
messages: [{ role: 'user', content: 'hi' }],
}),
);
const [usage] = recordSpy.mock.calls[0]!;
expect(usage.grounding_requests).toBe(0);
});
it('charges one grounding request when streaming response contains grounding extra_content', async () => {
const { provider } = makeProvider();
createMock.mockReturnValueOnce(
asAsyncIterable([
{ choices: [{ delta: { content: 'ans' } }] },
{
choices: [
{
delta: {
extra_content: {
grounding_metadata: {
web_search_queries: ['bar'],
},
},
},
},
],
},
{
choices: [{ delta: {} }],
usage: { prompt_tokens: 8, completion_tokens: 4 },
},
]),
);
const result = await withTestActor(() =>
provider.complete({
model: 'gemini-2.5-flash',
messages: [{ role: 'user', content: 'search bar' }],
stream: true,
}),
);
const harness = makeCapturingChatStream();
await (
result as {
init_chat_stream: (p: { chatStream: unknown }) => Promise<void>;
}
).init_chat_stream({ chatStream: harness.chatStream });
const flash = GEMINI_MODELS.find((m) => m.id === 'gemini-2.5-flash')!;
expect(recordSpy).toHaveBeenCalledTimes(1);
const [usage, , , overrides] = recordSpy.mock.calls[0]!;
expect(usage.grounding_requests).toBe(1);
expect(overrides!.grounding_requests).toBe(
1 * Number(flash.costs.grounding_requests),
);
});
});
// ── Error mapping ───────────────────────────────────────────────────
describe('GeminiChatProvider.complete error mapping', () => {
@@ -102,24 +102,52 @@ export class GeminiChatProvider implements IChatProvider {
}
return handle_completion_output({
usage_calculator: ({ usage }) => {
usage_calculator: (args) => {
// Cast to access Gemini-specific extras passed alongside usage:
// - choices: non-stream grounding metadata lives in choices[0].message.extra_content
// - extra_content: streaming grounding metadata accumulated by the stream handler
const { usage, choices, extra_content } = args as {
usage: typeof args.usage;
choices?: Array<{
message?: {
extra_content?: { grounding_metadata?: unknown };
};
}>;
extra_content?: { grounding_metadata?: unknown };
};
const cached_tokens =
usage?.prompt_tokens_details?.cached_tokens ?? 0;
// Thinking tokens are a subset of completion_tokens billed at a different rate
const thinking_tokens =
usage?.completion_tokens_details?.reasoning_tokens ?? 0;
const trackedUsage = {
prompt_tokens:
(usage.prompt_tokens ?? 0) -
(usage.prompt_tokens_details?.cached_tokens ?? 0),
completion_tokens: usage.completion_tokens ?? 0,
cached_tokens:
usage.prompt_tokens_details?.cached_tokens ?? 0,
prompt_tokens: (usage?.prompt_tokens ?? 0) - cached_tokens,
completion_tokens: Math.max(
0,
(usage?.completion_tokens ?? 0) - thinking_tokens,
),
cached_tokens,
thinking_tokens,
// Grounding search is a per-request fee not reflected in token counts
grounding_requests:
(choices?.[0]?.message?.extra_content
?.grounding_metadata ??
extra_content?.grounding_metadata)
? 1
: 0,
};
const costsOverrideFromModel = Object.fromEntries(
Object.entries(trackedUsage).map(([k, v]) => {
return [k, v * modelUsed.costs[k]];
return [k, v * (modelUsed.costs[k] ?? 0)];
}),
);
this.meteringService.utilRecordUsageObject(
trackedUsage,
actor,
actor!,
`gemini:${modelUsed?.id}`,
costsOverrideFromModel,
);
@@ -43,7 +43,10 @@ export const GEMINI_MODELS: IChatModel[] = [
tokens: 1_000_000,
prompt_tokens: 150,
completion_tokens: 900,
thinking_tokens: 900,
cached_tokens: 15,
// Gemini 3.x grounding is $14 / 1,000 requests
grounding_requests: 1_400_000,
},
},
{
@@ -68,6 +71,8 @@ export const GEMINI_MODELS: IChatModel[] = [
prompt_tokens: 10,
completion_tokens: 40,
cached_tokens: 3,
// Gemini 2.x grounding is $35 / 1,000 requests
grounding_requests: 3_500_000,
},
max_tokens: 8192,
},
@@ -115,8 +120,13 @@ export const GEMINI_MODELS: IChatModel[] = [
costs: {
tokens: 1_000_000,
prompt_tokens: 30,
// Output is $2.50/M; thinking tokens bill at the same output rate
completion_tokens: 250,
thinking_tokens: 250,
// Cache read is $0.03/M (10% of input)
cached_tokens: 3,
// Gemini 2.x grounding is $35 / 1,000 requests
grounding_requests: 3_500_000,
},
max_tokens: 65536,
},
@@ -141,7 +151,10 @@ export const GEMINI_MODELS: IChatModel[] = [
tokens: 1_000_000,
prompt_tokens: 10,
completion_tokens: 40,
thinking_tokens: 40,
cached_tokens: 1,
// Gemini 2.x grounding is $35 / 1,000 requests
grounding_requests: 3_500_000,
},
max_tokens: 65536,
},
@@ -166,7 +179,10 @@ export const GEMINI_MODELS: IChatModel[] = [
tokens: 1_000_000,
prompt_tokens: 125,
completion_tokens: 1000,
cached_tokens: 13,
thinking_tokens: 1000,
cached_tokens: 31,
// Gemini 2.x grounding is $35 / 1,000 requests
grounding_requests: 3_500_000,
},
max_tokens: 200_000,
},
@@ -191,7 +207,9 @@ export const GEMINI_MODELS: IChatModel[] = [
tokens: 1_000_000,
prompt_tokens: 200,
completion_tokens: 1200,
thinking_tokens: 1200,
cached_tokens: 20,
grounding_requests: 1_400_000,
},
max_tokens: 65536,
},
@@ -216,7 +234,9 @@ export const GEMINI_MODELS: IChatModel[] = [
tokens: 1_000_000,
prompt_tokens: 50,
completion_tokens: 300,
thinking_tokens: 300,
cached_tokens: 5,
grounding_requests: 1_400_000,
},
max_tokens: 65536,
},
@@ -245,7 +265,9 @@ export const GEMINI_MODELS: IChatModel[] = [
tokens: 1_000_000,
prompt_tokens: 25,
completion_tokens: 150,
thinking_tokens: 150,
cached_tokens: 2.5,
grounding_requests: 1_400_000,
},
max_tokens: 65536,
},
@@ -0,0 +1,360 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import {
afterAll,
afterEach,
beforeAll,
beforeEach,
describe,
expect,
it,
vi,
type MockInstance,
} from 'vitest';
import { SYSTEM_ACTOR } from '../../../../core/actor.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import { PuterServer } from '../../../../server.js';
import { setupTestServer } from '../../../../testUtil.js';
import { withTestActor } from '../../../integrationTestUtil.js';
import { MINIMAX_MODELS } from './models.js';
import { MiniMaxProvider } from './MiniMaxProvider.js';
const { createMock, openAICtor } = vi.hoisted(() => {
const createMock = vi.fn();
const openAICtor = vi.fn();
return { createMock, openAICtor };
});
vi.mock('openai', () => {
const OpenAICtor = vi.fn().mockImplementation(function (
this: Record<string, unknown>,
opts: unknown,
) {
openAICtor(opts);
this.chat = { completions: { create: createMock } };
});
return { OpenAI: OpenAICtor, default: { OpenAI: OpenAICtor } };
});
let server: PuterServer;
let recordSpy: MockInstance<MeteringService['utilRecordUsageObject']>;
beforeAll(async () => {
server = await setupTestServer();
});
afterAll(async () => {
await server?.shutdown();
});
const makeProvider = () => {
const provider = new MiniMaxProvider(
{ apiKey: 'test-key' },
server.services.metering,
);
return { provider };
};
const asAsyncIterable = <T>(items: T[]): AsyncIterable<T> => ({
async *[Symbol.asyncIterator]() {
for (const item of items) {
yield item;
}
},
});
beforeEach(() => {
createMock.mockReset();
openAICtor.mockReset();
recordSpy = vi.spyOn(server.services.metering, 'utilRecordUsageObject');
});
afterEach(() => {
vi.restoreAllMocks();
});
describe('MiniMaxProvider construction', () => {
it('points the OpenAI SDK at the MiniMax base URL with the configured key', () => {
makeProvider();
expect(openAICtor).toHaveBeenCalledTimes(1);
expect(openAICtor).toHaveBeenCalledWith({
apiKey: 'test-key',
baseURL: 'https://api.minimax.io/v1',
});
});
it('allows overriding the MiniMax API base URL', () => {
new MiniMaxProvider(
{
apiKey: 'test-key',
apiBaseUrl: 'https://example.test/v1',
},
server.services.metering,
);
expect(openAICtor).toHaveBeenCalledWith({
apiKey: 'test-key',
baseURL: 'https://example.test/v1',
});
});
});
describe('MiniMaxProvider model catalog', () => {
it('returns minimax-m2.7 as the default', () => {
const { provider } = makeProvider();
expect(provider.getDefaultModel()).toBe('minimax-m2.7');
});
it('exposes the static MINIMAX_MODELS list verbatim from models()', () => {
const { provider } = makeProvider();
expect(provider.models()).toBe(MINIMAX_MODELS);
});
it('list() flattens canonical ids and aliases', () => {
const { provider } = makeProvider();
const ids = provider.list();
for (const model of MINIMAX_MODELS) {
expect(ids).toContain(model.id);
for (const alias of model.aliases ?? []) {
expect(ids).toContain(alias);
}
}
expect(ids).toContain('minimax-m2.7');
expect(ids).toContain('MiniMax-M2.7');
expect(ids).toContain('minimax/minimax-m2.7');
});
it('uses MiniMax completion-token limits instead of the context window', () => {
for (const model of MINIMAX_MODELS) {
if (model.id === 'minimax-m3') {
expect(model.context).toBe(1_048_576);
expect(model.max_tokens).toBe(512_000);
continue;
}
expect(model.context).toBe(204_800);
expect(model.max_tokens).toBe(196_608);
}
});
});
describe('MiniMaxProvider.complete request shape', () => {
const baseCompletion = {
choices: [
{
message: { content: 'hi', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 1, completion_tokens: 1 },
};
it('sends the case-sensitive upstream MiniMax model id', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'minimax-m2.7',
messages: [{ role: 'user', content: 'hello' }],
}),
);
const [args] = createMock.mock.calls[0]!;
expect(args.model).toBe('MiniMax-M2.7');
expect(args.messages).toEqual([{ role: 'user', content: 'hello' }]);
expect(args.max_tokens).toBe(1000);
});
it('resolves the upstream-cased alias to the canonical API model', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'MiniMax-M2.7-highspeed',
messages: [{ role: 'user', content: 'hello' }],
}),
);
expect(createMock.mock.calls[0]![0].model).toBe(
'MiniMax-M2.7-highspeed',
);
expect(recordSpy).toHaveBeenCalledWith(
expect.any(Object),
expect.anything(),
'minimax:minimax-m2.7-highspeed',
expect.any(Object),
);
});
it('passes standard chat completion options through', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
const tools = [
{
type: 'function',
function: {
name: 'lookup',
parameters: { type: 'object', properties: {} },
},
},
];
await withTestActor(() =>
provider.complete({
model: 'minimax-m2.7',
messages: [{ role: 'user', content: 'hi' }],
tools,
tool_choice: 'auto',
max_tokens: 256,
temperature: 0.4,
top_p: 0.9,
}),
);
const [args] = createMock.mock.calls[0]!;
expect(args.tools).toBe(tools);
expect(args.tool_choice).toBe('auto');
expect(args.max_tokens).toBe(256);
expect(args.temperature).toBe(0.4);
expect(args.top_p).toBe(0.9);
});
it('clamps oversized max_tokens to the MiniMax completion limit', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'minimax-m2.1',
messages: [{ role: 'user', content: 'hi' }],
max_tokens: 200_000,
}),
);
const [args] = createMock.mock.calls[0]!;
expect(args.model).toBe('MiniMax-M2.1');
expect(args.max_tokens).toBe(196_608);
});
it('only sets stream_options.include_usage when streaming', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'minimax-m2.7',
messages: [{ role: 'user', content: 'hi' }],
stream: false,
}),
);
const [nonStreamArgs] = createMock.mock.calls[0]!;
expect(nonStreamArgs.stream).toBe(false);
expect('stream_options' in nonStreamArgs).toBe(false);
createMock.mockReturnValueOnce(asAsyncIterable([]));
await withTestActor(() =>
provider.complete({
model: 'minimax-m2.7',
messages: [{ role: 'user', content: 'hi' }],
stream: true,
}),
);
const [streamArgs] = createMock.mock.calls[1]!;
expect(streamArgs.stream).toBe(true);
expect(streamArgs.stream_options).toEqual({ include_usage: true });
});
});
describe('MiniMaxProvider.complete output and metering', () => {
it('returns the first choice and records MiniMax usage with model costs', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'hi there', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: {
prompt_tokens: 100,
completion_tokens: 50,
prompt_tokens_details: { cached_tokens: 10 },
},
});
const result = await withTestActor(() =>
provider.complete({
model: 'minimax-m2.7',
messages: [{ role: 'user', content: 'hi' }],
}),
);
expect(result).toMatchObject({
message: { content: 'hi there', role: 'assistant' },
finish_reason: 'stop',
});
expect((result as { usage: unknown }).usage).toEqual({
prompt_tokens: 100,
completion_tokens: 50,
cached_tokens: 10,
});
expect(recordSpy).toHaveBeenCalledTimes(1);
const [usage, actor, prefix, overrides] = recordSpy.mock.calls[0]!;
expect(usage).toEqual({
prompt_tokens: 100,
completion_tokens: 50,
cached_tokens: 10,
});
expect(actor).toBe(SYSTEM_ACTOR);
expect(prefix).toBe('minimax:minimax-m2.7');
expect(overrides).toEqual({
prompt_tokens: 100 * 30,
completion_tokens: 50 * 120,
cached_tokens: 10 * 6,
});
});
it('rethrows errors raised by the OpenAI client unchanged', async () => {
const { provider } = makeProvider();
const apiError = new Error('MiniMax exploded');
createMock.mockRejectedValueOnce(apiError);
await expect(
withTestActor(() =>
provider.complete({
model: 'minimax-m2.7',
messages: [{ role: 'user', content: 'boom' }],
}),
),
).rejects.toBe(apiError);
expect(recordSpy).not.toHaveBeenCalled();
});
});
describe('MiniMaxProvider.checkModeration', () => {
it('throws because MiniMax provider does not implement moderation', () => {
const { provider } = makeProvider();
expect(() => provider.checkModeration('anything')).toThrow(
/not implemented/i,
);
});
});
@@ -0,0 +1,135 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { OpenAI } from 'openai';
import { ChatCompletionCreateParams } from 'openai/resources/index.js';
import { Context } from '../../../../core/context.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import type { IChatProvider, ICompleteArguments } from '../../types.js';
import * as OpenAIUtil from '../../utils/OpenAIUtil.js';
import { MINIMAX_MODELS } from './models.js';
type MiniMaxConfig = {
apiKey: string;
apiBaseUrl?: string;
};
export class MiniMaxProvider implements IChatProvider {
#openai: OpenAI;
#meteringService: MeteringService;
#defaultModel = 'minimax-m2.7';
constructor(config: MiniMaxConfig, meteringService: MeteringService) {
this.#openai = new OpenAI({
apiKey: config.apiKey,
baseURL: config.apiBaseUrl ?? 'https://api.minimax.io/v1',
});
this.#meteringService = meteringService;
}
getDefaultModel() {
return this.#defaultModel;
}
models() {
return MINIMAX_MODELS;
}
list() {
const modelIds: string[] = [];
for (const model of this.models()) {
modelIds.push(model.id);
if (model.aliases) {
modelIds.push(...model.aliases);
}
}
return modelIds;
}
async complete({
messages,
stream,
model,
tools,
tool_choice,
max_tokens,
temperature,
top_p,
}: ICompleteArguments): ReturnType<IChatProvider['complete']> {
const actor = Context.get('actor');
const availableModels = this.models();
const modelUsed =
availableModels.find((m) =>
[m.id, ...(m.aliases || [])].includes(model),
) || availableModels.find((m) => m.id === this.getDefaultModel())!;
messages = await OpenAIUtil.process_input_messages(messages);
const requestedMaxTokens = max_tokens ?? 1000;
const completion = await this.#openai.chat.completions.create({
messages,
model: modelUsed.apiModel,
...(tools ? { tools } : {}),
...(tool_choice !== undefined ? { tool_choice } : {}),
max_tokens: Math.min(requestedMaxTokens, modelUsed.max_tokens),
...(temperature !== undefined ? { temperature } : {}),
...(top_p !== undefined ? { top_p } : {}),
stream,
...(stream
? {
stream_options: { include_usage: true },
}
: {}),
} as ChatCompletionCreateParams);
return OpenAIUtil.handle_completion_output({
usage_calculator: ({ usage }) => {
const trackedUsage = usage
? OpenAIUtil.extractMeteredUsage(usage)
: {
prompt_tokens: 0,
completion_tokens: 0,
cached_tokens: 0,
};
const costsOverride = Object.fromEntries(
Object.entries(trackedUsage).map(([key, value]) => {
return [key, value * Number(modelUsed.costs[key] ?? 0)];
}),
);
this.#meteringService.utilRecordUsageObject(
trackedUsage,
actor!,
`minimax:${modelUsed.id}`,
costsOverride,
);
return trackedUsage;
},
stream,
completion,
});
}
checkModeration(
_text: string,
): ReturnType<IChatProvider['checkModeration']> {
throw new Error('Method not implemented.');
}
}
@@ -0,0 +1,232 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { IChatModel } from '../../types.js';
type MiniMaxChatModel = IChatModel & {
apiModel: string;
};
// Hardcoded from MiniMax OpenAI-compatible API docs and pay-as-you-go pricing:
// https://platform.minimax.io/docs/api-reference/text-openai-api
// https://platform.minimax.io/docs/guides/pricing-paygo
export const MINIMAX_MODELS: MiniMaxChatModel[] = [
// -- MiniMax M3 (Flagship) --------------------------------------
// MiniMax Sparse Attention, 1M context, native multimodal input.
{
puterId: 'minimax:minimax/minimax-m3',
id: 'minimax-m3',
apiModel: 'MiniMax-M3',
name: 'MiniMax M3',
aliases: ['minimax/minimax-m3', 'MiniMax-M3', 'minimax/MiniMax-M3'],
modalities: { input: ['text', 'image', 'video'], output: ['text'] },
open_weights: false,
tool_call: true,
context: 1_048_576,
max_tokens: 512_000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 30, // $0.30 per 1M (input <= 512k tokens)
completion_tokens: 120, // $1.20 per 1M
cached_tokens: 6, // $0.06 per 1M
},
},
// -- MiniMax M2.7 -----------------------------------------------
{
puterId: 'minimax:minimax/minimax-m2.7',
id: 'minimax-m2.7',
apiModel: 'MiniMax-M2.7',
name: 'MiniMax M2.7',
aliases: [
'minimax/minimax-m2.7',
'MiniMax-M2.7',
'minimax/MiniMax-M2.7',
],
modalities: { input: ['text'], output: ['text'] },
open_weights: false,
tool_call: true,
context: 204_800,
max_tokens: 196_608,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 30, // $0.30 per 1M
completion_tokens: 120, // $1.20 per 1M
cached_tokens: 6, // $0.06 per 1M
},
},
{
puterId: 'minimax:minimax/minimax-m2.7-highspeed',
id: 'minimax-m2.7-highspeed',
apiModel: 'MiniMax-M2.7-highspeed',
name: 'MiniMax M2.7 Highspeed',
aliases: [
'minimax/minimax-m2.7-highspeed',
'MiniMax-M2.7-highspeed',
'minimax/MiniMax-M2.7-highspeed',
],
modalities: { input: ['text'], output: ['text'] },
open_weights: false,
tool_call: true,
context: 204_800,
max_tokens: 196_608,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 60, // $0.60 per 1M
completion_tokens: 240, // $2.40 per 1M
cached_tokens: 6, // $0.06 per 1M
},
},
// -- MiniMax M2.5 -----------------------------------------------
{
puterId: 'minimax:minimax/minimax-m2.5',
id: 'minimax-m2.5',
apiModel: 'MiniMax-M2.5',
name: 'MiniMax M2.5',
aliases: [
'minimax/minimax-m2.5',
'MiniMax-M2.5',
'minimax/MiniMax-M2.5',
],
modalities: { input: ['text'], output: ['text'] },
open_weights: false,
tool_call: true,
context: 204_800,
max_tokens: 196_608,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 30, // $0.30 per 1M
completion_tokens: 120, // $1.20 per 1M
cached_tokens: 3, // $0.03 per 1M
},
},
{
puterId: 'minimax:minimax/minimax-m2.5-highspeed',
id: 'minimax-m2.5-highspeed',
apiModel: 'MiniMax-M2.5-highspeed',
name: 'MiniMax M2.5 Highspeed',
aliases: [
'minimax/minimax-m2.5-highspeed',
'MiniMax-M2.5-highspeed',
'minimax/MiniMax-M2.5-highspeed',
],
modalities: { input: ['text'], output: ['text'] },
open_weights: false,
tool_call: true,
context: 204_800,
max_tokens: 196_608,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 60, // $0.60 per 1M
completion_tokens: 240, // $2.40 per 1M
cached_tokens: 3, // $0.03 per 1M
},
},
// -- MiniMax M2.1 -----------------------------------------------
{
puterId: 'minimax:minimax/minimax-m2.1',
id: 'minimax-m2.1',
apiModel: 'MiniMax-M2.1',
name: 'MiniMax M2.1',
aliases: [
'minimax/minimax-m2.1',
'MiniMax-M2.1',
'minimax/MiniMax-M2.1',
],
modalities: { input: ['text'], output: ['text'] },
open_weights: false,
tool_call: true,
context: 204_800,
max_tokens: 196_608,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 30, // $0.30 per 1M
completion_tokens: 120, // $1.20 per 1M
cached_tokens: 3, // $0.03 per 1M
},
},
{
puterId: 'minimax:minimax/minimax-m2.1-highspeed',
id: 'minimax-m2.1-highspeed',
apiModel: 'MiniMax-M2.1-highspeed',
name: 'MiniMax M2.1 Highspeed',
aliases: [
'minimax/minimax-m2.1-highspeed',
'MiniMax-M2.1-highspeed',
'minimax/MiniMax-M2.1-highspeed',
],
modalities: { input: ['text'], output: ['text'] },
open_weights: false,
tool_call: true,
context: 204_800,
max_tokens: 196_608,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 60, // $0.60 per 1M
completion_tokens: 240, // $2.40 per 1M
cached_tokens: 3, // $0.03 per 1M
},
},
// -- MiniMax M2 -------------------------------------------------
{
puterId: 'minimax:minimax/minimax-m2',
id: 'minimax-m2',
apiModel: 'MiniMax-M2',
name: 'MiniMax M2',
aliases: ['minimax/minimax-m2', 'MiniMax-M2', 'minimax/MiniMax-M2'],
modalities: { input: ['text'], output: ['text'] },
open_weights: false,
tool_call: true,
context: 204_800,
max_tokens: 196_608,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 30, // $0.30 per 1M
completion_tokens: 120, // $1.20 per 1M
cached_tokens: 3, // $0.03 per 1M
},
},
];
@@ -655,6 +655,123 @@ describe('MistralAIProvider.complete streaming', () => {
});
});
// -- Mistral image_url coercion --
describe('MistralAIProvider image_url coercion', () => {
const baseCompletion = {
choices: [
{
message: { content: 'ok', role: 'assistant' },
finishReason: 'stop',
},
],
usage: { promptTokens: 1, completionTokens: 1 },
};
it('flattens image_url objects to plain strings before sending to Mistral', async () => {
const { provider } = makeProvider();
completeMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'mistral-small-2603',
messages: [
{
role: 'user',
content: [
{ type: 'text', text: 'describe this image' },
{
type: 'image_url',
image_url: { url: 'https://example.com/img.png' },
},
],
},
],
}),
);
const [args] = completeMock.mock.calls[0]!;
const imagePart = (args.messages[0].content as unknown[]).find(
(p: unknown) => (p as { type: string }).type === 'image_url',
) as { type: string; image_url: unknown };
// Mistral expects a plain string, not { url: string }.
expect(imagePart.image_url).toBe('https://example.com/img.png');
});
it('leaves image_url parts that are already plain strings unchanged', async () => {
const { provider } = makeProvider();
completeMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'mistral-small-2603',
messages: [
{
role: 'user',
content: [
{
type: 'image_url',
image_url: 'https://example.com/already-flat.png',
},
],
},
],
}),
);
const [args] = completeMock.mock.calls[0]!;
const imagePart = (args.messages[0].content as unknown[]).find(
(p: unknown) => (p as { type: string }).type === 'image_url',
) as { type: string; image_url: unknown };
expect(imagePart.image_url).toBe('https://example.com/already-flat.png');
});
it('leaves messages with plain string content untouched', async () => {
const { provider } = makeProvider();
completeMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'mistral-small-2603',
messages: [{ role: 'user', content: 'plain text message' }],
}),
);
const [args] = completeMock.mock.calls[0]!;
expect(args.messages[0].content).toBe('plain text message');
});
it('coerces only image_url parts and leaves other content parts intact', async () => {
const { provider } = makeProvider();
completeMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'mistral-small-2603',
messages: [
{
role: 'user',
content: [
{ type: 'text', text: 'what is in this image?' },
{
type: 'image_url',
image_url: { url: 'https://example.com/photo.jpg' },
},
],
},
],
}),
);
const [args] = completeMock.mock.calls[0]!;
const parts = args.messages[0].content as { type: string; text?: string; image_url?: unknown }[];
const textPart = parts.find((p) => p.type === 'text');
const imgPart = parts.find((p) => p.type === 'image_url');
expect(textPart?.text).toBe('what is in this image?');
expect(imgPart?.image_url).toBe('https://example.com/photo.jpg');
});
});
// ── Error mapping ───────────────────────────────────────────────────
describe('MistralAIProvider.complete error mapping', () => {
Loaded 100 of 324 files, more files were not shown because too many files have changed in this diff. Show more