Compare commits

..
Author SHA1 Message Date
Nariman Jelveh 734715f461 Merge branch 'feat/app-user-feedback' of https://github.com/HeyPuter/puter into feat/app-user-feedback 2026-08-12 13:32:34 -07:00
jelveh b290c1d922 Merge remote-tracking branch 'origin/main' into feat/app-user-feedback 2026-08-12 09:22:00 -07:00
jelveh dd068e31d0 Require verified users for app feedback API
Add `requireVerified: true` to both app feedback routes (`GET /target` and `POST /`) in `AppFeedbackController`. This tightens access control so only verified user accounts can fetch feedback targets or submit app feedback.
2026-08-12 09:13:11 -07:00
jelveh 90cffbfcde refactor: prep the feedback cap error once instead of via a factory
Both throw sites are in one call and only one can ever run, so a plain
const reads the same and drops a function that existed only to defer a
constructor.
2026-08-11 22:11:24 -07:00
jelveh 25acf14c8b test: cover the feedback service and store layers directly
The service owns every feedback business rule — target resolution,
eligibility, message normalization, the durable caps, and the owner-email
preconditions — but was only reachable through the controller's tests. Give
it and the store their own suites so a regression names the layer it broke.

Service coverage adds the branches the route tests could not reach: a
blocked origin resolving to null rather than surfacing a 403, owners who are
suspended or unsubscribed, length measured after normalization, the 24h cap
window boundary, subject-header injection via the app title, and the email
links being rooted at config.origin.

The two describes already labelled `AppFeedbackService ...` move out of the
controller test, which keeps only the caller-facing promise that a failed
send still returns success.
2026-08-11 22:10:04 -07:00
Nariman Jelveh be781c5f1b fix: restore the feedback Send button's gradient on hover
1c2d33f14 added :not(:disabled) to the base .app-feedback-btn hover
rule, which raised its specificity above the primary button's hover rule
— so hovering Send swapped the blue gradient for the light hover
background, leaving white text on a pale pill. Give the primary hover
rule the same :not(:disabled) so it wins by source order again.
2026-08-11 19:57:56 -07:00
Nariman Jelveh 4cb0730875 feat: point the feedback email footer at the Dev Center
The footer told developers to turn feedback off via a puter.apps.update
one-liner, but the toggle lives in the Dev Center app settings — and
Dev Center is where apps get feedback enabled by default in the first
place. Reword it to 'manage it in the Dev Center' with a link built
from config.origin (like app_link) so it holds on self-hosted
deployments.
2026-08-11 19:19:38 -07:00
Nariman Jelveh 00b2fcdea3 docs: document the COOP false-resolve limitation of showFeedbackDialog
Under COOP the popup's opener link is severed, so the SDK deliberately
resolves false while the popup stays open and the user can still submit
(a feedback submission has no server read-back the way a permission
grant does). The documented contract ('resolves to true if the user
submitted feedback') was silently wrong on cross-origin-isolated pages —
state the limitation in the doc and the SDK jsdoc: false means 'not
confirmed', not 'not sent'.
2026-08-11 18:31:39 -07:00
Nariman Jelveh f70347b579 docs: state exactly what showFeedbackDialog shares with the developer
The doc claimed the dialog 'tells the user their username will be
shared', while the dialog's note talked only about the email address and
the implementation shares the username always plus the email (as
Reply-To) only when verified. Describe the actual disclosure: username
always, email when verified.
2026-08-11 18:31:02 -07:00
Nariman Jelveh 3d9691b11a fix: stop HTML-escaping email subject lines
Subjects were compiled with default Handlebars escaping, so the
app-user-feedback subject rendered a title like "Bob's App & Games" as
"Bob's App & Games" — literal entities in the recipient's mail
client. Subjects are plain-text headers, not HTML; compile them with
noEscape. Header safety is unaffected: the transport encodes newlines
and free-form values collapse whitespace upstream.
2026-08-11 18:30:29 -07:00
Nariman Jelveh c2fe813929 fix: make the feedback dialog's privacy note match what is shared
The note unconditionally said 'Your email address will be shared with
the developer so they can respond', but AppFeedbackService shares the
username always and the email only when it exists and is verified — an
unverified or temp-user sender was promised a reply path that never
materializes, and nobody was told about the username.

Show 'username and email' when the signed-in user's email is verified,
and 'username' otherwise.
2026-08-11 18:29:22 -07:00
Nariman Jelveh ef7fc05dcc fix: reject feedback origins longer than the source_origin column
readTargetParam accepted values up to 3000 chars but the raw origin is
stored verbatim into source_origin VARCHAR(2048) (MySQL/Postgres), so a
2049-3000 char origin passed every validation and then blew up the
INSERT with an HTTP 500 on Postgres/strict MySQL — or was silently
truncated on non-strict MySQL, corrupting the abuse-forensics value the
column exists for. Cap the param at the column size.
2026-08-11 18:27:42 -07:00
Nariman Jelveh 1c2d33f149 fix: gate feedback dialog Cancel/X on an in-flight submit
Escape and backdrop clicks were already ignored while the POST was
pending, but the X, Close, and Cancel buttons weren't — clicking one
mid-send settled the promise false and tore down the overlay while the
submission still landed server-side: the developer got the email, the
app was told sent=false, and a user who resubmitted 'the failed one'
sent a duplicate and burned a daily-cap slot.

Apply the same !sending gate to the buttons and disable them visually
while the send is in flight.
2026-08-11 18:26:47 -07:00
Nariman Jelveh b1fdc8d665 fix: carry feedback_enabled in suggested/recommended app summaries
launch_app uses options.app_obj verbatim when provided, and the
suggested-apps launch paths (open_item.js, UIWindowSearch.js,
UIDesktop.js) pass summaries from toAppSummary — which omitted
feedback_enabled. So an opted-in editor launched by opening a .txt file
showed no Send Feedback control in the dashboard drawer, while the same
app launched from the Apps tab (full puter.apps.get object) did.
2026-08-11 18:23:43 -07:00
Nariman Jelveh 1ecc23cf83 fix: guard app-triggered feedback dialog against desktop-lockout loops
The showFeedbackDialog IPC handler had no re-entry or abuse guard, and
the dialog it opens is a full-viewport overlay above the taskbar and
every window — so 'while (true) await puter.ui.showFeedbackDialog()'
kept the desktop permanently covered (for signed-out users, the same
loop spams the full-page signup window instead). Every dismissal just
settled the promise and let the app immediately reopen it.

Allow one dialog at a time, and back off reopens per app after each
dismissal that sent nothing: 10s, then 60s, then blocked until page
reload. A successful send resets the backoff, and user-initiated paths
(dashboard drawer) are unaffected since they don't go through IPC.
2026-08-11 18:21:43 -07:00
Nariman Jelveh 7230580b18 fix: settle showFeedbackDialog instead of hanging on older host GUIs
In the app environment showFeedbackDialog awaited an IPC reply with no
capability check. A host GUI that predates this feature (self-hosted
Puter running the live js.puter.com SDK) has no handler for the message
and never replies, so the promise documented as 'never rejects' also
never resolved.

The GUI now advertises the IPC dialogs it can answer via a
puter.gui_features param on the app iframe URL, and the SDK resolves
false when 'feedback-dialog' isn't listed. A reply timeout could not
substitute: legitimate replies only arrive when the user closes the
dialog, so any timeout would false-negative while the user is typing.
Older SDKs ignore the extra param.
2026-08-11 18:19:46 -07:00
Nariman Jelveh 1460751b41 fix: refuse feedback when the deployment cannot deliver it
With no email transport configured (the common self-hosted default),
submissions were stored in app_feedback — a table with no read path
beyond the abuse-cap COUNTs — the owner email was silently skipped, and
the sender was still shown 'Feedback sent. Thank you!'. The developer
never learns the feedback exists while the user believes it was
delivered.

Gate acceptsFeedback on clients.email.isConfigured so the pre-flight
reports enabled:false (the dialog shows its 'not accepting feedback'
pane) and submit returns 403 instead of swallowing messages. Owner-level
store-without-email cases (unconfirmed owner email, per-app email cap
overflow) keep their existing deliberate semantics.
2026-08-11 18:16:14 -07:00
Nariman Jelveh 4f96d371f4 fix: don't mint a user-app token as a side effect of the feedback popup
Every embedded_in_popup boot ran the user-app token exchange, and the
exchange is a write: /auth/get-user-app-token bootstraps an app row for
the opener origin, grants flag:app-is-authenticated (what makes the
site count as connected to the account), and creates its AppData dir.
So merely opening — or immediately cancelling — a send-feedback popup
recorded a user<->site relationship the read-only feedback flow never
needs: the server resolves the feedback target from the attested origin
without any of it.

Gate the exchange behind runsUserAppTokenExchange(action) in all three
popup paths that mint (main postAuthActions exchange, temp-user signup
success, manual signup fallback). request-permission keeps the exchange
since grants are written against the app row it bootstraps.
2026-08-11 18:13:13 -07:00
Nariman Jelveh c5cf1dc390 docs: disclose the Dev Center's feedback-on-by-default in SDK docs/types
The Dev Center deliberately creates apps with feedbackEnabled (see
32c950de5), but the SDK docs, apps.d.ts, and AppFeedbackService's class
doc all described feedback as strictly opt-in / default-false with no
qualification — so a Dev Center developer reading them would wrongly
conclude feedback is off for their app. State the Dev Center behavior
alongside the API default, and correct the update-path docs: an omitted
feedbackEnabled leaves the current value unchanged rather than
defaulting to false.
2026-08-11 18:07:47 -07:00
Nariman Jelveh 503a450b40 fix: make feedback daily caps fail closed under concurrent submissions
The per-user and per-user-per-app caps were check-then-insert and the
per-app email cap was count-then-send, so parallel requests (or multiple
nodes, or the route limiter failing open) could all read a stale
under-cap count and push past every limit — the exact scenario the
DB-backed caps exist to stop.

Now the user caps recount after the insert (own row included) and roll
the row back with 429 if a burst breached them, and the email cap claims
its slot (email_sent=1) before sending, recounts, and releases the slot
if over cap or if the send fails.
2026-08-11 18:06:05 -07:00
Nariman Jelveh f4dfbd735b fix: resolve 'app-'-prefixed app names in feedback target lookup
APP_NAME_REGEX allows names beginning with "app-" (e.g. the seeded
app-center), but resolveTargetApp's startsWith('app-') heuristic sent
those to a uid-only lookup with no name fallback, so feedback for such
apps 403'd even when enabled. Use AppStore.resolveApp (uid, then name)
like the rest of the codebase.
2026-08-11 18:03:15 -07:00
Nariman Jelveh 81402918d2 fix: feedback modal polish + share sender email
Address four issues with the app feedback UI:

- Dashboard app-drawer: the extra "feedback" control pushed the close
  button past the drawer's derived width and clipped it. A `has-feedback`
  modifier widens the surface by one button + gap so all three controls
  fit. The control's glyph is now a message bubble with text lines, which
  reads more clearly at 14px than the previous bare speech bubble.

- The feedback dialog is no longer a UIWindow. It's a from-scratch
  overlay modal in the spirit of the dashboard modals (uninstall,
  add-app): a fixed scrim + centered card with self-contained,
  theme-aware color tokens (light default + dark override), a bottom-sheet
  layout on narrow screens, backdrop/Escape close, and an entrance
  transition. This renders consistently across the three contexts it's
  opened from (desktop app-IPC, dashboard drawer, standalone popup), so
  the callers no longer pass UIWindow-specific window_options.

- Feedback now shares the sender's email (not just their username) with
  the developer so they can respond: the owner email sets Reply-To to the
  sender and shows the address in the body — but only when the sender's
  email is verified (an unverified address could be anyone's, so it's
  never used as a reply target). EmailClient.send gains an optional
  replyTo. The dialog note now says the email will be shared.

Tests: e2e updated for the new modal (7 pass); backend feedback suite
covers the verified/unverified sender-email split (sqlite + postgres);
EmailClient + GUI unit suites pass; type-check clean.
2026-08-11 17:11:36 -07:00
Nariman Jelveh 32c950de5b feat: enable app feedback by default in Dev Center
New apps created in Dev Center now have feedbackEnabled set on creation,
so users can send the developer feedback without any extra setup. A "User
Feedback" toggle in the app's settings lets developers turn it off (and
back on); it's wired into the save payload, the dirty-state tracking, and
the reset-to-original path like the neighboring toggles.

The Save update omits feedbackEnabled unless the toggle is present, and
the backend leaves an omitted field untouched, so the default survives
the create-then-save flow Dev Center runs. Add an SDK apps-suite guard
covering that round-trip (create-on -> unrelated update keeps it -> can
be turned off).
2026-08-11 16:47:12 -07:00
Nariman Jelveh 792361507e feat: add feedback control to the dashboard app-drawer
Surface the feedback dialog directly from the app window's chrome in
dashboard mode: apps that opt in (apps.feedbackEnabled) get a "Send
Feedback" button in the dashboard app-drawer, next to minimize/close.
It opens the same UIWindowAppFeedback dialog, targeting this app by uid.

The control is only rendered when the app opted in — feedback_enabled is
threaded from the launched app's metadata into the window options — and
the dialog still re-checks opt-in server-side, so a stale flag can't send
anywhere. Reuses the existing .dashboard-app-drawer-btn styling and the
app_feedback_title i18n string, so no new CSS or strings.

Adds e2e coverage: the control appears and opens the dialog for an
opted-in app, and is absent for an app that hasn't opted in.
2026-08-11 16:39:05 -07:00
Nariman Jelveh 4f267fb83e feat: app user feedback system
Add puter.ui.showFeedbackDialog(), letting users send feedback to an
app's developer. In the app environment the Puter desktop renders the
dialog; on a third-party website a puter.com popup hosts it. The message
is stored in a new app_feedback table and emailed to the app owner's
confirmed email — it never passes through the app's own code.

Feedback is strictly opt-in per app via a new apps.feedback_enabled
column (a real column, not an app-metadata key, so Dev Center's
whole-blob metadata saves can't silently erase it), settable through the
existing puter.apps.update path (feedbackEnabled).

Backend follows the layered stack: AppFeedbackStore (durable count
queries) -> AppFeedbackService (opt-in check, message normalization,
abuse caps, best-effort owner email) -> AppFeedbackController
(POST /app-feedback, GET /app-feedback/target). New app-user-feedback
email template uses the escaping-safe nl2br triple-stash.

Defensive by design:
- requireUserActor blocks app tokens, so feedback can't be submitted
  programmatically; guiOriginOnly keeps cross-origin pages out.
- App identity comes only from the validated IPC sender (desktop) or the
  browser-attested opener origin (popup), never from message contents.
- The send-feedback popup action is in NON_AUTH_POPUP_ACTIONS, so it
  never delivers a token to the opener.
- Layered limits: route rate limits, plus DB-count caps that fail closed
  when the limiter backend is down, plus a per-app daily owner-email cap.
- Owner email is fully best-effort: an unconfigured transport,
  unconfirmed/unsubscribed/suspended owner, or send failure never fails
  the request or blocks storage.
- The dialog and SDK method are resolve-only and always settle, so a
  caller is never left hanging.

Migrations for sqlite/mysql/postgres, puter.js types, docs, backend
tests (sqlite + postgres), and a Playwright e2e spec are included.
2026-08-11 16:31:27 -07:00
2396 changed files with 73378 additions and 526912 deletions

No files matched your search

-5
View File
@@ -16,8 +16,3 @@ MARIADB_PASSWORD=replace-with-strong-password
S3_ACCESS_KEY=puter
S3_SECRET_KEY=replace-with-strong-secret
S3_BUCKET=puter-local
# ── Inbound SMTP (compose profile `smtp`) ------------------------------
# Host port mapped to the receiver's 2525. 25 is what an MX record means;
# many hosting providers block it.
SMTP_PORT=25
-72
View File
@@ -1,72 +0,0 @@
# Default owner, including backend and SDK areas without a specialist below.
# The last matching rule takes precedence.
- @Salazareo
# GUI
/src/gui/ @jelveh @Salazareo
# Documentation, developer site, and CLI
/doc/ @reynaldichernando @Salazareo
/src/docs/ @reynaldichernando @Salazareo
/src/dev-center/ @reynaldichernando @Salazareo
/src/cli/ @reynaldichernando @Salazareo
# Backend AI
/src/backend/controllers/puterai/ @404oops @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/ai-\*/ @404oops @ProgrammerIn-wonderland @Salazareo
# Backend: AI, hosting, workers, networking, and WebDAV
/src/backend/controllers/hosting/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/webdav/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/wisp/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/util/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/workers/ @ProgrammerIn-wonderland @Salazareo
/src/backend/services/localworker/ @ProgrammerIn-wonderland @Salazareo
/src/backend/stores/subdomain/ @ProgrammerIn-wonderland @Salazareo
/extensions/workerSandbox\* @ProgrammerIn-wonderland @Salazareo
/src/worker/ @ProgrammerIn-wonderland @Salazareo
/src/worker-types/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/peer/ @ProgrammerIn-wonderland @Salazareo
# Backend: sharing, permissions, and teams
/src/backend/controllers/share/ @jfcastro92 @Salazareo
/src/backend/controllers/team/ @jfcastro92 @Salazareo
/src/backend/services/share/ @jfcastro92 @Salazareo
/src/backend/services/permission/ @jfcastro92 @Salazareo
/src/backend/services/acl/ @jfcastro92 @Salazareo
/src/backend/services/team/ @jfcastro92 @Salazareo
/src/backend/services/apps/AppPermissionService\* @jfcastro92 @Salazareo
/src/backend/stores/share/ @jfcastro92 @Salazareo
/src/backend/stores/permission/ @jfcastro92 @Salazareo
/src/backend/stores/group/ @jfcastro92 @Salazareo
/src/backend/stores/team/ @jfcastro92 @Salazareo
# SDK: AI, hosting, workers, and networking
/src/puter-js/src/modules/ai/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/hosting/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Workers* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/networking/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/lib/networkUtils* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/ai.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/hosting.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/workers.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/net.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Peer\* @ProgrammerIn-wonderland @Salazareo
# SDK: sharing and permissions
/src/puter-js/src/modules/perms/ @jfcastro92
/src/puter-js/src/modules/FileSystem/operations/share* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/unshare* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/getShares* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/listShared* @jfcastro92 @Salazareo
/src/puter-js/tests/api/suites/perms.suite.ts @jfcastro92 @Salazareo
/src/puter-js/tests/api/suites/sharing.suite.ts @jfcastro92 @Salazareo
-20
View File
@@ -1,20 +0,0 @@
version: 2
updates:
- package-ecosystem: "npm"
directory: "/"
schedule:
interval: "weekly"
open-pull-requests-limit: 10
ignore:
- dependency-name: "*"
update-types: ["version-update:semver-major", "version-update:semver-minor"]
- package-ecosystem: "docker"
directory: "/"
schedule:
interval: "weekly"
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
-106
View File
@@ -1,106 +0,0 @@
# This file is machine-generated by `gh actions-lock`.
# Do not edit by hand; run `gh actions-lock` to update.
# Docs: https://gh.io/actions-lockfile
version: 'v0.0.3'
workflows:
'.github/workflows/ai-provider-integration-tests.yaml':
- 'actions/[email protected]'
- 'actions/[email protected]'
'.github/workflows/backend-tests.yaml':
- 'actions/[email protected]'
- 'actions/[email protected]'
- 'actions/[email protected]'
- 'actions/[email protected]'
- 'davelosert/[email protected]'
- 'dorny/[email protected]'
'.github/workflows/dependabot-automerge.yml':
- 'dependabot/[email protected]'
'.github/workflows/docker-image.yaml':
- 'actions/[email protected]'
- 'docker/[email protected]'
- 'docker/[email protected]'
- 'docker/[email protected]'
- 'docker/[email protected]'
- 'docker/[email protected]'
- 'peter-evans/[email protected]'
'.github/workflows/docs-build.yml':
- 'actions/[email protected]'
- 'actions/[email protected]'
'.github/workflows/notify-dev.yaml': []
'.github/workflows/notify-prod.yaml': []
'.github/workflows/puterjs-tests.yaml':
- 'actions/[email protected]'
- 'actions/[email protected]'
- 'actions/[email protected]'
- 'actions/[email protected]'
- 'davelosert/[email protected]'
- 'dorny/[email protected]'
'.github/workflows/tag-alias.yaml':
- 'docker/[email protected]'
- 'docker/[email protected]'
dependencies:
'actions/[email protected]':
ref: 'v7.0.1'
commit: 'sha1-3d3c42e5aac5ba805825da76410c181273ba90b1'
owner_id: 44036562
repo_id: 197814629
'actions/[email protected]':
ref: 'v8.0.2'
commit: 'sha1-9000827ccba6bdab643e8b6fd33ac0654aef8333'
owner_id: 44036562
repo_id: 192626254
'actions/[email protected]':
ref: 'v7.1.0'
commit: 'sha1-949feb2413d6458794dcd2491c4babbbce0c15c1'
owner_id: 44036562
repo_id: 189476904
'actions/[email protected]':
ref: 'v7.0.2'
commit: 'sha1-cf430e030ddbb5b0abf93d22962f4752f3646cd9'
owner_id: 44036562
repo_id: 192625955
'davelosert/[email protected]':
ref: 'v2.13.2'
commit: 'sha1-74c2d9210b4fc44c84d8213f61d7ec99837b7aa0'
owner_id: 4287128
repo_id: 501176456
'dependabot/[email protected]':
ref: 'v3.1.0'
commit: 'sha1-25dd0e34f4fe68f24cc83900b1fe3fe149efef98'
owner_id: 27347476
repo_id: 371068214
'docker/[email protected]':
ref: 'v7.4.0'
commit: 'sha1-c3c9e263c25d99ce0380d002d59b67737d91b0dc'
owner_id: 5429470
repo_id: 241092383
'docker/[email protected]':
ref: 'v4.8.0'
commit: 'sha1-8aa5831fb8d48d8f76594c7ddde61ae58019f550'
owner_id: 5429470
repo_id: 287743349
'docker/[email protected]':
ref: 'v6.2.0'
commit: 'sha1-dc802804100637a589fabce1cb79ff13a1411302'
owner_id: 5429470
repo_id: 306769011
'docker/[email protected]':
ref: 'v4.4.1'
commit: 'sha1-f87e5991a6d7451dcb8d9637bfbc97413f497069'
owner_id: 5429470
repo_id: 288485773
'docker/[email protected]':
ref: 'v4.4.0'
commit: 'sha1-99012661954931238ded8c8b007157a8430204e1'
owner_id: 5429470
repo_id: 288487894
'dorny/[email protected]':
ref: 'v4.0.3'
commit: 'sha1-ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d'
owner_id: 1588572
repo_id: 265601040
'peter-evans/[email protected]':
ref: 'v5.0.0'
commit: 'sha1-1b9a80c056b620d92cedb9d9b5a223409c68ddfa'
owner_id: 18365890
repo_id: 193685475
@@ -17,14 +17,12 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v7.1.0
uses: actions/setup-node@v4
with:
node-version: '26'
node-version: '24'
cache: 'npm'
- name: Install dependencies
@@ -49,13 +47,3 @@ jobs:
PUTER_TEST_AI_MOONSHOT_API_KEY: ${{ secrets.PUTER_TEST_AI_MOONSHOT_API_KEY }}
PUTER_TEST_AI_ZAI_API_KEY: ${{ secrets.PUTER_TEST_AI_ZAI_API_KEY }}
PUTER_TEST_AI_ELEVENLABS_API_KEY: ${{ secrets.PUTER_TEST_AI_ELEVENLABS_API_KEY }}
# Realtime conversations through the SDK against live OpenAI. Builds
# the SDK bundle these tests load itself, so it runs even when the
# step above fails; the tests skip themselves without the key.
- name: Run live realtime conversation tests
if: ${{ !cancelled() }}
run: npm run setupExtensions && (cd src/puter-js && npm run build) && npx vitest run --config src/puter-js/tests/api/vitest.config.ts src/puter-js/tests/api/realtime/realtimeLive.test.ts
env:
CI: 'true'
PUTER_TEST_AI_OPENAI_API_KEY: ${{ secrets.PUTER_TEST_AI_OPENAI_API_KEY }}
+58 -155
View File
@@ -1,45 +1,28 @@
name: Backend Tests
# Always triggers on PRs so it can be a required check; the `changes` job below
# decides whether backend/extension paths actually changed (test-only changes
# match the same globs, since tests are co-located with the code), and `test`
# still reports success when they didn't. Pushes to main produce the coverage
# PRs compare against. puter.js SDK changes are covered by puterjs-tests.yaml.
# Runs on backend and extension changes. Tests are co-located with the
# code (src/backend/**/*.test.ts, extensions/**/*.test.ts), so test-only
# changes match the same globs. puter.js SDK changes are covered by
# puterjs-tests.yaml instead.
on:
pull_request:
types: [opened, synchronize, reopened]
push:
branches: [main]
paths:
- 'src/backend/**'
- 'extensions/**'
- 'tools/**'
- 'package.json'
- 'package-lock.json'
# The type-check job below reads these.
- 'tsconfig.json'
- 'tsconfig.build.json'
- '.github/workflows/backend-tests.yaml'
permissions:
contents: read
pull-requests: write
# Reads main's coverage artifact for the PR comparison.
actions: read
jobs:
changes:
runs-on: ubuntu-latest
outputs:
backend: ${{ steps.filter.outputs.backend }}
steps:
- uses: actions/[email protected]
with:
persist-credentials: false
- uses: dorny/[email protected]
id: filter
with:
filters: |
backend:
- 'src/backend/**'
- 'extensions/**'
- 'tools/**'
- 'package.json'
- 'package-lock.json'
- 'tsconfig.json'
- 'tsconfig.build.json'
- '.github/workflows/backend-tests.yaml'
# `tsconfig.build.json` builds with `noCheck: true`, so nothing else in CI
# runs the type checker and a missing export compiles to `undefined`,
# surfacing only when the call is finally reached at runtime. This diffs
@@ -48,20 +31,19 @@ jobs:
# The consuming repo runs an equivalent gate, but only once someone bumps the
# submodule pointer — too late to keep the error off this repo's default
# branch. Hence a gate here, on this repo's own pull requests.
#
# Its own job rather than a step in `test`: that one runs a base/PR matrix for
# coverage comparison, and the check only needs the PR ref, once.
typecheck:
needs: changes
if: github.event_name == 'pull_request' && needs.changes.outputs.backend == 'true'
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v7.1.0
uses: actions/setup-node@v4
with:
node-version: '26'
node-version: '24'
cache: 'npm'
- name: Install dependencies
@@ -70,148 +52,69 @@ jobs:
- name: Type check (new errors only)
run: npm run typecheck
# The suite is CPU-bound, so it's split across runners. Each shard writes a
# blob report (results plus coverage) that the `coverage` job merges.
test-shard:
needs: changes
if: needs.changes.outputs.backend == 'true'
name: test shard ${{ matrix.shard }}/${{ strategy.job-total }}
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4]
include:
- ref: ${{ github.base_ref }}
artifact: base
- ref: ${{ github.head_ref }}
artifact: pr
steps:
- name: Checkout
uses: actions/checkout@v7.0.1
- name: Checkout ${{ matrix.ref }}
uses: actions/checkout@v4
with:
persist-credentials: false
ref: ${{ github.event.pull_request.head.sha || github.sha }}
repository: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
ref: ${{ matrix.ref }}
repository: ${{ github.event.pull_request.head.repo.full_name }}
- name: Set up Node.js
uses: actions/setup-node@v7.1.0
uses: actions/setup-node@v4
with:
node-version: '26'
node-version: '24'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Run backend tests with coverage
run: >-
npm run test:backend --
--coverage
--shard=${{ matrix.shard }}/${{ strategy.job-total }}
--reporter=default
--reporter=blob
env:
CI: 'true'
- name: Upload blob report
if: ${{ !cancelled() }}
uses: actions/[email protected]
with:
name: backend-blob-${{ matrix.shard }}
path: .vitest-reports/
include-hidden-files: true
retention-days: 1
# Required check, so its PR name is pinned. A skipped job would satisfy it,
# so this always runs and fails on any shard that didn't pass.
test:
needs: [changes, test-shard]
if: ${{ !cancelled() }}
name: test (${{ github.event_name == 'push' && 'main' || 'pr' }})
runs-on: ubuntu-latest
steps:
- name: Check shard results
env:
CHANGES_RESULT: ${{ needs.changes.result }}
BACKEND_CHANGED: ${{ needs.changes.outputs.backend }}
SHARDS_RESULT: ${{ needs.test-shard.result }}
run: |
if [ "$CHANGES_RESULT" != "success" ]; then
echo "changes job: $CHANGES_RESULT"
exit 1
fi
if [ "$BACKEND_CHANGED" != "true" ]; then
echo "No backend changes."
exit 0
fi
if [ "$SHARDS_RESULT" != "success" ]; then
echo "test shards: $SHARDS_RESULT"
exit 1
fi
coverage:
needs: [changes, test-shard]
if: ${{ !cancelled() && needs.test-shard.result == 'success' }}
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/[email protected]
with:
persist-credentials: false
ref: ${{ github.event.pull_request.head.sha || github.sha }}
repository: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
- name: Set up Node.js
uses: actions/[email protected]
with:
node-version: '26'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Download blob reports
uses: actions/[email protected]
with:
pattern: backend-blob-*
path: .vitest-reports
merge-multiple: true
- name: Merge shard coverage
run: npx vitest run --config src/backend/vitest.config.ts --merge-reports --coverage
run: npm run test:backend -- --coverage
env:
CI: 'true'
- name: Upload coverage artifact
uses: actions/[email protected]
if: always()
uses: actions/upload-artifact@v4
with:
name: backend-coverage-${{ github.event_name == 'push' && 'main' || 'pr' }}
name: backend-coverage-${{ matrix.artifact }}
path: src/backend/coverage/
retention-days: ${{ github.event_name == 'push' && 30 || 14 }}
retention-days: 14
# Latest main coverage from this repo's own runs; a fork can't publish
# under this repository id.
- name: Download main coverage
id: base
if: github.event_name == 'pull_request'
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
run: |
id=$(gh api "repos/$REPO/actions/artifacts?name=backend-coverage-main&per_page=20" --jq '
[.artifacts[]
| select(.expired | not)
| select(.workflow_run.head_branch == "main")
| select(.workflow_run.head_repository_id == .workflow_run.repository_id)
][0].id // empty')
if [ -z "$id" ]; then
echo "No main coverage artifact; reporting without a comparison."
exit 0
fi
gh api "repos/$REPO/actions/artifacts/$id/zip" > coverage-base.zip
unzip -q coverage-base.zip -d coverage-base
echo "found=true" >> "$GITHUB_OUTPUT"
report-coverage:
needs: test
if: always()
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Download PR coverage
uses: actions/download-artifact@v4
with:
name: backend-coverage-pr
path: src/backend/coverage
- name: Download base coverage
uses: actions/download-artifact@v4
with:
name: backend-coverage-base
path: coverage-base
- name: Report coverage on PR
if: github.event_name == 'pull_request'
uses: davelosert/[email protected]
uses: davelosert/vitest-coverage-report-action@v2
with:
json-summary-path: src/backend/coverage/coverage-summary.json
json-final-path: src/backend/coverage/coverage-final.json
json-summary-compare-path: ${{ steps.base.outputs.found == 'true' && 'coverage-base/coverage-summary.json' || '' }}
json-summary-compare-path: coverage-base/coverage-summary.json
@@ -1,25 +0,0 @@
name: Dependabot auto-merge
on: pull_request
permissions:
contents: write
pull-requests: write
jobs:
automerge:
runs-on: ubuntu-latest
if: github.actor == 'dependabot[bot]'
steps:
- name: Dependabot metadata
id: metadata
uses: dependabot/[email protected]
with:
github-token: "${{ secrets.GITHUB_TOKEN }}"
- name: Enable auto-merge for patch updates
if: steps.metadata.outputs.update-type == 'version-update:semver-patch'
run: gh pr merge --auto --squash "$PR_URL"
env:
PR_URL: ${{ github.event.pull_request.html_url }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+8 -10
View File
@@ -45,32 +45,30 @@ jobs:
fi
- name: Checkout repository
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
uses: actions/checkout@v4
- name: Log in to GitHub Package Container registry
uses: docker/login-action@v4.8.0
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Log in to Docker Hub
uses: docker/login-action@v4.8.0
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Set up QEMU
uses: docker/setup-qemu-action@v4.4.0
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4.4.1
uses: docker/setup-buildx-action@v3
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v6.2.0
uses: docker/metadata-action@v5
with:
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
@@ -84,7 +82,7 @@ jobs:
type=raw,value=main,enable=${{ github.event_name == 'push' || inputs.push_main }}
- name: Build and push Docker image
uses: docker/build-push-action@v7.4.0
uses: docker/build-push-action@v5
with:
platforms: linux/amd64,linux/arm64
context: .
@@ -97,7 +95,7 @@ jobs:
cache-to: type=gha,mode=max
- name: Sync README to Docker Hub
uses: peter-evans/dockerhub-description@v5.0.0
uses: peter-evans/dockerhub-description@v4
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
+3 -8
View File
@@ -5,23 +5,18 @@ on:
paths:
- 'src/docs/**'
permissions:
contents: read
jobs:
build-docs:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v7.1.0
uses: actions/setup-node@v4
with:
node-version: '26'
node-version: '24'
cache: 'npm'
- name: Install dependencies
+40
View File
@@ -0,0 +1,40 @@
name: Maintain Release Merge PR
on:
push:
branches:
- main
jobs:
update-release-pr:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Git
run: |
git config user.name github-actions
git config user.email [email protected]
- name: Check for existing PR
id: find-pr
uses: juliangruber/find-pull-request-action@v1
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
branch: release
- uses: actions/checkout@v4
if: steps.find-pr.outputs.number == ''
with:
ref: release
- name: Reset release branch
if: steps.find-pr.outputs.number == ''
run: |
git fetch origin main:main
git reset --hard main
- name: Create/Update PR
if: steps.find-pr.outputs.number == ''
uses: peter-evans/create-pull-request@v6
with:
token: ${{ secrets.GITHUB_TOKEN }}
commit-message: Update release branch
title: Merge Release Auto-PR
body: Merging this PR will invoke release actions
branch: auto-update/release
-48
View File
@@ -1,48 +0,0 @@
name: Notify HeyPuter (dev)
on:
workflow_dispatch:
inputs:
puter_ref:
description: "puter branch, tag or SHA"
type: string
default: "main"
permissions:
contents: read
jobs:
authorize:
if: github.repository == 'HeyPuter/puter'
runs-on: ubuntu-latest
steps:
- name: Require maintain or admin
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
ACTOR: ${{ github.triggering_actor }}
run: |
ROLE=$(gh api "repos/$GITHUB_REPOSITORY/collaborators/$ACTOR/permission" --jq .role_name)
if [ "$ROLE" != "maintain" ] && [ "$ROLE" != "admin" ]; then
echo "::error::$ACTOR has role '$ROLE'; maintain or admin required"
exit 1
fi
notify:
needs: authorize
runs-on: ubuntu-latest
# Holds HEYPUTER_DISPATCH_TOKEN; limited to main so branch copies of this file can't read it.
environment: heyputer-dispatch
steps:
- name: Trigger heyputer dev deploy
env:
PUTER_REF: ${{ inputs.puter_ref }}
TOKEN: ${{ secrets.HEYPUTER_DISPATCH_TOKEN }}
run: |
jq -nc --arg ref "$PUTER_REF" \
'{event_type: "puter-dev-requested", client_payload: {puter_ref: $ref}}' |
curl -fsS -X POST \
-H "Authorization: token $TOKEN" \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/HeyPuter/heyputer/dispatches \
-d @-
echo "Requested \`$PUTER_REF\`" >> "$GITHUB_STEP_SUMMARY"
+1 -4
View File
@@ -5,12 +5,9 @@ on:
branches:
- main
permissions: {}
jobs:
notify:
runs-on: ubuntu-latest
environment: heyputer-dispatch
steps:
- name: Trigger heyputer build
run: |
@@ -18,4 +15,4 @@ jobs:
-H "Authorization: token ${{ secrets.HEYPUTER_DISPATCH_TOKEN }}" \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/HeyPuter/heyputer/dispatches \
-d '{"event_type":"puter-main-updated","client_payload":{"puter_ref":"main"}}'
-d '{"event_type":"puter-main-updated","client_payload":{"puter_ref":"main"}}'
+35 -86
View File
@@ -1,44 +1,28 @@
name: Puter.js API Tests
# Always triggers so it can be a required PR check; the `changes` job below
# decides whether puter.js-relevant paths actually changed (the suites live
# in src/puter-js/tests, so test-only changes are covered by the same glob,
# plus the pieces the runners depend on: the worker preamble, the in-memory
# test env in testUtil, and the backend vitest config the API-tests config
# extends), and every other job is skipped — not absent — when they didn't,
# which still satisfies a required check. Backend and extension changes run
# backend-tests.yaml instead.
# Runs on puter.js SDK changes — the suites live in src/puter-js/tests,
# so test changes are covered by the same glob — plus the pieces the
# runners depend on: the worker preamble, the in-memory test env in
# testUtil, and the backend vitest config the API-tests config extends.
# Backend and extension changes run backend-tests.yaml instead.
on:
pull_request:
types: [opened, synchronize, reopened]
paths:
- 'src/puter-js/**'
- 'src/worker/**'
- 'src/backend/testUtil.ts'
- 'src/backend/vitest.config.ts'
- 'tools/**'
- 'package.json'
- 'package-lock.json'
- '.github/workflows/puterjs-tests.yaml'
permissions:
contents: read
pull-requests: write
jobs:
changes:
runs-on: ubuntu-latest
outputs:
puterjs: ${{ steps.filter.outputs.puterjs }}
steps:
- uses: actions/[email protected]
with:
persist-credentials: false
- uses: dorny/[email protected]
id: filter
with:
filters: |
puterjs:
- 'src/puter-js/**'
- 'src/worker/**'
- 'src/backend/testUtil.ts'
- 'src/backend/vitest.config.ts'
- 'tools/**'
- 'package.json'
- 'package-lock.json'
- '.github/workflows/puterjs-tests.yaml'
# The JSDoc in src/puter-js/src is the source of truth for the SDK's public
# types. The declarations shipped to npm are generated from it at build time
# and never committed, so what needs guarding is the JSDoc itself: this
@@ -48,20 +32,16 @@ jobs:
# Its own job rather than a step in `test`: that one builds bundles and
# installs a browser, and this needs neither.
types:
needs: changes
if: needs.changes.outputs.puterjs == 'true'
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v7.1.0
uses: actions/setup-node@v4
with:
node-version: '26'
node-version: '24'
cache: 'npm'
- name: Install dependencies
@@ -71,20 +51,16 @@ jobs:
run: npm run check:puterjs:types
test:
needs: changes
if: needs.changes.outputs.puterjs == 'true'
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v7.1.0
uses: actions/setup-node@v4
with:
node-version: '26'
node-version: '24'
cache: 'npm'
- name: Install dependencies
@@ -107,68 +83,47 @@ jobs:
# SDK coverage on both refs, reported as a PR comment — same shape as
# backend-tests.yaml. This run uses the istanbul-instrumented bundle
# (the `test` job above keeps exercising the production build).
# `needs.changes` gates the steps below, not this job, on purpose: a
# job-level `if` on a matrix job skips the whole job as one unexpanded
# check ("coverage (${{ matrix.artifact }})", literally, since there's no
# matrix context to fill it in) instead of two real per-artifact checks —
# confusing to read, and unusable if this ever gets pinned as a required
# check. Letting the matrix always expand keeps the check names stable.
coverage:
needs: changes
# Overrides the default matrix naming, which would otherwise bake the
# base/head branch name (matrix.ref) into the check name — making it a
# different string on every PR and impossible to pin as a required check.
name: coverage (${{ matrix.artifact }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
# The base branch lives in this repo; for a fork PR, the head
# repo's copy of it can be arbitrarily stale.
- ref: ${{ github.base_ref }}
repository: ${{ github.repository }}
artifact: base
- ref: ${{ github.head_ref }}
repository: ${{ github.event.pull_request.head.repo.full_name }}
artifact: pr
steps:
- name: Checkout ${{ matrix.ref }}
if: needs.changes.outputs.puterjs == 'true'
uses: actions/[email protected]
uses: actions/checkout@v4
with:
persist-credentials: false
ref: ${{ matrix.ref }}
repository: ${{ matrix.repository }}
repository: ${{ github.event.pull_request.head.repo.full_name }}
- name: Set up Node.js
if: needs.changes.outputs.puterjs == 'true'
uses: actions/[email protected]
uses: actions/setup-node@v4
with:
node-version: '26'
node-version: '24'
cache: 'npm'
- name: Install dependencies
if: needs.changes.outputs.puterjs == 'true'
run: npm ci
- name: Install Playwright chromium
if: needs.changes.outputs.puterjs == 'true'
run: npx playwright install --with-deps chromium
# The base leg is best-effort — it only enriches the PR comment with
# a comparison, and the base ref may predate the coverage script.
- name: Run puter.js API tests with coverage
if: needs.changes.outputs.puterjs == 'true'
continue-on-error: ${{ matrix.artifact == 'base' }}
run: npm run test:puterjs:coverage
env:
CI: 'true'
- name: Upload coverage artifact
if: needs.changes.outputs.puterjs == 'true' && !cancelled()
uses: actions/upload-artifact@v7.0.2
if: always()
uses: actions/upload-artifact@v4
with:
name: puterjs-coverage-${{ matrix.artifact }}
path: src/puter-js/coverage/coverage-*.json
@@ -176,20 +131,15 @@ jobs:
if-no-files-found: ignore
report-coverage:
needs: [changes, coverage]
if: always() && needs.changes.outputs.puterjs == 'true'
needs: coverage
if: always()
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
uses: actions/checkout@v4
# Missing when the PR leg failed; `coverage (pr)` already reports that.
- name: Download PR coverage
id: pr-coverage
continue-on-error: true
uses: actions/[email protected]
uses: actions/download-artifact@v4
with:
name: puterjs-coverage-pr
path: src/puter-js/coverage
@@ -198,16 +148,15 @@ jobs:
# base run) — then report without the comparison column.
- name: Download base coverage
id: base-coverage
if: steps.pr-coverage.outcome == 'success'
continue-on-error: true
uses: actions/download-artifact@v8.0.2
uses: actions/download-artifact@v4
with:
name: puterjs-coverage-base
path: coverage-base
- name: Report coverage on PR
if: steps.pr-coverage.outcome == 'success' && steps.base-coverage.outcome == 'success'
uses: davelosert/vitest-coverage-report-action@v2.13.2
if: steps.base-coverage.outcome == 'success'
uses: davelosert/vitest-coverage-report-action@v2
with:
name: puter.js SDK
json-summary-path: src/puter-js/coverage/coverage-summary.json
@@ -215,8 +164,8 @@ jobs:
json-summary-compare-path: coverage-base/coverage-summary.json
- name: Report coverage on PR (no base comparison)
if: steps.pr-coverage.outcome == 'success' && steps.base-coverage.outcome != 'success'
uses: davelosert/vitest-coverage-report-action@v2.13.2
if: steps.base-coverage.outcome != 'success'
uses: davelosert/vitest-coverage-report-action@v2
with:
name: puter.js SDK
json-summary-path: src/puter-js/coverage/coverage-summary.json
+18
View File
@@ -0,0 +1,18 @@
on:
push:
branches:
- main
permissions:
contents: write
pull-requests: write
name: release-please
jobs:
release-please:
runs-on: ubuntu-latest
steps:
- uses: google-github-actions/release-please-action@v4
with:
release-type: node
+2 -2
View File
@@ -45,14 +45,14 @@ jobs:
fi
- name: Log in to GitHub Package Container registry
uses: docker/login-action@v4.8.0
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4.4.1
uses: docker/setup-buildx-action@v3
- name: Create alias tag
env:
-4
View File
@@ -66,13 +66,9 @@ AGENTS.md
coverage/
.vitest-reports/
*.log
undefined
servers.json
config.*.json
volatile/
# Local build override
docker-compose.override.yml
+54 -138
View File
@@ -1,18 +1,18 @@
# AGENTS.md
Guidance for AI coding agents working in this repository, and the source of truth for how we write code, tests, and docs here. Read it before making changes. FOLLOW GUIDANCE AS CLOSELY AS POSSIBLE. If you think a rule is wrong, raise an issue or flag a maintainer instead of doing what you think is right.
Guidance for AI coding agents working in this repository. Read this before making changes. FOLLOW GUIDANCE AS CLOSELY AS POSSIBLE. If you think the guidance is wrong, raise an issue or flag a maintainer — don't just do what you think is right. This is the source of truth for how we write code, tests, and docs in this repo.
## Documentation Index
Use these as the source of truth before exploring further:
- [README.md](README.md) — project overview and quickstart.
- [doc/architecture.md](doc/architecture.md) — backend layered stack (controllers → drivers → services → stores → clients), `PuterServer` wiring, `Context` (ALS), actors, and extensions.
- [doc/architecture.md](doc/architecture.md) — backend layered stack (controllers → drivers → services → stores → clients), `PuterServer` wiring, `Context` (ALS), and extensions.
- [doc/contributing-apis.md](doc/contributing-apis.md) — adding and maintaining public APIs end to end (backend surface → puter.js → types → docs → tests). Follow it for any API work.
- [doc/pagination.md](doc/pagination.md) — the one pagination convention for list APIs (limit/cursor/offset/includeTotal, envelope shape, cursor semantics).
- [doc/alarms.md](doc/alarms.md) — raising alarms and picking a severity (what pages, what only gets recorded), plus the config that routes them.
- [doc/self-hosting.md](doc/self-hosting.md) — running Puter outside hosted infra.
- [CONTRIBUTING.md](CONTRIBUTING.md) — contributor entry point and PR conventions.
- [CONTRIBUTING.md](CONTRIBUTING.md) — testing, security, AI-assisted code, PR conventions, Boy Scout Rule.
- [SECURITY.md](SECURITY.md) — how to report vulnerabilities (do not file them publicly).
- [BUG-BOUNTY.md](BUG-BOUNTY.md) — bounty program scope.
- [TRADEMARK.md](TRADEMARK.md) — trademark usage.
@@ -26,16 +26,14 @@ These apply everywhere — backend, puter.js, and GUI.
### Language & files
- Write ES modules, not CommonJS — we transpile and build as needed.
- TypeScript preferred for new files in the backend and extensions; convert existing JS there opportunistically when you're already touching a file. The GUI and puter.js are plain JavaScript — don't introduce TypeScript files in them.
- In plain-JS files, typing with JSDoc (`@type`/`@param`/`@returns`, `@typedef` for shared shapes, using the TypeScript type system) is encouraged. API types must not be `unknown` or untyped `...args`: spell out the real parameter and return shapes, unless the value passes through transparently to an upstream layer that owns its type. puter.js has stricter rules ([below](#types)).
- Make new types findable: descriptive `PascalCase` names, exported from the obvious entry point. A type used from many places goes in the owning module's `types.ts` (e.g. [src/backend/controllers/types.ts](src/backend/controllers/types.ts)); a type with a single consumer can stay next to it.
- Naming: `camelCase` for variables and functions, `PascalCase` for classes and files containing a class (`AuthService.ts`). **Prefer `camelCase` for all new code**, including parameters and internal object properties. `snake_case` is reserved for external contracts that already use it: wire/JSON keys, stable API error codes, public option names that already ship as `snake_case`, and established `puter_*` namespaced fields. Don't rename those (it's a breaking change), and don't add new `snake_case` beside them; a camelCase local can carry a value into a wire key (`{ operation_id: operationId }`).
- TypeScript preferred for new files in the backend and extensions; convert existing JS there opportunistically when you're already touching a file. The GUI and puter.js are plain JavaScript — don't introduce TypeScript files in them. In puter.js, type with JSDoc instead (see the puter.js section for what must be typed).
- **Reuse before adding.** Search for an existing type, helper, or implementation and extend it; only add a new one when nothing suitable exists.
- Make new types findable: descriptive `PascalCase` names, exported from the obvious entry point. A type used from many places belongs in the owning module's `types.ts` (e.g. [src/backend/controllers/types.ts](src/backend/controllers/types.ts)) rather than bloating a logic file; a type with a single consumer can stay next to it.
- Naming: `camelCase` for variables/functions, `PascalCase` for classes and files containing a class (`AuthService.ts`). **Prefer `camelCase` for all new code** — new local variables, parameters, functions, and internal object properties. `snake_case` is reserved for identifiers that are part of an external contract: wire/JSON keys sent to or received from the backend, stable API error codes, public option names that already ship as `snake_case`, and established `puter_*` namespaced fields. Don't rename those (it's a breaking change), but don't introduce new `snake_case` alongside them either — a camelCase local can carry a value into a `snake_case` wire key (`{ operation_id: operationId }`).
### Comments
Prefer self-documenting code. Comment only when the _why_ is non-obvious or a usage detail would trip the next reader. **A comment is a sentence or two**: one or two lines of `//`, a few of `/** ... */` JSDoc. State the constraint and stop. Don't restate the code, narrate history ("used to swallow the failure, which meant…"), argue the design, or explain parameters the signature already shows. Reasoning that needs paragraphs belongs in a design doc.
Don't reference the current task, PR, or version; those rot. **No ticket references** (`PUT-1234`, `// fix for FOO-99`) in code, comments, or test names: describe the why in domain terms. Section dividers use plain ASCII (`// -- Section --`), never box-drawing characters.
Keep comments light; prefer self-documenting code. Comment only when the _why_ is non-obvious or a usage detail would trip the next reader. Use `//` for single lines and `/** ... */` JSDoc when it genuinely needs more — if a comment runs long, it's probably too long. Don't restate the code and don't reference the current task, PR, or version — those rot. **No ticket references** (`PUT-1234`, `// fix for FOO-99`) in code, comments, or test names; describe the why in domain terms, not project-management terms. Use plain ASCII `-` in comment section dividers (`// -- Section --`), never box-drawing characters.
### Security & privacy
@@ -47,179 +45,97 @@ Before opening a PR, scan the diff for:
When in doubt, return less. Auth-, permission-, or data-export-related changes deserve an explicit callout in the PR description.
Commit messages and PR descriptions never credit anyone or mention a bug report. For security fixes, state what the change does mechanically ("scoped tokens do xyz now"), not the vulnerability or how it could be abused.
### Reuse and abstraction
- **Search before you build.** Before writing a type, helper, cache/TTL map, retry, lock, pagination, validator, serializer, error mapper, host/email normalizer, or provider adapter, grep `src/backend/util/`, the owning layer, and `extensions/` for one that exists, and extend it instead of adding a parallel copy. If you find two copies, consolidate them or flag it. Before opening the PR, check the diff for new helpers that duplicate existing ones.
- **One path per operation.** Single-item variants call the batch version. Legacy and v2 endpoints share one service method. A second entry point (OIDC, WebDAV, admin, cron) calls the same service as the first instead of re-implementing it.
- **No abstraction without a second caller.** No base class, interface, factory, registry, decorator, or option bag with one implementation. No wrappers that only forward, no interface methods nobody calls, no methods only tests call.
- **Variants are data.** Providers or models that differ only in constants (base URL, prices, key names) are entries in a table read by one implementation, not one file each.
### Hot paths
- No per-row queries or awaits in loops: use the store's batch method (add one if missing) or [src/backend/util/concurrency.ts](src/backend/util/concurrency.ts).
- Bound everything: queries get `LIMIT`/pagination, in-memory maps get eviction, outbound calls get a timeout and a size cap.
- Build clients, catalogs, and config-derived lookups once, not per request.
- Listeners on fleet-wide events (`fs.write.file`, route lifecycle) return before any I/O for requests they don't concern.
### Working rules of thumb
- **Run it, don't just compile it.** "It type-checks" is not "it works." Exercise the code path end-to-end at least once.
- **Read the neighbors before writing.** Match the shape of similar things already in the tree. If you think the existing pattern is wrong, raise it — don't quietly diverge.
- **Test new behavior.** Every new function, endpoint, driver method, or logic branch gets a test; every bug fix gets a regression test that fails before the fix. If something is genuinely hard to test, skip it but say so in the PR.
- **Boy Scout Rule, proportional to the change.** In files you're already touching, fix the typo, the dead import, the missing test, the bit you had to read twice. Don't ride a refactor along with a bug fix.
- **Understand what you commit.** AI assistance is fine; shipping code you couldn't have written, debugged, or defended in review is not.
- **A limit change is not done until the docs change with it.** Every rate limit, concurrency cap, quota, and allowance in the code is published in [src/docs/src/rate-limits-and-quotas.md](src/docs/src/rate-limits-and-quotas.md) — an undisclosed limit is one developers discover as a service failure. If a PR moves any of these numbers, the same PR updates that page. The numbers live in `src/backend/controllers/fs/limits.ts` (filesystem), `src/backend/drivers/util/aiLimits.ts` (all AI drivers), per-driver `rateLimit`/`concurrent` configs, and `src/backend/data/subPolicies/` (free-tier allowances).
---
## Local development
To test a local Docker build, use the gitignored `docker-compose.override.yml` described in [Self-Hosting → Building from source](doc/self-hosting.md#building-from-source-instead-of-pulling). Keep local build settings there, not in `docker-compose.yml`.
When budget checks block unrelated local API tests, you may set `"unlimitedMetering": true` without further approval. Merge it into the gitignored `config.json` (`npm start`) or `puter/config/config.json` (Docker) and restart Puter. It covers every account on that instance, guests included, and usage is still recorded. Keep normal settings when testing budget or subscription enforcement, and leave shared defaults unchanged.
- **Boy Scout Rule, proportional to the change.** Fix the obvious typo or dead import in files you're already touching; don't ride a refactor along with a bug fix.
- **Understand what you commit.** AI assistance is fine; shipping code you couldn't defend in review is not.
---
## Backend
Controllers → drivers (optional) → services → stores → clients, all configured from `IConfig`. Each layer depends only on the layers beneath it and receives them through its constructor; `PuterServer` ([src/backend/server.ts](src/backend/server.ts)) wires them. [doc/architecture.md](doc/architecture.md) is the full reference.
A layered stack with explicit dependency injection: each layer depends only on the layers beneath it, receives them through its constructor, and `PuterServer` ([src/backend/server.ts](src/backend/server.ts)) wires the whole thing together. [doc/architecture.md](doc/architecture.md) is the full reference.
A public API can be a controller or a driver. Prefer a controller when you need fine-grained control over routes and gates; [doc/contributing-apis.md](doc/contributing-apis.md#controller-or-driver) has the decision guide.
| Layer (top → bottom) | Lives in | Responsibility |
| -------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Controllers** | [src/backend/controllers/](src/backend/controllers/) | Route handlers: parse/validate input, per-route gates via `RouteOptions` (auth, subdomain, rate limit, body parsers), call services, format responses. |
| **Drivers** | [src/backend/drivers/](src/backend/drivers/) | Optional RPC-style handlers on `/drivers/*`; thin shells that validate inputs and call services/stores. |
| **Services** | [src/backend/services/](src/backend/services/) | Business logic. Assume the caller is already authenticated/authorized. |
| **Stores** | [src/backend/stores/](src/backend/stores/) | Persistence; wraps clients in the domain shapes services consume. |
| **Clients** | [src/backend/clients/](src/backend/clients/) | Adapters for external/internal services (sql, redis, s3, email, event bus). Protocols, not domain concepts. |
| **Config** | `config.*.json` → `IConfig` | Flat, typed config object every layer receives at construction. |
Layer rules:
A public API can be exposed through either a controller or a driver — both are supported; prefer a controller when you need fine-grained control over routes and gates. [doc/contributing-apis.md](doc/contributing-apis.md) has the decision guide with links to the decorators and middleware for each.
- **Don't reach across layers.** Controllers don't poke clients; services don't register routes; lower layers never import from controllers. If you want to, the abstraction is wrong; fix the abstraction.
- **Gates run at the edge.** Controllers and drivers validate input and apply auth gates; services assume the caller is already authenticated and authorized.
Cross-layer rules:
- **Don't reach across layers.** Controllers don't poke clients; services don't register routes. If you want to, the abstraction is wrong — fix the abstraction.
- **Don't call sideways within a layer for code reuse.** Two services needing the same logic means a util/helper, not a service-to-service dependency.
- **A table's store owns every query against it.** Services, controllers, and extensions don't run raw SQL/redis against a table that has a store; add the store method so cache invalidation stays with the write.
- **Drivers and controllers stay thin.** Domain rules a second entry point needs (signup, app CRUD, worker deploy, AI routing/billing) live in a service, not in the first driver or controller that needed them.
- **Prefer explicit arguments over [`Context`](src/backend/core/context.ts) (ALS).** Use it only for request-scoped values that would otherwise thread through many layers; today that's mostly `actor` and `req`.
- **Prefer explicit arguments over `Context` (ALS).** Reach for [Context](src/backend/core/context.ts) only for genuinely request-scoped values that would otherwise thread through many layers — today mostly `actor` and `req`.
### Extensions
[extensions/](extensions/) is for **non-crucial parts of the system**: Puter must still work with the extension removed. If core needs to call it, it belongs in core ([whoami](extensions/whoami.ts) is the cautionary example). Inside an extension, follow the same layered structure, unless it only needs a few route handlers, in which case the `extension.get/post/...` helpers are enough on their own. The `extension` API is in [doc/architecture.md](doc/architecture.md#extension-api).
[extensions/](extensions/) parallels the layered stack and is for **non-crucial parts of the system** — Puter still works with the extension removed. If core needs to call it, it belongs in core, not an extension (see [whoami](extensions/whoami.ts) as the cautionary example). The `extension` global ([src/backend/extensions.ts](src/backend/extensions.ts)) exposes:
- `extension.registerClient/Store/Service/Driver/Controller(name, Class)` for first-class additions.
- `extension.on(event, handler)` and `extension.get/post/put/delete/patch/use(path, opts?, handler)` for the lightweight common case. `opts` is the same `RouteOptions` controllers use.
- `extension.import('client' | 'store' | 'service' | 'controller' | 'driver')` — lazy proxy to instantiated core objects. `extension.config` exposes live config.
Follow the same layered structure inside an extension — unless it only needs a few route handlers, in which case the lightweight helpers are enough on their own.
### Backend tests
- Vitest; test files sit next to the code they test (`*.test.ts` / `*.test.js`). Run with `npm run test:backend`.
- **Mock data, not methods.** Stub inputs (fixtures, fake rows, payloads), not the function under test or the layer beneath it — over-mocking produces tests that pass while production breaks. If you must mock, mock at a real boundary (a client/external service).
- **Prefer the test server over mocking deps.** `setupPuterTestEnv()` in [src/backend/testUtil.ts](src/backend/testUtil.ts) boots a fully in-memory backend; hit a real database/client shape where reasonable — integration shapes catch what mocked unit tests miss.
- **Test code, not docs.** Never write a test that reads a file under `src/docs/` and asserts on its wording or numbers. Docs are kept in step by the PR (see the limits rule above) and checked in review; a test that greps a markdown page fails on every rewording and verifies nothing about behavior.
---
## puter.js (the SDK)
[src/puter-js/](src/puter-js/) is the public SDK. It ships live from `https://js.puter.com/v2/` with no version pinning, so every change reaches every existing app at once; see [Rule zero](doc/contributing-apis.md#rule-zero-dont-break-callers).
[src/puter-js/](src/puter-js/) is the public SDK. It ships live from `https://js.puter.com/v2/` with no version pinning — every existing app picks up changes immediately. Treat every observable behavior (signatures, response fields, error codes) as something a production app depends on.
Layout: SDK modules in [src/puter-js/src/modules/](src/puter-js/src/modules/) (one file or directory per area — `FileSystem/`, `kv/`, `ai/`, …), shared plumbing in [src/puter-js/src/lib/](src/puter-js/src/lib/), generated (gitignored) type declarations in `src/puter-js/types/`, API tests in [src/puter-js/tests/api/](src/puter-js/tests/api/), UI e2e tests in [src/puter-js/tests/e2e/](src/puter-js/tests/e2e/), developer docs in [src/docs/](src/docs/).
Every SDK change carries all five of these, or the PR is incomplete. [doc/contributing-apis.md](doc/contributing-apis.md) has the steps for each.
Language & typing: puter.js source is plain JavaScript — never TypeScript files — typed via JSDoc. **The JSDoc is the source of truth for the SDK's types.** `src/puter-js/types/` is `tsc --emitDeclarationOnly` output: the SDK build (`npm run build` in `src/puter-js`) generates it, the npm tarball ships it so TypeScript consumers get declarations, and git ignores it — it is never committed and never edited by hand. `npm run check:puterjs:types` generates it in CI and type-checks the result without `skipLibCheck`. The one hand-written declaration file is [src/puter-js/index.d.ts](src/puter-js/index.d.ts), which decides what is public and re-exports the generated names.
1. **Backward compatibility**, unless a maintainer explicitly signs off on a break ([rules](doc/contributing-apis.md#maintaining-an-existing-api)).
2. **Tests** in the cross-platform API suite, or the Playwright harness for desktop-rendered `puter.ui.*` ([steps](doc/contributing-apis.md#6-tests)).
3. **Docs** in [src/docs/src/](src/docs/src/) that match the implementation exactly ([steps](doc/contributing-apis.md#5-docs), [writing rules](#developer-docs)).
4. **Types**, per the rules below.
5. **Error handling**: `{ message, code }` with stable `snake_case` codes, as in [modules/kv/](src/puter-js/src/modules/kv/) ([steps](doc/contributing-apis.md#3-puterjs)).
### Types
puter.js source is plain JavaScript typed via JSDoc, and **the JSDoc is the source of truth for the SDK's types**. `src/puter-js/types/` is `tsc --emitDeclarationOnly` output: the SDK build (`npm run build` in `src/puter-js`) generates it, the npm tarball ships it, and git ignores it. Never commit or hand-edit it. `npm run check:puterjs:types` generates it and type-checks the result without `skipLibCheck`; run it after any type change. The one hand-written declaration file is [src/puter-js/index.d.ts](src/puter-js/index.d.ts): it decides what is public and re-exports the generated names, so name a new type there if consumers should be able to import it.
- Public (exposed) methods carry JSDoc types for parameters and return value, with one `@overload` block per accepted call form; those overloads _are_ the published signature. Typing private helpers is at your discretion; annotate where it helps the next reader.
- Tag members `@internal`, not `@private`, to strip them from the generated declarations.
- Declarations must match runtime behavior exactly. A wrong type is worse than a missing one.
Declare a shape where it belongs and reference it elsewhere with an `import(...)` type. A shape more than one file needs lives in the module's `types.js`; one shared across modules lives in [src/puter-js/src/lib/types.js](src/puter-js/src/lib/types.js); one with a single consumer can stay next to it:
Declare a shape where it belongs, and reference it with an `import(...)` type from elsewhere. A shape more than one file needs lives in the module's `types.js`; a shape shared across modules lives in [src/puter-js/src/lib/types.js](src/puter-js/src/lib/types.js); a shape with one consumer can stay next to it:
```js
/** @typedef {import('./types.js').KVOptConfig} KVOptConfig */
```
Use `@typedef {Object}` + `@property` for any shape whose fields need documenting; it's the only JSDoc form that carries a per-field doc comment into the generated declarations. Use the inline form for small internal shapes with nothing to say per field, and prefer `unknown` over `*`:
Use `@typedef {Object}` + `@property` for any shape whose fields need documenting — it is the only JSDoc form that carries a doc comment per field into the generated declaration. Keep the inline object-literal form for small internal shapes with nothing to say about each field, and prefer `unknown` over `*`:
```js
/** @typedef {{ key: string; value: unknown }} KVEntry */
/** @typedef {{ key: string, value: unknown }} KVEntry */
```
---
Public (exposed) methods must carry JSDoc types — parameters and return value — with one `@overload` block per accepted call form, since those overloads *are* the published signature. Unexposed/private helpers are typed at the contributor's discretion: annotate where it helps the next reader, and either way keep them clean. Members tagged `@internal` are stripped from the generated declarations, so use that tag rather than `@private` to keep something off the public surface.
## Developer docs
Typing in JS files is encouraged: annotate with JSDoc `@type`/`@param`/`@returns` using the TypeScript type system, and define shared shapes with `@typedef`. API types must not be `unknown` or untyped `...args` — spell out the real parameter and return shapes; the only exception is values passed through transparently to an upstream layer that owns their type. For example:
[src/docs/src/](src/docs/src/) is the published puter.js documentation: one reference page per method (`<Area>/<method>.md`, [steps](doc/contributing-apis.md#5-docs)) and recipes in [src/docs/src/recipes/](src/docs/src/recipes/).
```js
/**
* @typedef {{key:string, value: unknown}} KVEntry
*/
### Page structure
/** @type {KVEntry[]} */
let entries = [];
```
The reference has two kinds of page:
Every SDK change carries all five of the following — a puter.js PR missing one is incomplete:
- **API overview pages** (`<Area>.md`, such as [KV.md](src/docs/src/KV.md)) give a quick top-level picture of what the API is for and what it can do, with short examples and a list of its functions that links every function page in the area. Leave out the details. The first sentence of the intro says what the API is about. This is a judgment call too: when the API depends on a Puter concept the reader may not know, such as Teams, the first sentence can name it and the sentences after it explain it.
- **Function pages** (`<Area>/<method>.md`) hold the details: parameters, defaults, return values, errors, limits, and edge cases.
1. **Backward compatibility.** Mandatory unless a maintainer explicitly signs off on a break. Existing call signatures keep working (including both positional and options-object forms where a method supports them); new parameters are optional with defaults that preserve old behavior; never rename or repurpose existing params, response fields, or error codes. New parameter names are `camelCase` (existing `snake_case` stays for compatibility). Say in the PR how existing callers are unaffected.
2. **Tests.** Add or extend a suite in [tests/api/suites/](src/puter-js/tests/api/suites/) (`<area>.suite.ts`; register new suites in `suites/index.ts` — no globbing). One suite runs unchanged on node, browser, and workerd via `npm run test:puterjs`; never write a per-platform test. The runners execute the **built** bundle — run `npm run build:workerLib` after SDK changes or the suite silently tests stale code. For `puter.ui.*` methods rendered by the desktop, use the Playwright e2e harness instead — see [src/puter-js/TESTING.md](src/puter-js/TESTING.md).
3. **Docs.** New or changed APIs update [src/docs/src/](src/docs/src/): the method page (`<Area>/<method>.md`, with frontmatter and a runnable example) and the area overview when the surface changes. Docs are the contract users code against — signatures, defaults, and return shapes must match the implementation exactly.
4. **Types.** Type the change in JSDoc on the implementation, then run `npm run check:puterjs:types` to confirm it still produces declarations that type-check. Name any new type in [src/puter-js/index.d.ts](src/puter-js/index.d.ts) if consumers should be able to import it. Declarations must match runtime behavior exactly — a wrong type is worse than a missing one.
5. **Error handling.** Reject/throw `{ message, code }` objects with stable `snake_case` codes, matching the existing modules (see `KV.js`). Validate cheap preconditions client-side before making the network call; pass backend errors through unchanged rather than swallowing or re-wrapping them. Error codes are API surface — changing one is a breaking change.
An overview page answers "what is this API and is it what I need?", and a function page answers "how exactly does this call behave?". A change to one function usually only touches that function's page, so a new `kv.list()` option goes in `KV/list.md` without a matching update to `KV.md`. Don't copy a function's options, limits, or edge cases onto the overview page for completeness. Where a detail goes is a judgment call, not a hard rule: update the overview when a change affects how someone would describe the API as a whole.
When you add a page, add it to [src/docs/src/sidebar.js](src/docs/src/sidebar.js) under its area, or it won't show up in the navigation.
**Sidebar order.** Order the API areas in the sidebar by how often developers use them, most common first. Overview stays first and Objects stays last. Inside an area, order the pages by use too, and keep pages that make up one feature next to each other, such as `share()`, `unshare()`, `listShared()`, and `listSharedByMe()`. A group goes where its most-used page would go. A helper that only supports another function, such as `listModels()` for `chat()`, goes by its own use. Put create, list, get, update, and delete in that order, and constants such as `MAX_KEY_SIZE` last. The function list on the area's overview page uses the sidebar's order. When you add a runnable example, add its HTML to `src/docs/src/playground/examples/<slug>.html`, register the slug in [src/docs/src/examples.js](src/docs/src/examples.js), and link the code block to it by putting the slug after the language with a semicolon (` ```html;kv-set `).
### Function page layout
A function page opens with one sentence that says what the function does, in its own paragraph, such as "Given a prompt, generate an image using AI." Put any more detail in the paragraphs after it.
After the intro, a function page has these `##` sections, in this order:
1. Syntax
2. Parameters
3. Return value
4. Errors, if the function has any
5. Sections for specific topics, if the page needs them, with `## Model-specific features` last
6. Examples
[chat.md](src/docs/src/AI/chat.md) follows this layout.
- **Options as a table.** Document an options object as a table with Option, Type, and Description columns. Give each option its own row, even when two options go together, such as `width` and `height`. Order the rows by how often developers use them, most common first, without group rows. In chat.md that's the core options, then reasoning, tools, response handling, request control, and output.
- **Common options in the table, model-specific behavior in its own section.** A page whose function works with many models or providers, such as [chat.md](src/docs/src/AI/chat.md) and [txt2img.md](src/docs/src/AI/txt2img.md), splits its options this way:
- **The options table** under Parameters lists the options that work with every model, and describes each one the way it works for every model.
- **`## Model-specific features`**, the last section before Examples, covers everything else. It has a `###` for each model family or provider, named after it, such as "OpenAI models" or "Cloudflare models". Under it go the options only some models accept, and the common options a model handles differently, such as a narrower range, other allowed values, or an option it ignores.
- **Write there only what's different.** A `###` lists an option only when that model or provider adds it or handles it differently. An option that works the same everywhere stays out, even if the provider supports it, and a model or provider with nothing different gets no `###`.
- **Model-specific features is the only place for model-specific details.** Every limit, format, allowed value, default, price, or quirk that applies to some models goes there, in that model's or provider's `###`, and nowhere else. The options table and topic sections (such as Image inputs or Video input) describe what's true for every model and link to Model-specific features for the rest. Don't add a table or sentence there that compares providers or repeats a model's limits. Each copy is another source of truth to keep in sync, and copies drift when a model changes.
- **Link the two.** Directly under the options table, put `Some models accept additional options or handle these options differently. See [Model-specific features](#model-specific-features).` A row whose option behaves differently on some models can also end with `See [Model-specific features](#model-specific-features).`
- **A long topic gets its own `##` section.** When explaining an option or feature takes more than a sentence or two, such as function calling or output dimensions, give it a section between Errors and Examples and link to it from the options table. This keeps the reference sections short.
- **Usage patterns are examples.** A way of using the API, such as stopping a stream, goes in a runnable example that the relevant section links to ("See the [stop a stream example](/playground/ai-chat-stop-stream/)."), not in a section of its own.
- **Each example has a `###` heading** under `## Examples`, so every example shows in the table of contents.
### Writing docs
These apply to reference pages and recipes alike.
- **Document the developer-facing side only.** The reader is building on top of puter.js, so cover what affects their code: signatures, defaults, return shapes, errors, limits, and behavior they can observe. Leave out internal workings (which service handles the call, storage layout, caching, retries) unless they change what the developer has to do. [rate-limits-and-quotas.md](src/docs/src/rate-limits-and-quotas.md) is the exception: it publishes every limit, including timeouts and caps that come from how the backend works.
- **Document Puter as it runs on puter.com.** The docs are for developers building on production Puter. Leave out behavior that only happens on a self-hosted or differently configured server, such as a provider that isn't set up there.
- **Describe the current state.** Docs aren't a changelog. Write how the API behaves now, without "now", "no longer", "existing code is unaffected", or other wording that compares it to an older version.
- **Verify before you document.** Run the calls you describe against a running Puter instead of inferring behavior from the source.
- **Write plainly.** Use the literal words a developer would say out loud, with contractions and everyday words. No idioms or figurative verbs ("walk away from something shared with you" → "remove your own access", "nothing to page through" → "without pagination"). No formal register ("present only while more pages remain" → "includes a `cursor` if there are more pages to load"). Prefer dev terms such as atomic, concurrent, and insertion order.
- **State the rule, then its consequence, in short sentences.** No metaphor in place of a rule. Name a concrete action ("you can create any files and folders inside it") over an abstract permission ("yours to organize freely"). Cut a sentence that only repeats the next one.
- **Describe behavior directly.** No quoted inner-voice phrases. Write "`isNew` is `true` when the person did not have access before", not "this tells 'I just gave this to someone' from 'they already had it'".
- **Keep references easy to follow.** Name what a list or code block belongs to in the sentence directly above it. Name a thing once and keep that name. Introduce an actor as "a user" before giving them a role: "A user can create a team. The user who creates the team becomes its owner."
- **Sentence mechanics.** Never start a sentence with inline code: write "To store data, use the [`set()`](/KV/set/) method." Link every inline mention of an API to its page, every time. No em dashes, and no colons or semicolons mid-sentence; a colon that introduces a code block is fine.
- **No `##` or `###` headings inside tabs.** The table of contents lists every `##` and `###` heading on the page, including the ones in a hidden tab (an `example-content` block switched by the `example-group` pills), so its links lead to content the reader can't see. Put the heading above the tabs, or use `####` inside a tab, which the table of contents skips.
- **Show the correct way.** No "don't" and no wrong-way-first examples, in recipes and playground examples alike. Mention a wrong approach only when developers already hit it in practice, and state how it fails as a fact: "`add()` reads a plain object as a path map, so the call rejects with …".
### Recipes
Recipes are tested, copy-pasteable patterns for a task, one layer above the per-method reference, often combining several APIs. Agents find them through the `## Recipes` section of `llms.txt`.
- **Files.** One `.md` per recipe in [src/docs/src/recipes/](src/docs/src/recipes/), with `title`, `description`, `tags`, and optional `order` front matter; they're picked up automatically. Tags must exist in [src/docs/src/recipe-tags.js](src/docs/src/recipe-tags.js) or the build fails. Quote a `description` that contains a colon followed by a space. Every top-level `.md` in that directory is built as a recipe, so keep other markdown out of it.
- **Title.** Short, Title Case, starts with a verb, and leaves out "to Your App" ("Add Team Support", "Deploy Workers Programmatically"). The slug is the title in kebab case, with no API-name prefix. In `llms.txt` an agent sees only the title and description before deciding to open a recipe, so word them to match the task the reader is trying to do.
- **Description.** Written for someone who has never heard of Puter. Start with "Learn how to", put the action first, and say what the reader's app gains: "Learn how to add file sharing to your app with Puter.js, so users can share files in their Puter account with other people." No API names, no mechanism, no list of example cases, and no "not just …" contrast.
- **Intro.** Explain the situation and why you'd do this, tied to something the reader already knows ("`puter.kv` is your database in Puter.js"). No "Use this when" bullet list.
- **Structure.** Get to the first working call as early as possible, then add validation, rate limits, and sign-in after it. Explain a Puter concept the first time it appears, such as what `me.puter` is or that `puter.workers.exec()` opens the Puter sign-in window. Split recipes by what the reader needs, not by technique, and give a family of recipes a shared tag.
- **Reading order.** A recipe reads top to bottom. Each code block uses only functions and variables from blocks above it, so the reader never has to search further down the page to understand it. Start with a small version that works, and have each section add one case. When a section changes a function from earlier, show the whole new version of that function there. If the code is built up over several sections, end with a `## Full Code` block that has all of it.
- **Link to the reference for full option lists.** Mention only the options the recipe uses, then link to the function page for the rest ("All the sort options are listed on [`puter.fs.readdir()`](/FS/readdir/)."). A copied list of options, rules, or scopes goes stale when the reference changes.
- **Limits.** Mention a limit only when it's the reason for a pattern the recipe uses, such as one wildcard subscription instead of one per key, and link to [rate-limits-and-quotas.md](src/docs/src/rate-limits-and-quotas.md) or the function page for the number. Leave out limits that code following the recipe never reaches.
- **Code.** Platform-neutral: just the code, with top-level `await`. No `<script>` or `<html>` wrapper, no IIFE.
- **Tradeoffs.** Justify a choice with one scenario the reader can picture right away, in about two short paragraphs, or fold each point into the section where it matters. The description states the benefit only.
- **Notes.** Leave out `## Notes` by default. Use it only to point to an architecturally different approach. A limit the recipe needs goes in the section it applies to, and other facts belong on the linked reference page.
[doc/contributing-apis.md](doc/contributing-apis.md) walks the full lifecycle of adding an API across backend + SDK.
---
@@ -227,10 +143,10 @@ Recipes are tested, copy-pasteable patterns for a task, one layer above the per-
[src/gui/](src/gui/) is the Puter desktop: deliberately plain JavaScript + jQuery with HTML-string templates. Don't introduce a UI framework or a new rendering pattern.
**Conformity over novelty**: match the existing design and code structure even where you'd choose differently. A visually or structurally divergent addition is a defect even when it works.
The guiding rule here is **conformity over novelty** — match the existing design and code structure even where you'd personally choose differently. A visually or structurally divergent addition is a defect even when it works.
- **Reuse existing UI primitives before writing new ones.** Windows and dialogs are `UIWindow*` functions in [src/gui/src/UI/](src/gui/src/UI/); generic pieces already exist (`UIAlert`, `UIPrompt`, `UIContextMenu`, `UINotification`, `UIPopover`, widgets in [UI/Components/](src/gui/src/UI/Components/)). A new window should read like its neighbors: an async function taking an options object, composing an HTML string, wiring behavior with jQuery, delegating to `UIWindow(...)`.
- **Match the visual language.** Use existing CSS classes (`button`, `button-primary`, window chrome, form styles) and copy the layout patterns of neighboring windows; new styles go in [src/gui/src/css/](src/gui/src/css/) following existing conventions. Verify anything positional (menus, overlays, z-index) on both desktop and mobile viewports.
- **i18n every user-facing string.** No hardcoded UI text — use `i18n('key')` and add the key to [src/gui/src/i18n/translations/en.js](src/gui/src/i18n/translations/en.js).
- **i18n every user-facing string.** No hardcoded UI text — use `i18n('key')` and add the key to [src/gui/src/i18n/translations/en.js](src/gui/src/i18n/translations/en.js). Run `npm run check-translations` before opening the PR.
- **Shared logic goes in helpers/services.** Reusable non-UI logic belongs in [src/gui/src/helpers/](src/gui/src/helpers/) or [src/gui/src/services/](src/gui/src/services/), not copy-pasted between windows.
- **Tests.** Vitest is wired for the GUI (`src/**/*.test.js`; see [appOrder.test.js](src/gui/src/UI/Dashboard/appOrder.test.js) for the shape). Extract pure logic into functions and test those. Desktop behavior driven through puter.js (`puter.ui.*`) is covered by the Playwright harness in [src/puter-js/tests/e2e/](src/puter-js/tests/e2e/) — add a spec there when you change how the desktop renders SDK-driven UI.
+37 -66
View File
@@ -6,56 +6,33 @@ We at **Puter** are committed to maintaining a secure experience for our users a
The following are in scope for this program:
- **The Puter open-source project** (available at [github.com/HeyPuter/puter](https://github.com/HeyPuter/puter))
- **`puter.com`**
- **`api.puter.com`**
* **The Puter open-source project** (available at [github.com/HeyPuter/puter](https://github.com/HeyPuter/puter))
* **`puter.com`**
* **`api.puter.com`**
Out-of-scope:
- Third-party services, applications, or libraries not maintained by Puter.
- Social engineering attacks (e.g., phishing against staff).
- Denial of Service (DoS), spam, or volumetric attacks.
- Physical security issues.
## Eligibility
We pay bounties for issues that threaten user security and privacy, or the security and health of the platform. We appreciate every report, but hardening suggestions and correctness bugs are not eligible, even when we fix them. Please open a GitHub issue for those.
A report must show how an attacker can directly perform the action against another account or the system. It is not eligible if it:
- Starts from access the attacker should not already have: a stolen session or token, the victim's password, a root or full-access token, or script execution on `puter.com`.
- Needs the victim to do more than ordinary use (opening a link or an app), such as publishing their own secrets, sending a verification link to an address they don't control, or approving a prompt that accurately states what it grants.
- Gives the attacker nothing they couldn't already get through a supported API.
- Only reproduces in a non-default or development setup (SQLite, a `custom` OIDC provider, an unset outbound proxy, cookies or config planted by hand). Findings in the open-source code must affect `puter.com` or a default self-hosted install.
**Shared state.** Sharing files, KV entries, or cross-app KV access means the sharer trusts the sharee. Exploits that depend on that trust, such as a sharee misusing access the sharer granted, may be treated as hardening or lower severity.
Hardening we ship in a public PR does not make similar code paths vulnerabilities. They are judged by these same rules.
* Third-party services, applications, or libraries not maintained by Puter.
* Social engineering attacks (e.g., phishing against staff).
* Denial of Service (DoS), spam, or volumetric attacks.
* Physical security issues.
## Known Non-Issues (Please Check Before Submitting)
The following have already been reviewed and determined **not to be vulnerabilities**. Reports that only re-describe one of these are **not eligible for a reward** and will be closed as non-issues — even if they include new code references. Please review this list before submitting:
- **XSS / CORS / token issues scoped to `api.puter.com`.** The primary user session cookie lives on `puter.com`, not on the API origin, so reflected/stored XSS, CORS, or token handling on `api.puter.com` is out of scope. This explicitly includes **missing `Content-Security-Policy: sandbox` on any endpoint that serves a file inline** — `/token-read`, `/file`, `/fs/read`, WebDAV, and any MIME type the `applyInlineContentSecurity` allow-list does not cover (`text/xsl`, `image/svg+xml`, `application/xhtml+xml`, …). Script execution on the API origin has nothing to reach: there is no session cookie, no `localStorage`, and no ambient credential there. WebDAV additionally has to return an accurate MIME type for real WebDAV clients. The one exception we _do_ evaluate on its merits is anything that abuses the app-scoped `puter_token_v2` companion cookie set on the API host — if your report does not name a specific credential that is actually reachable from the API origin, it is this non-issue.
- **SSRF via `secureFetch`.** Production routes outbound requests through an isolated proxy that has no access to internal/SSRF-sensitive resources.
- **Attacks that depend on guessing an `appInstanceID` or app UID.** These are random 128-bit secret values and are not considered guessable.
- **Apps invoking drivers, creating workers, or using KV.** Applications are intended to do this; worker permissions are scoped to the owning app. This is by design.
- **App metadata or app user-count "leaks".** This information is currently public by design. This covers **private apps** too: private apps are an unreleased, admin-only beta for paid listings in the Puter App Center. Their `index_url` and other metadata are not secret, the flag is not user-settable, and no production app relies on it. Reports that a private app's metadata is reachable through a sibling route (`/get-launch-apps`, `index_url` drift on the hosting gate, etc.) are normalization work, not vulnerabilities.
- **Worker hot-reload driven by a share recipient.** Sharing a worker's source file with write access is how you let a collaborator change that worker's code and have it deploy live. The redeploy running as the file owner, with the owner's worker credentials, is the intended behavior of that grant — the owner chose to hand out write access to executable code. If you do not want that, do not share the source file with write access.
- **Re-claiming a site or worker name its owner deleted.** A hosted site or worker name that its owner deletes is treated as abandoned by that owner. If one of their apps still points at it, whoever claims the name next serves that app's `index_url` and receives what the app is launched with, and a new app on the same origin may share the old one's identity. Guarding app identities against being recycled this way is ongoing hardening, not a security boundary, so new ways to re-claim an abandoned name, or to inherit an app's tokens or data through one, are not eligible.
- **Share-mode downgrades that do not cascade.** Demoting a delegate does not retroactively revoke the grants that delegate already issued downstream. Shares are independent grants once issued, not a live-evaluated chain. Revoking the downstream grant is a separate, supported action.
- **Metering, credit-hold, and quota inaccuracies.** Credit holds and quota checks are up-front estimates, and actual usage is billed afterwards. Overspend that is bounded and still billed is a correctness bug, including concurrent requests that exceed a hold or quota by some margin; the per-account concurrency and rate-limit gates are the control we rely on. Report these only if you can use a service without it being billed, or overspend without bound.
- **Existence-only oracles.** A response that tells "exists but forbidden" apart from "not found" for a path or ID the caller already supplied, without revealing anything else about it, is hardening.
- **Session and credential lifecycle around 2FA enrollment.** Enrolling in 2FA is not a credential-revocation event: existing authenticated sessions stay valid, and outstanding password-reset links stay valid. Likewise, account-management actions behind the `/user-protected/*` gate are password-gated by design, not TOTP-gated. A recovery code that stops being accepted fails _closed_ — it denies access rather than granting it, so it is a bug report, not a vulnerability.
- **Group permissions** (`/auth/revoke-user-group` and siblings). The group-permission surface is unfinished and not exposed anywhere in the product. Findings there are not reachable by any user.
- **Missing Subresource Integrity (SRI) on external scripts/styles.** Accepted as hardening, not as a vulnerability, absent a demonstrated compromise of the CDN in question.
- **General "token in a URL" / token-lifetime / token-storage designs** — signed directory URLs exposing children, a write signature implying read, app tokens outliving a web session, or the session token being held in `localStorage` on `puter.com`. These are current intended behaviors. Shorter-lived tokens with a refresh flow are on the roadmap; their absence is not a vulnerability on its own. Show a reproducible attack that does not already presuppose script execution on `puter.com`.
- **Missing PKCE, unverified `id_token` signatures, or other OIDC hardening.** Provider tokens are obtained through a server-to-server authorization-code exchange with the provider's token endpoint over TLS, so the resulting `id_token` / userinfo claims are trusted from that channel rather than from local JWKS signature verification — the callback never accepts a caller-supplied `id_token`. Adding local signature / `aud` / `iss` / `exp` checks is welcome defense-in-depth (please open a GitHub issue/PR), but their absence is not an account-takeover vector on its own.
- **JWT "algorithm confusion" / unpinned `algorithms` in `jwt.verify`.** Puter's session tokens are HMAC-signed (HS256) with a server-side secret, and there is no asymmetric public key anywhere in the verification path, so `alg`-substitution attacks do not apply. Explicitly pinning `algorithms` is a fine hardening PR, but it is not a vulnerability.
- **Static-source "SQL injection" in internal pagination.** Findings such as `LIMIT ${limit}` in list/notification queries: the limit is numerically coerced and clamped before it reaches the query, so it is not reachable with attacker-controlled string input. Hardening PRs to the internal stores are welcome, but these are not exploitable as reported.
- **Deprecated `saveTo*` GUI app messages.** The legacy `saveToDesktop` / `saveToDocuments` / etc. app-IPC handlers can create — never overwrite — new files in standard user folders. The behavior is non-destructive, path-traversal-safe, and deprecated (slated for removal); it is not treated as a sandbox escape.
- **Best-practice suggestions** such as login/registration username enumeration (kept intentionally for UX) or unauthenticated unsubscribe links (industry norm).
- **Rate-limiting suggestions for TURN credential issuance** (intentional; not billed per tunnel).
* **XSS / CORS / token issues scoped to `api.puter.com`.** The primary user session cookie lives on `puter.com`, not on the API origin, so reflected/stored XSS, CORS, or token handling on `api.puter.com` is generally out of scope. Two exceptions we *do* evaluate on their merits — please report these: (a) attacker-controlled content served **inline** (e.g. an HTML `Content-Type`) from API file/response endpoints, and (b) anything that abuses the app-scoped `puter_token_v2` companion cookie set on the API host.
* **SSRF via `secureFetch`.** Production routes outbound requests through an isolated proxy that has no access to internal/SSRF-sensitive resources.
* **Attacks that depend on guessing an `appInstanceID` or app UID.** These are random 128-bit secret values and are not considered guessable.
* **Apps invoking drivers, creating workers, or using KV.** Applications are intended to do this; worker permissions are scoped to the owning app. This is by design.
* **App metadata or app user-count "leaks".** This information is currently public by design.
* **General "token in a URL" / token-lifetime designs** — signed directory URLs exposing children, a write signature implying read, or app tokens outliving a web session. These are current intended behaviors.
* **Missing PKCE, unverified `id_token` signatures, or other OIDC hardening.** Provider tokens are obtained through a server-to-server authorization-code exchange with the provider's token endpoint over TLS, so the resulting `id_token` / userinfo claims are trusted from that channel rather than from local JWKS signature verification — the callback never accepts a caller-supplied `id_token`. Adding local signature / `aud` / `iss` / `exp` checks is welcome defense-in-depth (please open a GitHub issue/PR), but their absence is not an account-takeover vector on its own.
* **JWT "algorithm confusion" / unpinned `algorithms` in `jwt.verify`.** Puter's session tokens are HMAC-signed (HS256) with a server-side secret, and there is no asymmetric public key anywhere in the verification path, so `alg`-substitution attacks do not apply. Explicitly pinning `algorithms` is a fine hardening PR, but it is not a vulnerability.
* **Static-source "SQL injection" in internal pagination.** Findings such as `LIMIT ${limit}` in list/notification queries: the limit is numerically coerced and clamped before it reaches the query, so it is not reachable with attacker-controlled string input. Hardening PRs to the internal stores are welcome, but these are not exploitable as reported.
* **Deprecated `saveTo*` GUI app messages.** The legacy `saveToDesktop` / `saveToDocuments` / etc. app-IPC handlers can create — never overwrite — new files in standard user folders. The behavior is non-destructive, path-traversal-safe, and deprecated (slated for removal); it is not treated as a sandbox escape.
* **Best-practice suggestions** such as login/registration username enumeration (kept intentionally for UX) or unauthenticated unsubscribe links (industry norm).
* **Rate-limiting suggestions for TURN credential issuance** (intentional; not billed per tunnel).
If you believe you have a **genuinely new** exploit chain that defeats one of these rationales (for example, demonstrating a sensitive credential that really is reachable on `api.puter.com`), say so explicitly and show why the reasoning above does not apply.
@@ -63,43 +40,37 @@ If you believe you have a **genuinely new** exploit chain that defeats one of th
To participate, you must:
1. **Report responsibly**: Provide detailed steps to reproduce the issue, including proof-of-concept code, screenshots, or a screen recording (see _Proof of Reproduction_ below).
1. **Report responsibly**: Provide detailed steps to reproduce the issue, including proof-of-concept code, screenshots, or a screen recording (see *Proof of Reproduction* below).
2. **Do no harm**: Do not exfiltrate, modify, or delete data. Only access your own account or test data.
3. **Respect availability**: Do not perform denial-of-service attacks or automated scans that degrade service.
4. **Follow disclosure policy**: Do not publicly disclose vulnerabilities until we have confirmed and patched the issue.
5. **Act in good faith**: Make every effort to avoid privacy violations, destruction of data, and interruption or degradation of services.
6. **Check Eligibility and Known Non-Issues**: Reports that fall outside "Eligibility" or match an item in "Known Non-Issues" above are not eligible and will be closed.
6. **Check the Known Non-Issues list**: Reports matching an item in the "Known Non-Issues" section above are not eligible and will be closed.
Reports that do not meet these guidelines may not be eligible for a reward.
## Proof of Reproduction
Reports **MUST** demonstrate a **working, reproducible exploit with real impact** — not a theoretical or static-source-review finding. Please include:
Reports must demonstrate a **working, reproducible exploit with real impact** — not a theoretical or static-source-review finding. Please include:
- Exact steps to reproduce, the relevant request/response or code path, and the commit or version you tested.
- The **observed** result versus the **expected** result.
- For client-side, UI, or authentication-flow bugs: a short screen recording (≤ 2 minutes) showing the exploit working end-to-end on a real Puter instance.
- For server-side bugs: a runnable proof-of-concept and screenshot or video showing the attack process. A unit test or in-process harness alone does not count; show it against a running instance.
* Exact steps to reproduce, the relevant request/response or code path, and the commit or version you tested.
* The **observed** result versus the **expected** result.
* For client-side, UI, or authentication-flow bugs: a short screen recording (≤ 2 minutes) showing the exploit working end-to-end on a real Puter instance.
* For server-side bugs: a runnable proof-of-concept.
Reports based solely on reading the source ("source review only, not tested") or unverified AI/LLM-generated reports are the **lowest triage priority and not eligible for bounty**.
If you used an AI tool to help find an issue, you must personally verify that it actually reproduces before submitting. Reports sent by automated agents are not eligible.
Reports based solely on reading the source ("source review only, not tested") or unverified AI/LLM-generated reports are the **lowest triage priority and are generally not eligible**. If you used an AI tool to help find an issue, you must personally verify that it actually reproduces before submitting.
Please submit **one issue per report**.
Bundled "audit packs" of many speculative findings will not be eligible for bounty; send each confirmed issue separately.
Additionally please **DO NOT** submit too many issues at once; these will be treated as a bundled report, and will not be eligible for bounty.
Please submit **one issue per report**. Bundled "audit packs" of many speculative findings will be declined; send each confirmed issue separately.
## Reporting Process
To report a vulnerability, email us at: **[[email protected]](mailto:[email protected])**.
Include:
- A description of the vulnerability
- Steps to reproduce
- Potential impact
- Suggested remediation (if available)
- Paypal id where we can send the bounty (if eligible)
Send the full report in your first email. We don't reply to messages that only ask whether we run a bounty or where to report.
* A description of the vulnerability
* Steps to reproduce
* Potential impact
* Suggested remediation (if available)
We aim to acknowledge receipt within **72 hours** and provide a resolution timeline.
@@ -107,13 +78,13 @@ We aim to acknowledge receipt within **72 hours** and provide a resolution timel
We offer monetary rewards based on the severity of the vulnerability, as determined by our internal assessment (using CVSS as a guide).
- **Critical: \$1,000 – \$2,000**
- **High: \$500 – \$1,000**
- **Medium: \$200 – \$500**
- **Low: \$50 – \$100**
* **Critical: \$1,000 – \$2,000**
* **High: \$500 – \$1,000**
* **Medium: \$200 – \$500**
* **Low: \$50 – \$100**
Non-security issues, suggestions, and best practices feedback are always welcome, but may not qualify for a reward.
If multiple researchers report the same issue, the bounty will be awarded to the first eligible report we receive. Issues we already know about, from internal review, an open ticket, or a fix already in a public commit or PR, count as duplicates.
If multiple researchers report the same issue, the bounty will be awarded to the first eligible report we receive.
## Payments Disclaimer
+1 -1
View File
@@ -1 +1 @@
FOLLOW @AGENTS.md
FOLLOW ./AGENTS.md
+34 -11
View File
@@ -1,21 +1,44 @@
# Contributing to Puter
# Contributing to Puter Backend
Thanks for contributing. These rules aren't strictly enforced, but following them makes every PR easier. If anything's unclear, ping a core maintainer or open the PR and ask.
Thanks for contributing. These rules aren't strictly enforced — but following them makes every PR easier. If anything's unclear, ping a core maintainer or open the PR and ask.
- New to the backend? Start with [doc/architecture.md](doc/architecture.md).
- Code, test, and docs conventions for the whole repo live in [AGENTS.md](AGENTS.md). It's written for AI agents; the conventions apply to human contributors too.
- Adding or changing a public API (an endpoint, driver method, or puter.js method)? Follow [doc/contributing-apis.md](doc/contributing-apis.md): backward compatibility, [developer docs](src/docs/), types, and tests all move in the same PR.
- Found a vulnerability? Report it privately per [SECURITY.md](SECURITY.md).
New to the backend? Start with [doc/architecture.md](doc/architecture.md).
---
## Before you open a PR
## 1. Test it. Run it.
Check your diff against these sections of AGENTS.md:
Run the affected code path end-to-end before opening a PR. "It builds" is not "it works."
- [Working rules of thumb](AGENTS.md#working-rules-of-thumb): run it end to end, test new behavior and bug fixes, understand every line you commit (AI assistance is fine), leave touched files a bit better, and publish limit changes.
- [Reuse and abstraction](AGENTS.md#reuse-and-abstraction) and [hot paths](AGENTS.md#hot-paths).
- [Security & privacy](AGENTS.md#security--privacy): expose nothing the caller doesn't need.
Add tests for new behavior, endpoints, or bug fixes. If something's genuinely hard to test, say so in the PR.
## 2. Follow existing patterns
Match the shape of similar code already in the repo. [doc/architecture.md](doc/architecture.md) is the source of truth for layers, wiring, and naming. If you think a pattern is wrong, raise it — don't quietly diverge.
In plain-JS files, typing is encouraged via JSDoc `@type` annotations using the TypeScript type system, with `@typedef` for shared shapes. Don't type API surfaces as `unknown` or untyped `...args` unless the values are passed through transparently to an upstream layer that owns their type.
## 3. Don't expose system or user information
Scan your diff for stray logs, debug routes, internal paths, secrets, tokens, or user data in errors/responses. When in doubt, return less. Flag any auth, permission, or data-export changes in the PR description.
For private security reports, see [SECURITY.md](SECURITY.md).
## 4. AI-assisted code is fine — understood code is required
Don't commit code you couldn't have written, debugged, or defended yourself. Read the diff, run it, and be ready to explain it in review.
## 5. Adding or changing APIs
If you add or change a public API (an endpoint, driver method, or puter-js method), follow [doc/contributing-apis.md](doc/contributing-apis.md) — backward compatibility, [developer docs](src/docs/), types, and tests all move in the same PR.
## 6. Boy Scout Rule — leave it 1% better
![Boy Scout Rule](https://imgs.search.brave.com/DMmIWl5-NuZVtrR9kXBb06AKF8kturkgSW9UMb2-6m4/rs:fit:860:0:0:0/g:ce/aHR0cHM6Ly9sYXdz/b2Zzb2Z0d2FyZWVu/Z2luZWVyaW5nLmNv/bS9pbWFnZXMvbGF3/cy9ib3ktc2NvdXQt/cnVsZS5wbmc)
Fix the typo, the dead import, the missing test, the bit you had to read twice. Keep cleanup proportional to the change — no refactors riding along on bug fixes.
---
## Opening a PR
+3 -11
View File
@@ -14,7 +14,7 @@
# config.default.json, so partial overrides work. Absent file = defaults.
# ---- Build stage ----
FROM node:26-bookworm-slim AS build
FROM node:24-slim AS build
WORKDIR /opt/puter
@@ -59,7 +59,7 @@ RUN set -e; \
wait $pjs_pid
# ---- Runtime stage (slim — no build tools) ----
FROM node:26-bookworm-slim
FROM node:24-slim
WORKDIR /opt/puter
@@ -73,24 +73,16 @@ COPY --from=build --chown=node:node /opt/puter .
RUN mkdir -p /etc/puter /var/puter && \
chown -R node:node /etc/puter /var/puter
# Fix outdated HOME_URL in /etc/os-release (see issue #4329)
# The base image may contain a reference to the old phoenix repository.
RUN if [ -f /etc/os-release ]; then \
sed -i 's|HOME_URL="https://github.com/HeyPuter/phoenix"|HOME_URL="https://github.com/HeyPuter/puter"|' /etc/os-release; \
fi
# Self-hosters mount their override at this exact path. The v2 loader
# deep-merges it over config.default.json (see backend/index.ts).
ENV PUTER_CONFIG_PATH=/etc/puter/config.json
ENV NODE_OPTIONS=--enable-source-maps
EXPOSE 4100
# The inbound SMTP receiver, when run from this image with the `smtp` profile.
EXPOSE 2525
USER node
HEALTHCHECK --interval=30s --timeout=3s --start-period=30s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://puter.localhost:4100/test || exit 1
CMD ["node", "-r", "./dist/src/backend/telemetry.js", "./dist/src/backend/index.js"]
CMD ["node", "-r", "./dist/src/backend/telemetry.js", "./dist/src/backend/index.js"]
+7 -6
View File
@@ -10,10 +10,12 @@
·
<a href="https://apps.puter.com/">App Store</a>
·
<a href="https://builder.puter.com/">AI Builder</a>
·
<a href="https://developer.puter.com" target="_blank">Developers</a>
·
<a href="https://discord.com/invite/PQcx7Teh8u">Discord</a>
·
<a href="https://reddit.com/r/puter">Reddit</a>
·
<a href="https://twitter.com/HeyPuter">X</a>
</p>
@@ -48,8 +50,6 @@ npm start
```
**→** This should launch Puter at http://puter.localhost:4100
To run this checkout with Docker, follow [Building from source](./doc/self-hosting.md#building-from-source-instead-of-pulling). Create a local `docker-compose.override.yml` to select the local build; keeping these settings out of `docker-compose.yml` avoids conflicts when pulling updates and keeps local configuration out of pull requests.
<br/>
@@ -82,7 +82,10 @@ Puter is available as a hosted service at [**puter.com**](https://puter.com).
Connect with the maintainers and community through these channels:
- Bug report or feature request? Please [open an issue](https://github.com/HeyPuter/puter/issues/new/choose).
- Discord: [discord.com/invite/PQcx7Teh8u](https://discord.com/invite/PQcx7Teh8u)
- X (Twitter): [x.com/HeyPuter](https://x.com/HeyPuter)
- Reddit: [reddit.com/r/puter/](https://www.reddit.com/r/puter/)
- Mastodon: [mastodon.social/@puter](https://mastodon.social/@puter)
- Security issues or abuse reports? [[email protected]](mailto:[email protected])
- Email maintainers at [[email protected]](mailto:[email protected])
@@ -117,9 +120,7 @@ This repository, including all its contents, sub-projects, modules, and componen
- [Korean / 한국어](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.ko.md)
- [Malay / Bahasa Malaysia](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.my.md)
- [Malayalam / മലയാളം](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.ml.md)
- [Dutch / Nederlands](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.nl.md)
- [Polish / Polski](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.pl.md)
- [Odia / ଓଡ଼ିଆ](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.od.md)
- [Portuguese / Português](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.pt.md)
- [Punjabi / ਪੰਜਾਬੀ](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.pa.md)
- [Romanian / Română](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.ro.md)
-4
View File
@@ -22,10 +22,6 @@
"default_temp_group": "b7220104-7905-4985-b996-649fdcdb3c8f",
"storage_capacity": 104857600,
"disable_user_signup": false,
"teams_enabled": false,
"max_teams_per_user": 1,
"max_seats_per_team_free": 3,
"max_seats_per_team_paid": 40,
"strict_email_verification_required": false,
"gui_assets_root": "./src/gui",
"puterjs_root": "./src/puter-js/dist",
+7 -158
View File
@@ -7,8 +7,9 @@
// Each setting lives at exactly one canonical key — there are no fallback
// aliases. Values shown are illustrative, not production secrets.
//
// Keys consumed only by closed-source / hosted-prod extensions are
// intentionally omitted.
// Keys consumed only by closed-source / hosted-prod extensions (clickhouse,
// cacheUpdateHandler, pages, prodMeteringAndBilling, …) are intentionally
// omitted.
// ── Environment / identity ──────────────────────────────────────────
"config_name": "template",
@@ -31,10 +32,6 @@
"port": 4100,
// Externally-visible port (set this when behind a reverse proxy on 80/443).
"pub_port": 4100,
// Idle keep-alive timeout in ms. Must stay above the idle timeout of any
// proxy in front, or the proxy reuses connections this server has closed
// and its clients see 502s.
"keep_alive_timeout": 920000,
"protocol": "http",
"domain": "puter.localhost",
// Fully-qualified externally-visible URL. Computed from protocol/domain/
@@ -62,10 +59,6 @@
// direct socket peer. NEVER set to `true` in prod — it trusts every hop
// and makes X-Forwarded-For forgeable.
"trust_proxy": false,
// Request header a TLS-terminating proxy in front of the server sets to
// the client's TLS fingerprint (JA3/JA4). Only set it when that proxy
// overwrites any value the client sends. Unset disables it.
// "tlsFingerprintHeader": "x-tls-fingerprint",
"no_browser_launch": false,
// ── Dev watcher (devWatcher extension) ──────────────────────────────
@@ -219,9 +212,7 @@
// "bootstrapTables": true,
"path": "",
// Credentials. NOTE: snake_case here, unlike `s3.s3Config` below.
// For dynamodb-local, any non-empty values work. On AWS,
// `"useCredentialChain": true` uses the default credential chain
// instead of keys.
// For dynamodb-local, any non-empty values work.
"aws": {
"access_key": "",
"secret_key": "",
@@ -258,116 +249,8 @@
"user": "",
"pass": ""
}
// Each send names a stream: "auth", "signup" or "notify". A stream
// listed here adds its headers to every message on it, and its "from"
// when the sender gives none. Tags are written to "tagsHeader".
// "streams": {
// "notify": {
// "from": "\"Puter\" <[email protected]>",
// "headers": { "X-Example-Stream": "notify" },
// "tagsHeader": "X-Example-Tags"
// }
// }
},
// ── Inbound mail (separate process) ─────────────────────────────────
// Mailboxes addressed at one of your own domains. `secret` gates the
// ingress endpoint that files a message into the recipient's ~/.mail.
//
// `localServer` additionally runs an SMTP listener that accepts mail from
// other mail servers and hands each message to that endpoint. It runs as
// its own process - `npm run start:smtp`, or the `smtp` compose profile -
// and stores nothing itself. Without it, nothing delivers to the endpoint
// unless you point something else at it.
//
// The listener is unauthenticated by nature: anything on the internet may
// hand it a message for an address in `localDomains`. It performs no spam,
// SPF, DKIM or DMARC checks. Read the self-hosting guide before exposing
// it.
// "userEmail": {
// "secret": "a long random string",
// "localServer": true,
// "localDomains": ["example.com"],
// // Container-internal by default; publish 25 onto it from outside.
// "localPort": 2525,
// // Defaults to <api_base_url>/email/ingress. The endpoint is served
// // on the `api` subdomain, so when this points somewhere internal,
// // `localIngressHost` has to name that virtual host.
// "localIngressUrl": "http://puter:4100/email/ingress",
// "localIngressHost": "api.example.com",
// "localHostname": "mx.example.com",
// "localMaxRecipients": 50,
// // Concurrent connections. Each message is held in memory while it
// // is delivered, so this also bounds memory use.
// "localMaxClients": 20
// },
// ── Sharing ─────────────────────────────────────────────────────────
// Email a recipient who already has an account about a new share. On
// unless set to false; they opt out via the unsubscribe link, or by
// blocking a sender. An invite to an address with no account always goes.
// "share_email_notifications": false,
//
// New shares one account may create per UTC day. Default 200.
// "share_daily_limit": 200,
//
// How often a share may interrupt its recipient — the notification pushed
// to their screen and the email with it. Over budget the share still
// succeeds and their notification is still brought up to date; only the
// interruption is dropped. The recipient bounds are what stop many senders
// burying one person between them. A non-positive value removes a bound.
// "share_notify_limits": {
// "pairWindowSeconds": 900,
// "pairDaily": 20,
// "recipientHourly": 10,
// "recipientDaily": 50,
// "emailBatchSeconds": 90
// },
// ── Teams / teams ──────────────────────────────────────────────
// Off by default, and off is the state every existing install stays in.
// The tables ship either way and sit inert; this gates whether `/teams`
// is registered at all, so with it off the paths do not exist rather than
// existing and refusing. It is also the backout: turning it off removes
// the feature without touching data. Teams ships as an extension; on,
// the server refuses to boot unless one registers it.
"teams_enabled": false,
//
// Live teams one user may own. Soft-deleted ones do not count, so
// deleting yours frees the slot. Default 1.
"max_teams_per_user": 1,
//
// Seats one team may provision. Each seat is a real Puter account taking a
// name from the global username pool, so this is what bounds a team's blast
// radius. The owner's plan decides which of the two applies; defaults are 3
// and 40. A plan registered with `teamSeatCap` uses that instead. Lowering
// a cap below a team's current seat count blocks new provisioning and
// disables nobody. Deployments upgrading from the old flat default of 50:
// without a billing extension every owner reads as free, so set
// `max_seats_per_team` to keep the old behavior.
"max_seats_per_team_free": 3,
"max_seats_per_team_paid": 40,
//
// One flat cap for every team, whatever the owner pays. Setting it
// overrides both of the above, so a deployment that does not sell seats can
// ignore plans entirely. Unset by default — leave it that way to keep the
// free cap meaningful.
// "max_seats_per_team": 50,
//
// Staged rollout: only these email domains may create a team, and the tab
// renders for them even with `gui_params.teams_ui` off. Members of an
// existing team always pass. Unset means everyone, gated by `teams_ui`.
// "teams_allowed_email_domains": ["puter.com"],
// ── Notifications ───────────────────────────────────────────────────
// How long a notification is kept, in days from creation. Acknowledged or
// not, a row past this is swept. Off by default; any value above 0 arms
// the sweep.
// "notificationRetentionDays": 14,
//
// Ms a live notification waits for open tabs to acknowledge it.
// "notificationAckTimeoutMs": 5000,
// ── Alarms / alerting ───────────────────────────────────────────────
// Where system alarms go. Severity is the routing decision — each
// transport takes everything at or above its own `minSeverity`:
@@ -441,17 +324,11 @@
"apiURL": ""
},
"gemini": { "apiKey": "" },
// Meta Model API (Muse Spark). `muse-spark-1.2-contributor`
// trades a ~12x discount for Meta training on its prompts and
// completions, so it is only served to callers that name it.
"meta": { "apiKey": "" },
"groq": { "apiKey": "" },
"deepseek": { "apiKey": "" },
"mistral": { "apiKey": "" },
"xai": { "apiKey": "" },
"moonshot": { "apiKey": "" },
// MiniMax. One key powers chat and video generation (H3, Hailuo);
// video calls go to the same host, beside the chat API's `/v1`.
"minimax": {
"apiKey": "",
"apiBaseUrl": "https://api.minimax.io/v1"
@@ -464,24 +341,6 @@
"apiKey": "",
"apiBaseUrl": "https://llm.onerouter.pro/v1"
},
// BytePlus ModelArk. One key powers chat (Seed/GLM/DeepSeek), image
// generation (Seedream) and video generation (Seedance); `apiBaseUrl`
// selects the region; see
// https://docs.byteplus.com/en/docs/ModelArk/1330310 for options.
"byteplus": {
"apiKey": "",
"apiBaseUrl": "https://ark.ap-southeast.bytepluses.com/api/v3"
},
// Kling AI. One API key (not the legacy AccessKey/SecretKey pair)
// powers video generation, image generation and TTS; `apiBaseUrl`
// defaults to https://api-singapore.klingai.com.
"kling": { "apiKey": "" },
"zai": { "apiKey": "" },
"alibaba": { "apiKey": "" },
"hoonify": {
"apiKey": "",
"apiBaseUrl": "https://api.hoonify.ai/v1"
},
"together-ai": { "apiKey": "" },
// Local Ollama. `enabled: false` skips the auto-probe at startup
// (otherwise Puter logs ECONNREFUSED on every boot when no Ollama
@@ -503,6 +362,7 @@
"xai-image-generation": { "apiKey": "" },
// ─ Video generation ─
"openai-video-generation": { "apiKey": "" },
"together-video-generation": { "apiKey": "" },
"gemini-video-generation": { "apiKey": "" },
@@ -514,8 +374,6 @@
"defaultVoiceId": "",
"speechToSpeechModelId": ""
},
// Both AWS providers also take `"useCredentialChain": true` in place
// of keys, like `dynamo.aws`.
"aws-polly": {
"access_key": "",
"secret_key": "",
@@ -590,11 +448,8 @@
// operations that spend it — file transfers, KV calls. Recording is
// unaffected either way. `workers` extends the same refusal to
// worker-driven calls, which are exempt by default because a deployed
// worker has nowhere to surface a payment prompt. `subscriptions` covers
// the separate question of surfaces reserved for paid plans (the
// vendor-compatible AI endpoints): turn it off on a deployment with no
// paid plans, or every account is refused them.
// "meteringEnforcement": { "enabled": true, "workers": false, "subscriptions": true },
// worker has nowhere to surface a payment prompt.
// "meteringEnforcement": { "enabled": true, "workers": false },
// Fleet-wide spend rate, in micro-cents per minute, past which metering
// raises the `metering:excessiveGlobalUsageRate` alarm. Omit it (the
@@ -602,10 +457,4 @@
// what this deployment's normal traffic costs, so it has to be measured
// rather than guessed, and a stale number here alarms on healthy growth.
// "maxGlobalUsagePerMinute": 200000000
// How many full months past its own a monthly usage record is kept before
// it expires. Defaults to 3; set to 0 to keep every month's records
// forever. Changing it, including to 0, applies to records first written
// after the change; records already stamped keep their expiry.
// "meteringRetentionMonths": 3
}
+5 -31
View File
@@ -36,24 +36,15 @@ pass one explicitly unless you really mean "page someone".
### Choosing one
- Did the server fail to do its job in a way nobody expected? → `critical`
- Is a background job, rate, or dependency degraded, or has an account spent
far past its credits? → `warning`
- Is this a user doing something notable (tripping an abuse heuristic)?
→ `info`
- Is a background job, rate, or dependency degraded? → `warning`
- Is this a user doing something notable (tripping an abuse heuristic,
overspending)? → `info`
- Did one of *our own* limits reject a caller — a rate limit, a concurrency
cap, a quota? → don't alarm at all. The limit doing its job is not an
event; the 429 is the whole signal, and alarming on it only produces noise
proportional to traffic. An *upstream provider* rate-limiting us is the
opposite case and still alarms (`upstream_rate_limited`, `info`) — that one
is not something we chose. The exception is a free model: nothing is billed,
nothing is actionable, and the throttling is the price of the model, so the
AI chat driver marks those `noAlarm` and only paid models still record.
`noAlarm` is the code-level mute: an `HttpError` carrying it skips the
terminal gate entirely, no matter its status or code. It is for failures the
call site *already knows* are expected and traffic-proportional — reach for it
there, not as a way to quiet an alarm you haven't diagnosed
(`severityOverrides` below is the knob for that).
is not something we chose.
An extension whose signals are all one tier can default its own local
`raiseAlarm` helper to that tier instead of repeating it at every call site —
@@ -79,11 +70,6 @@ this.clients.alarm.create(alarmId, message, fields, 'critical', {
The HTTP error handler uses it: its id is route + error signature, so a hot
loop of the same crash is one incident with N occurrences instead of N pages.
Dedup and Slack's repeat throttle mean a responder sees only the latest
occurrence, so the handler attaches the `HttpError`'s `fields` to the alarm as
`details` — put what a responder needs (which upstreams failed, and how)
there. Everything in `fields` is also returned to the client in the response
body, so it has to be safe to show the caller.
Reach for it anywhere else only when the id is that specific — otherwise a
per-request alarm can flood the pager.
@@ -127,19 +113,7 @@ config always has the last word.
### Repeat throttling
Two levels, and they stack.
The client itself backs off before a repeat is reported at all: every
occurrence is reported up to a small burst, then one per interval. A fault
recurring hundreds of times a second still counts every occurrence, but it
does not write a log line or build an alert payload for each one.
The count lasts only as long as the client remembers the alarm. One that
hasn't recurred for 24 hours is forgotten, and the client holds at most 1000
at once, dropping the least recently raised first. Either way the next
occurrence is a new alarm, with its count and burst starting over.
Below that, the chat transport won't repost the same alarm id within
The chat transport won't repost the same alarm id within
`repeatThrottleMs` (default 15 minutes). The first occurrence always posts,
and the next one that gets through reports how many piled up in between —
so a hot loop reads as one message with a count, not a wall of them.
+16 -27
View File
@@ -1,6 +1,6 @@
# Backend Architecture
Loosely inspired by the Controller–Service–Repository pattern with dependency injection. The backend is a stack of layers; each depends only on the layers beneath it and receives them through its constructor from `PuterServer` ([src/backend/server.ts](../src/backend/server.ts)), which instantiates the layers in order. The rules for working within the layers are in [AGENTS.md](../AGENTS.md#backend).
Loosely inspired by the Controller–Service–Repository pattern with dependency injection. The backend is organized as a stack of layers where each layer only depends on the layers beneath it, and `PuterServer` ([src/backend/server.ts](../src/backend/server.ts)) instantiates each layer in order and hands the instances down to the next.
## Layers
@@ -31,6 +31,8 @@ block-beta
style CFG fill:#334155,stroke:#1e293b,color:#fff
```
Each layer only depends on the layers beneath it, and every dependency is injected through the constructor by `PuterServer`. Extensions sit alongside this stack and can register into any layer — see [Extensions](#extensions) below.
| Layer | Lives in | Responsibility |
| --- | --- | --- |
| **Controllers** | [src/backend/controllers/](../src/backend/controllers/) | Route handlers. Parse + validate input, apply per-route gates (auth, subdomain, rate limit, body parsers — see `RouteOptions`), call into services, format responses. |
@@ -40,7 +42,7 @@ block-beta
| **Clients** | [src/backend/clients/](../src/backend/clients/) | Adapters for external/internal services (sql, redis, s3, dynamodb, email, event bus, …). Knows protocols, not domain concepts. |
| **Config** | `config.*.json` → `IConfig` | The flat, typed config object every layer receives at construction. |
Extensions sit alongside this stack and can register into any layer; see [Extensions](#extensions).
Each layer receives the layers beneath it through its constructor, so dependencies are explicit and traceable from `PuterServer`. A controller does not reach into a client directly; if it needs one, the right move is usually a service.
## Entry point: `PuterServer`
@@ -53,25 +55,17 @@ Extensions sit alongside this stack and can register into any layer; see [Extens
## Context (ALS)
[`Context`](../src/backend/core/context.ts), backed by `AsyncLocalStorage`, carries per-request state without threading it through every function signature. The request-context middleware opens a scope per request after the auth probe runs, so anything inside a request handler can call `Context.get('actor')` / `Context.get('req')`. It's used sparingly, mostly for `actor` and `req`; see the [Context rule](../AGENTS.md#backend).
We use [`Context`](../src/backend/core/context.ts) — backed by `AsyncLocalStorage` — to carry per-request state without threading it through every function signature. It is used **sparingly**, mostly for `actor` and `req`. The request-context middleware opens a scope per request after the auth probe runs; anything inside a request handler can call `Context.get('actor')` / `Context.get('req')` instead of plumbing it as an argument.
## Actors
An [`Actor`](../src/backend/core/actor.ts) is who a request is acting as. Its two app fields are not interchangeable: `app` is the app the actor carries *directly* (an app-under-user token), while `effectiveApp` is the app it ultimately acts as — its own, or the one that issued its access token. An access-token actor has no `app` of its own, so a gate that reads `app` answers "no app" for it and falls open.
`makeActor` resolves `effectiveApp` once at construction, and `assertResolvedActor` at the request edge rejects any actor that skipped it — so reading `effectiveApp` needs no fallback. Reach for `app` only where the direct app genuinely is the question, and say why in a comment.
- `actor.effectiveApp` — which app is acting. This is the default; nearly every scoping rule, namespace key and attribution wants it.
- `isAppActor(actor)` — a direct app actor specifically. Use it where a token must *not* be treated as its issuer, e.g. the permission scanners that read an app's own grant rows.
- `isPlainUserActor(actor)` — the account acting through nothing at all; `isAccountContext(actor)` for the wider "the account's own reach", which also admits a full-access token.
Prefer explicit arguments. Reach for `Context` only when the value is truly request-scoped and would otherwise need to thread through many layers.
## Extensions
Extensions live in [extensions/](../extensions/), parallel the layered stack, and hold the parts of the system Puter still works without. [AGENTS.md](../AGENTS.md#extensions) has the rule for what belongs in one.
Extensions live alongside core ([packages/puter/extensions/](../extensions/)) and parallel the layered stack. They are meant for **non-crucial parts of the system** — things Puter still works without if removed.
- **Good extensions**: [thumbnails](../extensions/thumbnails.ts), [serverInfo](../extensions/serverInfo.ts), [devWatcher](../extensions/devWatcher.ts) — opt-in features cleanly bolted on.
- **Should probably be core**: [metering](../extensions/metering.ts), [appTelemetry](../extensions/appTelemetry.ts) — clients now expect these to be present, so the "extension" framing is misleading.
- **Shouldn't have been an extension**: [whoami](../extensions/whoami.ts) — it's load-bearing for every authenticated client.
- **Shouldn't have been an extension**: [whoami](../extensions/whoami.ts) — it's load-bearing for every authenticated client. Keep this one in mind as a cautionary example when deciding whether something belongs in an extension.
### Extension API
@@ -94,22 +88,17 @@ import { extension } from '@heyputer/backend/src/extensions';
const services = extension.import('service');
extension.get('/healthcheck/deep', { subdomain: 'api', adminOnly: true }, async (_req, res) => {
res.json(await services.health.getStatus());
res.json({ ok: await services.health.runDeepCheck() });
});
extension.on('puter.signup.success', (_key, data) => {
console.log('new user', data.username);
extension.on('user.signup', (_key, data) => {
console.log('new user', data.user.username);
});
```
The `app.recommended` event lets extensions customize desktop recommendations.
Its `appNames` array starts as a fresh copy of the default ordered names on each
call. Listeners may mutate or replace it, including setting it to an empty array.
Async listeners are awaited before names are resolved to app summaries; names
that do not exist are skipped.
## Conventions
```ts
extension.on('app.recommended', (_key, data) => {
data.appNames = ['editor', 'camera'];
});
```
- **TypeScript preferred** in new code where feasible. Existing JS is fine; convert opportunistically when you're already touching a file.
- **`camelCase`** for variable/function names; **`PascalCase`** for classes and for files that contain a class (`AuthService.ts`, `KVStoreDriver.ts`).
- **Deduplicate**. If two services need the same logic, lift it into a util/helper rather than calling sideways across the same layer — services should not depend on other services for code reuse.
- **Don't reach across layers.** Controllers do not poke clients directly; services do not register routes. If you find yourself wanting to, that's usually a signal the abstraction is wrong.
+27 -21
View File
@@ -12,7 +12,21 @@ Every change is backward compatible unless a maintainer has explicitly agreed to
## Core or extension?
The first decision is where the API lives. If **nothing in core will call it, prefer an extension** over wiring it into core. [AGENTS.md](../AGENTS.md#extensions) has the rule and its test; [architecture.md](architecture.md#extension-api) documents the `extension` API.
The first decision is where the API lives.
If the API is **not crucial to core functionality — nothing in core will call it — prefer an extension** over wiring it into core. Extensions live in [extensions/](../extensions/), parallel the core layered stack, and reach core through `extension.import(...)`:
```js
const services = extension.import('service');
const stores = extension.import('store');
extension.registerDriver('myFeature', MyFeatureDriver); // first-class driver
extension.post('/my-feature/frobnicate', opts, handler); // or plain routes
```
Follow the same layered structure inside the extension (driver/controller → service → store) unless it genuinely only needs a couple of route handlers — then the lightweight `extension.get/post/...` helpers are enough on their own.
The test is the direction of dependency: Puter must still work with the extension removed. The moment core needs to call your API, it belongs in core — see [whoami](../extensions/whoami.ts) for the cautionary example of a load-bearing "extension".
## Adding a new API
@@ -21,13 +35,13 @@ Work through all seven steps; the PR is complete when every one is.
### 1. Design the surface first
- Sketch the signature, options, return shape, and error cases before writing code. Find the two or three most similar existing APIs and match their conventions.
- New parameter and field names follow the [naming rule](../AGENTS.md#language--files) (`camelCase`).
- New parameter and field names are `camelCase`. (Existing `snake_case` names stay where they already exist.)
- Anything returning a list follows the [pagination convention](pagination.md): `limit`/`cursor` in, `{ items, cursor, total }` envelope out.
- Prefer an options object over a growing list of positional parameters, but keep the common case callable with a single argument where siblings do.
### 2. Backend
Follow the layered stack ([architecture.md](architecture.md), [layer rules](../AGENTS.md#backend)): a controller or driver at the edge, business logic in a service, persistence in a store. Return exactly what the caller needs and no more — every response field you ship is permanent — and use stable `snake_case` error codes.
Follow the layered stack ([architecture.md](architecture.md)): a controller or driver at the edge, business logic in a service, persistence in a store. The edge parses and validates input and applies gates; services assume the caller is already authorized. Return exactly what the caller needs and no more — every response field you ship is permanent — and use stable `snake_case` error codes.
#### Controller or driver?
@@ -39,17 +53,8 @@ Both are supported ways to define an API. **Prefer a controller when you need fi
#### Middleware and gates
- **Controller routes** (and extension routes — same options) take [`RouteOptions`](../src/backend/core/http/types.ts): auth gates (`requireAuth`, `requireUserActor`, `noUserSession`, `adminOnly`, `allowedAppIds`, and the access-token controls), `subdomain` routing, per-route `rateLimit`, body parsers, and arbitrary extra `middleware`. The auth flavors are subtle and default-deny — read the JSDoc on each field before picking.
- **Driver methods** get their policies from the `@Driver` options: per-method `rateLimit` (limit/window/backend), `concurrent` in-flight caps (optionally `bySubscription`), `requireSubscription`, `requireReputation`, and `noUserSession`. The `/drivers/call` surface enforces them.
- **Paying accounts only:** `requireSubscription`, as a route option on a controller endpoint or in the per-method `@Driver({ requireSubscription })` block on a driver (`/drivers/call` is one shared route, so a driver's requirement can't live in route options). `true` accepts any plan that isn't free, so a plan an extension registers counts without core naming it; an array of policy ids (`['business', 'pro']`) accepts only those; `false` or leaving it out means no requirement. An empty array is a boot error, since it would read as subscribers-only while admitting everyone. The check is a map lookup on the metering service's cached per-actor subscription. It asks which plan the account is on; `requireCredits` asks whether it has budget left. Deployments with no paid plans (self-hosted installs) turn it off with `meteringEnforcement.subscriptions: false`.
- **Trusted-enough accounts only:** `requireReputation`, as a route option or in the per-method `@Driver({ requireReputation })` block. Opt-in; implies `requireAuth`. The value names a tier, and `reputationGate.tiers` in config sets the score that tier takes, so a deployment can retune it without editing the surface. A tier the running config doesn't define is inert and everyone passes, so an install that doesn't score its accounts never turns traffic away on a score it never computed. `reputationGate.enabled: false` stops every declared gate at once. Denials are a bare 403 `reputation_required` that names neither the score nor the tier. Nothing in the tree declares one yet.
- **A specific verified factor:** `requireVerified` (email, and only under `strict_email_verification_required`), `requirePhoneVerified`, `requireCardVerified`. Opt-in, and the factor must actually have been verified, unlike the default-on gate that turns away accounts still carrying a pending verification the abuse harness asked for. `requireAnyVerified: ['phone', 'card']` is the OR of the last two: any listed factor verified at any point passes, as does a paid plan when `card` is listed (a paying account already has a card on file). Otherwise only the factors the deployment can verify are asked for (an SMS provider configured, a card gate an extension reports on); with none verifiable the gate is inert rather than locking the route on a self-hosted install. A denial carries the first verifiable factor's code plus `factors` (the verifiable ones, in the route's order), so a client can lead with one flow and offer the other. `verifiedFactorGate.enabled: false` in config stops every declared gate at once, e.g. during an SMS-provider outage.
- **Metered spend:** an endpoint that spends metered resources on the caller's behalf (moving file content, object-store requests, anything else the account is billed for) declares `requireCredits: true`, which turns an account with no budget left away with a 402 before the handler runs. Endpoints that only describe or delete things deliberately don't: an account that has run out still has to see what it has, clear it, and reach its billing pages. Drivers have no route options, so they call `assertActorHasCredits` themselves ([src/backend/services/metering/enforcement.ts](../src/backend/services/metering/enforcement.ts)) — see `KVStoreDriver`, which does it once for every method.
Provider sign-in requires the account's authenticator or a recovery code when 2FA is enabled, including signup callbacks that resolve an existing account. The five-minute challenge uses the same single-use `/login/otp` and `/login/recovery-code` verification as password sign-in. The GUI carries it in a URL fragment and removes it before initialization. A session and successful popup-return proof are issued only after verification; cancelling retains the popup's signed retry context without marking it authenticated.
Disabling 2FA through `/user-protected/disable-2fa` requires both password or OIDC revalidation and either the current six-digit authenticator `code` or an unused recovery code in `code`. Recovery codes are consumed atomically. Missing or incorrect codes leave the secret and remaining recovery codes intact. A team's requirement for 2FA still prevents disabling it.
Email changes through `/user-protected/change-email` require the password or OIDC revalidation cookie, plus a six-digit `code` from the authenticator when 2FA is enabled. Missing or malformed codes return `400 bad_request`; incorrect codes return `401 code_mismatch`. Both include `factor: 'otp'`. Confirmation links are bound to that verified request. A 2FA-enabled account must request a new link if its existing link was issued without second-factor verification.
- **Driver methods** get their policies from the `@Driver` options: per-method `rateLimit` (limit/window/backend), `concurrent` in-flight caps (optionally `bySubscription`), and `noUserSession`. The `/drivers/call` surface enforces them.
- **An endpoint that spends metered resources** on the caller's behalf — moving file content, making object-store requests, anything else the account is billed for — also declares `requireCredits: true`, which turns an account with nothing left of its budget away with a 402 before the handler runs. Endpoints that only describe or delete things deliberately don't: an account that has run out still has to be able to see what it has, clear it, and reach its billing pages. Drivers have no route options to declare this on, so they call `assertActorHasCredits` themselves ([src/backend/services/metering/enforcement.ts](../src/backend/services/metering/enforcement.ts)) — see `KVStoreDriver`, which does it once for every method.
### 3. puter.js
@@ -58,32 +63,33 @@ Email changes through `/user-protected/change-email` require the password or OID
### 4. Types
Type the method in JSDoc on the implementation, following the [puter.js type rules](../AGENTS.md#types) (overloads, where shapes live, `index.d.ts`), then run `npm run check:puterjs:types`. Declarations are generated from the JSDoc, so there is nothing to keep in sync by hand.
- Type the method where you wrote it, in JSDoc: `@param`/`@returns` on the implementation, one `@overload` block per accepted call form, and `@typedef {Object}` + `@property` for any new shape. The JSDoc is the source of truth — declarations are generated from it, so there is nothing to keep in sync by hand.
- Put a shape more than one file needs in the module's `types.js` (e.g. [src/puter-js/src/modules/kv/types.js](../src/puter-js/src/modules/kv/types.js)); anything shared across modules goes in [src/puter-js/src/lib/types.js](../src/puter-js/src/lib/types.js). A shape with one consumer can stay next to it.
- Run `npm run check:puterjs:types` — it generates the declarations and type-checks the published surface without `skipLibCheck`. **Never edit anything under `src/puter-js/types/`**: it is gitignored build output, produced by the SDK build and shipped in the npm tarball, and the next build overwrites it.
- Name the new type in [src/puter-js/index.d.ts](../src/puter-js/index.d.ts) if consumers should be able to import it. That file is the one hand-written declaration in the package: it decides what is public and re-exports nothing else.
### 5. Docs
- Add the method page at [src/docs/src/](../src/docs/src/)`<Area>/<method>.md` — frontmatter (`title`, `description`, `platforms`), syntax, parameters, return value, and at least one runnable example — and update the area overview (`<Area>.md`). Copy the structure of an existing page.
- The docs are the contract users code against: signatures, defaults, and return shapes match the implementation exactly.
### 6. Tests
- **Backend:** see [backend tests](../AGENTS.md#backend-tests).
- **SDK:** add cases to [src/puter-js/tests/api/suites/](../src/puter-js/tests/api/suites/)`<area>.suite.ts` (register new suites in `suites/index.ts`; there is no globbing). One suite runs on node, browser, and workerd via `npm run test:puterjs`; never write per-platform tests. The runners execute the built bundle, so run `npm run build:workerLib` first or the suite silently tests stale code.
- **Backend:** colocated Vitest tests; prefer the in-memory test server (`setupPuterTestEnv` in [src/backend/testUtil.ts](../src/backend/testUtil.ts)) over mocking.
- **SDK:** add cases to [src/puter-js/tests/api/suites/](../src/puter-js/tests/api/suites/)`<area>.suite.ts` (register new suites in `suites/index.ts`). One suite runs on node, browser, and workerd via `npm run test:puterjs` — never write per-platform tests, and rebuild first with `npm run build:workerLib` since the runners execute the built bundle.
- **Desktop-rendered UI** (`puter.ui.*`): add a Playwright spec per [src/puter-js/TESTING.md](../src/puter-js/TESTING.md).
### 7. Security pass
Run the [security & privacy](../AGENTS.md#security--privacy) check on the diff, and confirm the auth gates are present.
Scan the diff before opening the PR: no internals leaked in errors or logs, no over-broad responses, auth gates present. Flag anything auth-, permission-, or data-export-related in the PR description.
## Maintaining an existing API
Changes are **additive by default**:
- Existing call signatures keep working, including both positional and options-object forms where a method supports them. Say in the PR how existing callers are unaffected.
- New parameters are optional, with defaults that reproduce the old behavior exactly.
- Never rename, repurpose, or remove existing parameters, response fields, or error codes. Don't change types, ordering guarantees, or which fields are present when.
- New behavior that could surprise existing callers goes behind an opt-in flag.
- Docs, types, and tests move in the same PR as the behavior; a signature change with stale docs is a bug.
- Docs, types, and tests move in the same PR as the behavior. A signature change with stale docs is a bug — the docs are the contract users code against.
- Bug fixes come with a regression test that fails before the fix. Be suspicious of fixes that change observable behavior: someone may depend on the bug. When in doubt, ask a maintainer.
### Breaking changes
-365
View File
@@ -1,365 +0,0 @@
# Embedded app browser
`GET /embed/apps` on the deployment's main GUI origin renders the same app browser
as the dashboard's Apps tab, without the desktop or dashboard navigation.
It supports search, paging, saved folders, and saved ordering. It uses the same
installed-app and recommendation data as the dashboard, including the user's
removed recommendations. Editing, adding, reordering, renaming, and uninstalling
are disabled. Selecting an app opens it in a new browser tab at its own URL, the
same way the dashboard does, and the host page stays open. The app's token isn't
put in the URL: the app's SDK asks the embed for it once the tab loads, so the
embed must still be open then. If the iframe is removed first, the app falls back
to its normal sign-in. Apps that need the desktop (built-in and godmode apps,
private apps the user can't open, or a failed token request) open `/app/<name>`
on the GUI origin, cut off from the embed. Website shortcuts open their external
destination with no opener.
The embed follows the OS's light or dark setting. A host with a fixed theme adds
`theme=dark` or `theme=light` to the fragment (`#user=<uuid>&theme=dark`), since
Chromium doesn't pass an iframe's `color-scheme` on to the page inside it.
Apps opened from the embed carry `utm_source=puter.com&utm_medium=referral&utm_campaign=app_browser_embed`;
the desktop and dashboard send `utm_campaign=desktop` and `utm_campaign=dashboard`.
## Session and framing
The embed shows the GUI session only to the user the host page is signed in as.
The host names that user's `uuid` (from `puter.auth.getUser()`) in the iframe
URL's fragment, as `/embed/apps#user=<uuid>`. The embed reads the session, asks
`/whoami` which user it belongs to, and shows the signed-out view unless the two
match. So a host that names no user, a malformed one, or a user other than the
session's, gets the signed-out view, and so does a session the API refuses. A
failed `/whoami` check shows the error state. The fragment never reaches the
server and can only narrow what the embed shows: naming the session's own user
gives the same view as a direct visit to the GUI. Changing the fragment of a
loaded embed reloads it.
The embed reads the existing GUI session from local storage on its own origin.
If that storage holds no session, the embed gets the session from the GUI origin's session cookie through
`/get-gui-token`. This matters in Safari, which gives a framed origin separate
local storage under each top-level origin, so even a sibling subdomain's iframe
can't see the GUI's stored token. The session cookie is still sent, because the
host is on the same site. The embed doesn't write that token back to storage. It
does not create accounts, show a login dialog, or accept a token from its parent.
Being signed in on a subdomain alone does not create a session on the main GUI
origin.
Without a session it can show, the embed shows the recommended apps (`RecommendedAppsService`)
that the server put in the page, and nothing else: no installed apps, folders, or
saved order, and no puter.js. Selecting one opens it at the same URL a signed-in
launch uses, with no opener and no token on offer, so the app signs in on its own.
Apps that need the desktop open `/app/<name>`. If the page carries no recommended
apps, or a load fails, the embed displays an unavailable/error state.
The route's `Content-Security-Policy: frame-ancestors` allows the GUI origin and
the embedding page's origin when it is exactly one label under `config.domain`
with the scheme and port from `config.origin`. Subdomains the server answers
itself (`api`, `dav`, `js`, `www`, the native-app hosts, and the host of
`api_base_url`) and the user-content hosting domains (`static_hosting_domain`,
`private_app_hosting_domain`, and their `_alt` variants) are refused. For
example, `https://puter.com` permits `https://spreadsheet.puter.com` but not
`https://api.puter.com`, and `http://puter.localhost:4100` permits
`http://spreadsheet.puter.localhost:4100` but not
`http://someone.site.puter.localhost:4100`. The embedding page is read from the
iframe request's referrer, so the host must not suppress it (the browser default
`strict-origin-when-cross-origin` is enough). All ancestors in a nested iframe
chain must be allowed. Other GUI routes retain `X-Frame-Options: SAMEORIGIN`.
The embed accepts no query parameters. Requests with a query string redirect to
the same embed path so SDK bootstrap parameters cannot override the session or API
origin. The host needs to allow the GUI origin in its own `frame-src` CSP if it
sets one. If sandboxing the iframe, allow scripts, same-origin storage, popups,
and popups to escape the sandbox.
## Host integration
A first-party app that signs in with puter.js can place `<puter-user-menu>` from
the SDK in its header instead (see `src/puter-js/src/ui/components.md`). It opens
this embed with the signed-in user named and closes it when the app's sign-in
changes. The rest of this section is for a host that builds the iframe itself.
The iframe can be shown whether or not the user is signed in. Create it when
“More apps” is opened, naming the signed-in user in its fragment, and remove it
on close so reopening loads current apps and preferences. Remove it when the
host's own sign-in changes too, such as on sign-out, so an open embed doesn't
keep the previous user. The host owns the modal and close control;
there is no parent messaging API and no app list or credential is sent to it.
The app browser fills its iframe and responds to its size. It shows a spinner until its apps load.
This example assumes `puterOrigin` comes from the host app's deployment
settings. Translate the button and frame labels using the host app's
localization system.
```html
<button id="more-apps" type="button">More apps</button>
<dialog id="apps-dialog" aria-label="More apps">
<button id="close-apps" type="button">Close</button>
<div id="apps-frame"></div>
</dialog>
<style>
#apps-dialog { padding: 12px; width: min(900px, 90vw); }
#apps-frame iframe { width: 100%; height: min(650px, 75vh); border: 0; }
</style>
<script type="module">
const puterOrigin = 'https://puter.com'; // Use this deployment's configured origin.
const dialog = document.querySelector('#apps-dialog');
const container = document.querySelector('#apps-frame');
const trigger = document.querySelector('#more-apps');
trigger.addEventListener('click', async () => {
const url = new URL('/embed/apps', puterOrigin);
// Signed out, the embed shows the recommended apps.
const user = puter.auth.isSignedIn() ? await puter.auth.getUser().catch(() => null) : null;
if (user) url.hash = `user=${encodeURIComponent(user.uuid)}`;
const frame = document.createElement('iframe');
frame.title = 'Your apps';
frame.src = url.href;
container.replaceChildren(frame);
dialog.showModal();
});
document.querySelector('#close-apps').addEventListener('click', () => dialog.close());
dialog.addEventListener('close', () => {
container.replaceChildren();
trigger.focus();
});
</script>
```
## Implementation
`createAppBrowser` in `src/gui/src/UI/Dashboard/AppBrowser.js` owns the shared UI
and data behavior; its `guestApps` option is the signed-out list. `TabApps.js`
injects desktop actions. `appsEmbed.js` supplies new-tab launching through
`launchAppInTab` (signed in) and `guestAppLaunchUrl` (signed out) in
`src/gui/src/helpers/appTabLaunch.js`, which build the same app URL as
`launch_app`, and loads only the embed entry, its SDK, and shared styles. `embedHostUser` in
`src/gui/src/helpers/embedHostUser.js` reads the host's user from the fragment, and
`embedHostTheme.js` beside it applies the host's theme.
The normal GUI webpack build emits `bundle.min.js`, `apps-embed.min.js`, and `feedback-embed.min.js`.
The `/embed/apps` route puts the recommended list in the page's config as
`guestApps`. No puter.js API or app-list endpoint changes are needed.
## Feedback dialog
`GET /embed/feedback` is the app feedback dialog (`puter.ui.showFeedbackDialog()`)
for a first-party page, with the same framing rules, query redirect, and
host-user binding as `/embed/apps`. It sends the feedback for the framing
page's origin, which the server reads from the referrer and pins with
`frame-ancestors`; opened top-level, it does nothing. Its page is transparent
so the dialog's own scrim covers the host.
puter.js uses it on a page one label under the GUI origin, other than the
subdomains the server refuses, when the page is signed in, framing it in a
modal `<dialog>` that stays hidden until the embed is ready. A session check that takes longer than 200ms shows a loading scrim
with a spinner in its place; Escape or a click on it cancels. The embed posts `{ msg: 'feedbackEmbed', state }` to its parent
(`loading`, then `ready` or `unavailable`) and finally
`{ msg: 'feedbackDialogClosed', sent }`. It reports `unavailable` without the
named user's session, or while that user's email still needs confirming, and
puter.js then uses the `/action/send-feedback` popup, which can sign in. So does
a frame that never answers, such as one the host's CSP blocks. `feedbackEmbed.js`
is the entry, sharing `embedSession` in `src/gui/src/helpers/embedSession.js`
with the app browser; `showFeedbackEmbed` in `src/puter-js/src/lib/guiEmbed.js`
is the host side.
## Sign-in and file pickers
`GET /embed/sign-in`, `/embed/show-open-file-picker`, `/embed/show-save-file-picker`
and `/embed/show-directory-picker` serve the same GUI popups as `/action/<action>`
for a first-party page, with the same framing rules, query redirect, and
`no-store` as `/embed/apps`. Any other spelling of the path (another case, a
trailing slash) redirects to it, since the shell boots the desktop for it. They render the normal GUI shell, which names the
framing page (read from the referrer, pinned with `frame-ancestors`) as
`gui_params.embed_host_origin`. The GUI then runs the popup boot unchanged, with
the framing page as `window.opener`, and `window.close()` asks the page to
remove the frame (`src/gui/src/helpers/popupFrame.js`). Without a framing page
to answer (opened top-level, or framed by a refused page) it does nothing. The
popup's parameters come from the fragment instead of the query: `msg_id`,
`options` and `request_auth`, plus `user` for a picker.
The frame posts `{ msg: 'puterPopupFrame', state }` to its parent: `loading` as
soon as the shell boots, `ready` once it shows a window or dialog, `unavailable`
when the page has to use the popup, `provider` when the user picks a sign-in
provider (below), and `closed` where a popup would close. Its
answers (`puter.token`, `fileOpenPicked`, `directoryPicked`, `fileSaved`, the
cancel messages, `sendMeFileData`) are the popup's.
puter.js uses them on a page one label under the GUI origin, other than the
subdomains the server refuses, unless the page is cross-origin isolated or in a
frame of another site (which `frame-ancestors` refuses too).
`showPopupFrame` in `src/puter-js/src/lib/guiEmbed.js` shows the frame in a box
the size of the popup window, over a scrim, hidden until `ready` and behind the
loading spinner after 200ms. Escape closes it, on the page or in the frame. One shows per popup, as each
popup has its own window name: a second picker request of the same kind
replaces the one showing, and a `signIn()` call made while a framed sign-in is
open settles with it. On
screens up to 600px wide the box fills the screen. A frame that is refused,
never answers, or reports `unavailable` falls back to the popup, opened through
a consent dialog when the click that asked for it has expired. A reload or navigation the frame
starts itself (logging out, a sign-out in another tab) names the frame as its
referrer, so the server refuses it; the page then shows its spinner and opens
the frame again, once per `ready`.
- **Sign-in** (`puter.auth.signIn()`) has no host-user binding: it is how the
page gets a user. The box has a close button, which rejects with
`auth_window_closed` as closing the popup does. Providers refuse to be
framed, so choosing one posts `{ state: 'provider', url }` with the
provider's start URL for this sign-in's `msg_id`, then `unavailable`. The
page opens the popup straight at that URL, once it checks the URL is the
GUI's `/auth/oidc/<provider>/start` for this `msg_id`, and the provider
returns it as this sign-in's popup. `attempt_temp_user_creation` keeps the popup, since
only the popup creates a temporary account.
- **Pickers** frame only for a signed-in page. The GUI reports `unavailable`
unless its session belongs to the user named in `user`, and the page then uses
the popup, which can sign in and switch accounts. The token the GUI sends
before a picker's answer isn't adopted, since it is for the page's own user.
Dismissing the loading scrim settles the picker as cancelled.
## Upgrade
`GET /embed/upgrade` is the deployment's upgrade dialog for a first-party page,
served like the popups above: the GUI shell with the same framing rules, query
redirect, exact-path redirect and `no-store`, naming the framing page as
`gui_params.embed_host_origin`. It runs only when framed by an allowed page.
puter.js frames it on a signed-in first-party page (the same pages as the
pickers) for `puter.ui.requestUpgrade(details)`, and for the Upgrade button of
the usage limit dialog a refused call shows there. Elsewhere both keep their
behavior: inside an app `requestUpgrade()` asks the desktop, and on any other
page the usage limit dialog opens the dashboard's upgrade page in a new tab.
The frame covers the page with no box or close button, and the GUI page is
transparent (`html.popup-frame-transparent`) so the dialog's own scrim covers
the host. The frame allows `payment` as well as `otp-credentials`. Escape on the
page is left to the dialog, since one there shouldn't drop a payment in
progress. Its fragment names `user` (the page's signed-in user) and, when the
refusal gave them, `reason` (`funds`, `subscription` or `storage`) and `method`.
The GUI reports `unavailable` unless its session belongs to `user`, the account
has no verification or password change waiting (which the frame couldn't be
closed from), and the deployment defines `window.UIUpgradeAccount` with
`UIUpgradeAccount.embeddable === true`. Otherwise it runs:
```js
new UIUpgradeAccount({ attribution: { appUid, reason, method }, onClose }).open_as_window();
```
`appUid` is the app the framing page's origin resolves to, never one the
fragment names. `open_as_window()` resolves once the dialog shows, and the GUI
then posts `ready`. While the page still hides the frame, which Chromium and
WebKit lay out at 0x0, the GUI posts `ready` first and opens the dialog once
the frame has a viewport, so a window it opens doesn't center on 0x0.
`onClose({ upgraded })` must run exactly once, when the
dialog closes or when it declines to open for the account; the GUI then posts
`{ msg: 'puterPopupFrame', state: 'closed', upgraded }` and the page removes
the frame. If `open_as_window()` rejects, the GUI posts `unavailable` instead. `requestUpgrade()` resolves `{ upgraded }`. A call made while the
frame shows settles with it rather than replacing it, and a call refused while
it shows still rejects but opens no usage limit dialog over it.
When the frame can't show the upgrade (refused, never ready, `unavailable`, or
no user to name), puter.js opens the dashboard's upgrade page in a new tab with
what was refused, through a consent dialog when the click has expired or the
tab is blocked, and `requestUpgrade()` resolves `{ upgraded: false }`.
## Top-up
`GET /embed/top-up` is the deployment's top-up dialog, where a user with a plan
buys add-on credit, served and framed like the upgrade: a transparent GUI page
covering the host with no box or close button, allowing `payment` and
`otp-credentials`, with Escape left to the dialog. Its fragment names `user`.
puter.js frames it for the **Add credits** item of `<puter-user-menu>`, which
shows only when whoami reports `subscribed: true` and no `team` paying for the
account. On a page that can't frame the GUI, the item opens the dashboard with
`?topup=` in a new tab instead.
The GUI reports `unavailable` unless its session belongs to `user`, the account
has no verification or password change waiting, and the deployment defines
`window.UITopUp` with `UITopUp.embeddable === true`. Otherwise it runs:
```js
new UITopUp({ onClose }).open();
```
`open()` resolves once the dialog has loaded the account and shows what it can
buy, and `onClose({ toppedUp })` must run exactly once, when the dialog closes
or when it declines to open for the account. The GUI then posts `{ msg: 'puterPopupFrame', state: 'closed', toppedUp }`
and the page removes the frame. When the frame can't show the top-up, puter.js
opens the dashboard's top-up in a new tab, as it does the upgrade.
While the top-up shows, a call refused for funds still rejects but opens no
usage limit dialog over it, and `requestUpgrade()` frames nothing over it: it
resolves `{ upgraded: false }` once the top-up closes.
## Email confirmation
`GET /embed/request-email-confirmation` walks a first-party page's user through
confirming their email, served like the upgrade: the GUI shell with the same
framing rules, redirects and `no-store`, run only when framed by an allowed
page. puter.js frames it on a signed-in first-party page for
`puter.ui.requestEmailConfirmation()`. Inside an app the desktop shows the same
windows over the app instead.
Like the upgrade, the frame covers the page with no box or close button, over a
transparent GUI page, and its fragment names `user`. Escape closes it, on the
page or on the save-account or code window. On a window, Escape closes that
window as its close button does, so a code already sent still answers. The GUI
reports `unavailable` unless its session belongs to `user` and the account has
no phone, card or password check waiting.
An email check is fine: the boot skips its own email gate for this action, so
the user gets one code window they can close.
The GUI then runs `ensureConfirmedEmail` (`src/gui/src/helpers/ensureConfirmedEmail.js`),
which the desktop's handler uses too: a temporary account gets the save-account
window first, then an unconfirmed email gets the code window, each at most
once. When the page's whoami already reads the email as confirmed,
`requestEmailConfirmation()` resolves `{ response: true }` without framing
anything, and a confirmation in the frame marks that whoami confirmed for the
calls after it. An email the GUI reads as confirmed shows nothing. Before
opening a window the GUI waits for the frame to have a viewport, as the upgrade
does. It then posts
`{ msg: 'puterPopupFrame', state: 'closed', confirmed }`, and
`requestEmailConfirmation()` resolves `{ response: confirmed }`, the field an
app reads from the desktop's answer. A call made while the frame shows settles
with it. A frame that is refused, never ready, or `unavailable` resolves
`{ response: false }`, since there's no popup to fall back to.
## Custom domains
`GET /embed/attach-custom-domain` is the custom domain dialog of
`puter.hosting.attach()`, the popup at `/action/attach-custom-domain`, served
like the upgrade: the GUI shell with the same framing rules, redirects and
`no-store`, run only when framed by an allowed page. puter.js frames it on a
signed-in first-party page (the same pages as the pickers).
The frame covers the page with no box or close button, over a transparent GUI
page, so the dialog floats over the host with its own scrim as it does on the
desktop. The frame allows `clipboard-write`, for the dialog's copy buttons, as
well as `otp-credentials`. Escape on the page is left to the dialog, which
doesn't close while a request is in flight. Its fragment names `user` (the
page's signed-in user), `subdomain` (the site to attach a domain to) and
`msg_id`.
The GUI reports `unavailable` unless its session belongs to `user` and the
account has no verification or password change waiting. It then runs the
popup's action unchanged, with the framing page as the requester the dialog
names: it posts `ready` once the dialog is on the page, answers
`{ msg: 'customDomainAttached', domain, original_msg_id }` once the dialog has
closed and its exit has played, and then posts `closed`. `domain` is the domain the user chose, or
null. `attach()` resolves it. A frame that closes without an answer, or a
loading scrim the user dismisses, resolves null.
A call made while the frame shows resolves null without replacing it, as a
second call does on the desktop. A frame that is refused, never ready, or
`unavailable` falls back to the popup, which can sign in, through a consent
dialog when the click has expired.
## Checks
Run the GUI component/browser tests (with the SDK bundle built and Playwright
Chromium installed), and the backend route tests:
```sh
npx vitest run --config src/gui/vitest.config.js src/gui/src/UI/Dashboard/AppBrowser src/feedbackEmbed
npx vitest run src/puter-js/src/lib/guiEmbed.test.js src/puter-js/src/modules/UI.showFeedbackDialog.test.js src/puter-js/src/modules/Auth.signInFrame.test.js src/puter-js/src/modules/UI.pickerFrame.test.js src/puter-js/src/modules/UI.requestUpgrade.test.js src/puter-js/src/modules/UI.showTopUp.test.js src/puter-js/src/modules/UI.attachCustomDomainFrame.test.js src/puter-js/src/lib/upgradePrompt.test.js src/puter-js/src/modules/UsageLimitDialog.upgrade.test.js
npx vitest run --config src/gui/vitest.config.js src/gui/src/helpers/popupFrame.test.js
npx vitest run --config src/backend/vitest.config.ts src/backend/server.test.ts
```
+7 -18
View File
@@ -28,6 +28,9 @@ Paginated responses are an envelope:
`cursor` is still returned. Never use `items.length < limit` as an
end-of-list signal.
New request params are camelCase (`includeTotal`, `fetchUntilFull`).
Pre-existing snake_case params stay for compatibility.
## Backward compatibility
Requests without pagination params keep returning the full result in the
@@ -45,27 +48,14 @@ The envelope trigger depends on the endpoint's history:
## Cursors
Cursors are opaque base64-encoded JSON, produced and consumed only by the
backend (`src/backend/util/pagination.ts`). A cursor over a global sequence
(an auto-increment `id`) is sealed with `sealCursor`/`openCursor` instead, so
two pages can't be decoded into how fast the table grows; `openCursor` still
reads a plain cursor. What a cursor wraps is an implementation detail per
store:
backend (`src/backend/util/pagination.ts`). What a cursor wraps is an
implementation detail per store:
- DynamoDB-backed lists wrap `LastEvaluatedKey`. A kv listing of another
namespace (cross-app or through a share handle) resumes after the last entry
it returned instead, since the scan can stop on a private one, and its cursor
is sealed to that listing and padded to one length.
- DynamoDB-backed lists wrap `LastEvaluatedKey`.
- SQL-backed lists wrap a keyset position — `(sortValue, id)` of the last
row — and the query seeks past it (`WHERE (col, id) > (?, ?) ORDER BY col,
id`). Any SQL list gaining a cursor must have a deterministic `ORDER BY`
ending in a unique tiebreaker (`id`).
- The notification catch-up (`GET /events/fetch`) holds two keysets in one
sealed cursor: the last `id` it returned, and the last `(replaced_at, id)`
among rows up to that id. Each is its own indexed read, merged into one
page, so a row replaced after a client read it comes back from that cursor.
`from=end` with no cursor answers an empty page whose cursor holds the
slice's newest `id` and the slice's latest `(replaced_at, id)` as the
replacement mark.
Cursors that carry a sort also pin it: a request that passes a cursor plus a
conflicting sort is rejected with 400.
@@ -101,8 +91,7 @@ hit, the response simply carries a cursor — still convention-legal.
## Adding pagination to a new endpoint
1. Use `encodeCursor`/`decodeCursor`/`normalizeLimit`/`normalizeOffset` from
`src/backend/util/pagination.ts` — `sealCursor`/`openCursor` when the
cursor wraps a global id.
`src/backend/util/pagination.ts`.
2. Push equality filters into the query so pages and counts operate on the
true result set; only genuinely per-actor filtering may remain post-query
(short-pages rule covers it).
-80
View File
@@ -1,80 +0,0 @@
# Browser push client
Puter's desktop shows pushes through `/sw.js`. Core shows Settings → Notifications only when the `pushEnabled` gui param is `true`; core never sets it. The worker is registered only when push is turned on in the browser, and a load removes one left without a subscription. A backend extension that delivers pushes sets it with `services.homepage.setGuiParam('pushEnabled', true)` and can load its own code into the worker with `services.homepage.registerServiceWorkerScript(path)`.
Core subscribes the browser to a push service only when the server also publishes `pushWebPushKey` (see [Built-in Web Push client](#built-in-web-push-client)), and it has no `/push/*` routes of its own. The device, app, preference and subscription endpoints below come from that extension. On a deployment without one, such as most self-hosted servers, they answer core's plain 404 (`code: 'not_found'`), which `puter.notifications` reports as `push_unavailable` and the Notifications tab shows as unavailable. A device the extension can't find is `device_not_found`.
## Push payload
`sw.js` reads the push's JSON body. The message is its `puter` field, or `data.puter` for services that only carry string data, as an object or a JSON string:
```ts
{ uid?: string; title?: string; body?: string; icon?: string; group?: string;
target?: { app?: string; path?: string }; data?: unknown; channel?: string;
type?: 'challenge'; challenge?: string }
```
`uid` is the notification's uid, a later push with the same `group` replaces the one showing, and `target` and `data` reach the app when the user taps it.
When the browser replaces the subscription (`pushsubscriptionchange`), `sw.js` keeps the new one, or subscribes again with Puter's stored key when the browser gave none and the event, the receipt or an earlier mark names the subscription that went. It writes `puter:subscription-changed:<scope>` to the `puter-push` IndexedDB as `{ at, oldEndpoint }`, the old endpoint kept from an earlier mark no page handled yet, else taken from the event or from the receipt, forgets the receipt kept for that scope, and posts `{ type: 'puter-push-subscription-changed' }` to every open window on Puter's origin. The worker has no token or session, so an open page runs `helpers/pushSync.js` on that message, after any run already in flight, and a later load does the same through the marker.
## Receipts
Before a notification is pushed, the server asks the open clients whether they received it live. It sends `notif.receipt` with `{ uid }` on the user's sockets for the app that sent it, or on the desktop's sockets for Puter's own notifications. The SDK remembers the uids its page received through `onNotification()`, `onPush()` or the desktop's feed, and answers once its page has that notification, or with `{ received: false }` when it never gets it. The server waits up to 5 seconds (`notificationAckTimeoutMs`).
An answer carries `visibilityState` and `focused` as they were when the page received the notification. Pages never show a browser notification themselves; pushes do.
- An answer from a page that was visible and focused stops the push (`handled` on `notif.delivery.settled`). An older client that doesn't send `focused` counts while visible.
- Otherwise the notification is pushed, to every browser the user subscribed. A website's pushes go only to the browsers that subscribed through its own worker, never to Puter's `/sw.js`. An app inside the desktop pushes through `/sw.js`, which shows the app's name and icon.
- A client that can't answer, such as an older puter.js, doesn't hold the push back.
The ask reaches only the sockets in the asking region. So that other regions hear about a page too, a page that receives a notification live while visible and focused also sends `notif.handled` with `{ uid }` on the socket it came on. Core relays it as the node-local `notif.handled` event with the socket's user and app, at most 60 a minute per user; an extension that delivers pushes decides what to do with it.
## Registering
The registry is in [src/gui/src/helpers/pushClient.js](../src/gui/src/helpers/pushClient.js). A service script registers its client like this:
```js
service_script(async (api) => {
api.use()['push.registerPushClient']?.(client);
});
```
`use(id)` throws when nothing is registered under `id`, so the form above also runs on a Puter version without the registry and does nothing there. Any script that runs after the GUI bundle can call `use()['push.registerPushClient']?.(client)` the same way. Registering again replaces the client. Registering dispatches `push-client-registered` on `window`, and an open Notifications tab redraws.
With no client registered, the tab says push notifications aren't available in this browser.
### Built-in Web Push client
When the server also sets the `pushWebPushKey` gui param (a VAPID public key), core registers its own client ([webPushClient.js](../src/gui/src/helpers/webPushClient.js)) unless an extension registered one first. It uses `puter.notifications.enable()` / `disable()`, the same standard Web Push flow apps use on their own origins:
1. `GET /push/web/key` for the key, then `pushManager.subscribe()` on the `/sw.js` registration.
2. `POST /push/web/subscriptions { subscription }`. A subscription the server already has answers `{ status: 'active' }`. A new one answers `{ status: 'pending' }`, and the server sends it a push with `{ type: 'challenge', challenge }`.
3. The worker shows that push and posts `{ type: 'puter-push-challenge', challenge }` to the origin's windows; the page sends it to `POST /push/web/subscriptions/verify`. The device is registered once the page hands the confirmation back.
4. `DELETE /push/web/subscriptions { endpoint }` removes it.
Apps on their own origin do the same through `push-sw.js`, which they import into their own worker. A subscription made with an app token is the app's: its pushes are that app's notifications only.
## Interface
```ts
interface PushClient {
enable(options?: { background?: boolean }): Promise<{ deviceUid: string }>;
disable(): Promise<void>;
status(options?: { cached?: boolean }): Promise<{
permission: 'default' | 'granted' | 'denied';
registered: boolean;
deviceUid?: string;
}>;
}
```
- `enable()` may ask for notification permission, so core calls it from a click: Turn on in the Notifications tab, Allow on a permission prompt that includes `notifications:push` while this browser is off, and the "Turn on notifications" button of the `/action/enable-push` popup and the matching desktop dialog (`UIEnablePushDialog`). Core also calls it without a click, from `helpers/pushSync.js`, when `Notification.permission` is already `granted`, `status()` reports the browser as not registered, the user has not turned push off in this browser, and an app may push to Puter's devices (`puterDevices` in `GET /push/apps`); the browser does not prompt then. That call passes `{ background: true }`: a browser the user removed in Puter stays removed, and the built-in client rejects with code `push_device_removed`, after which core records the browser as turned off. Before a background call the built-in client asks `POST /push/web/subscriptions/status` about every earlier endpoint it knows: the marker's `oldEndpoint` and the last endpoint the server answered `active` for (`puter.push.last-endpoint:<user uuid>` in `localStorage`). On a 429 it registers nothing and rejects with `too_many_requests`, which the sync reports as `later`, keeping the marker for the next run. When any answer is `removed`, it ends the current subscription in the browser, clears the marker and rejects with `push_device_removed`, so a subscription change or a dropped subscription never brings back a browser removed from another device. A click renews it as before. For a click the built-in client runs `puter.notifications.enable()`, which subscribes with the `/sw.js` registration and registers the browser through the Web Push flow above (`platform: 'webpush'`); in the background it runs the same registration without replacing a removed subscription or opening Puter's sign-in. It then resolves the `uid` of this browser's `webpush` device from `listDevices()` and clears the subscription-changed marker. It rejects when permission is refused or the browser wasn't registered.
- `disable()` runs `puter.notifications.disable()`, which sends `DELETE /push/web/subscriptions { endpoint }` and ends the browser's subscription. The tab calls it for Turn off, and for Remove on the device marked as this browser. Remove on any other device is `DELETE /push/devices/:uid`.
- `status()` reports the state without prompting. `permission` is `Notification.permission`. `registered` is true when this browser has a device registered for the signed-in user and `POST /push/web/subscriptions/status` answers `active` for the current subscription's endpoint, and `deviceUid` is that device's `uid`. A server that can't answer leaves the device match alone, and so does an `unknown` for an endpoint `enable()` registered in this page in the last 5 minutes, since that check can trail a registration. While the subscription-changed marker is set, `registered` is false. With `{ cached: true }` a client may answer with the last registered state it read in this page. Core passes it for the silent registration on app launch and for an app's `puter.notifications.status()`; the Notifications tab and the push dialogs read fresh. The built-in client keeps that state in memory for the same user and subscription for up to 5 minutes, from `enable()` or once the server answered `active` for the endpoint, and answers from it without either request. Reads of the same user and endpoint share one real check while it is in flight. It drops it on `disable()`, a failed `enable()`, an unregistered answer, a set subscription-changed marker, and the worker's `puter-push-subscription-changed` message, so a new page load always asks the server. Whether the calling app may push is checked on every call, against a `GET /push/apps` answer the page reuses for 30 seconds and drops on `enable()`, `disable()`, a grant, a revoke and a user change. When it can't be read, or leaves the app out, the grant alone counts.
After `enable()` resolves, the tab calls `POST /push/devices/:uid/test` and asks the user whether a notification showed up.
The tab also reads `GET /push/apps` and `GET`/`POST /push/preferences`.
Core keeps its own record of a browser the user turned off, `puter.push.browser-off:<user uuid>` in `localStorage`, written by Turn off, by Remove on this browser's device, and by a silent registration the server refuses with `push_device_removed`, and cleared when `enable()` resolves. The silent registration above skips a browser with that record, so a device removed here or from another device stays removed until the user turns it on from a click, while a new sign-in in a browser that already allows notifications registers again.
+16 -204
View File
@@ -12,17 +12,7 @@ Generates secrets, writes `.env` + `puter/config/config.json`, downloads `docker
## Requirements
- **Docker** with the `compose` plugin. This is the one thing you have to install
yourself — the installer checks for it and stops with an explanation rather
than installing it for you.
- **macOS / Windows:** [Docker Desktop](https://docs.docker.com/get-docker/).
Both ship the `compose` plugin.
- **Linux:** [Docker Engine](https://docs.docker.com/engine/install/) plus the
[`docker-compose-plugin`](https://docs.docker.com/compose/install/) package.
- Confirm before running anything else: `docker compose version` should print a
version, not `command not found`.
- If Docker is installed but the shell can't see it, make sure your user is in
the `docker` group (Linux) or that Docker Desktop is actually running.
- **Docker** with the `compose` plugin.
- A **domain** with DNS access — you need a wildcard record (`*.your-domain.com` → server IP). Puter routes by subdomain (`api.<domain>`, `site.<domain>`, `app.<domain>`).
- Optional: **TLS certs** (or `certbot` to grab them — see Step 3).
@@ -45,12 +35,6 @@ Optional services (compose profile `ai`, opt-in):
| `puter-ollama` | `ollama/ollama` | Local LLM provider (CPU; GPU passthrough opt-in) |
| `puter-ollama-init` | `ollama/ollama` | One-shot — pulls the default model (`tinyllama`) on first boot |
Optional services (compose profile `smtp`, opt-in):
| Container | Image | Role |
| ------------ | ------------------------ | -------------------------------------------------------- |
| `puter-smtp` | `ghcr.io/heyputer/puter` | Receives mail over SMTP and files it into user mailboxes |
State lives under `./puter/data/<service>/`.
---
@@ -138,10 +122,6 @@ cat > puter/config/config.json <<EOF
"ollama": { "enabled": false }
},
"meteringEnforcement": {
"subscriptions": false
},
"trust_proxy": 1
}
EOF
@@ -156,20 +136,9 @@ Why these knobs:
- `database.migrationPaths` — Puter applies the bundled MySQL/MariaDB schema on boot. The migration files are idempotent, so it is safe to leave this configured across restarts.
- `dynamo.bootstrapTables: true` — Puter creates its KV table on boot. **Only set against a local emulator**, never real AWS.
- `dynamo.aws` keys are dummies; DynamoDB-local doesn't validate them but the AWS SDK requires _something_. **Note:** DynamoDB uses `access_key` / `secret_key` (snake_case); S3 below uses `accessKeyId` / `secretAccessKey` (camelCase). Not interchangeable.
- `meteringEnforcement.subscriptions: false` — a few surfaces (the OpenAI/Anthropic-compatible AI endpoints) are declared paid-plan-only. A self-hosted install has no paid plans, so every account sits on the free policy and those routes would answer `402 subscription_required` for everyone. Turning the plan gates off opens them; nothing else about metering changes.
- `providers.ollama.enabled: false` — Puter auto-probes a local Ollama at `127.0.0.1:11434` by default; without one running you'd see `ECONNREFUSED` on every boot. To run a bundled Ollama, see [Optional: local LLM (Ollama)](#optional-local-llm-ollama) below.
- `s3.s3Config.forcePathStyle: true` — RustFS / MinIO / fauxqs need path-style URLs (`<endpoint>/<bucket>`). Real AWS S3 wants virtual-hosted (`<bucket>.<endpoint>`) — drop this flag (or set `false`) when you swap to real S3.
- `s3.s3Config.publicEndpoint` — `endpoint` (`http://s3:9000`) only resolves inside the docker network; presigned upload/download URLs handed to the browser need a host-reachable URL. Caddy routes the `s3.<domain>` subdomain to RustFS internally and preserves the Host header end-to-end (required for S3 signature validation), so the browser hits the same port/protocol as the rest of the app — no separate published port, no mixed-content surprises when you turn on TLS. Switch to `https://s3.<your-domain>` once you enable TLS in Step 3. Real AWS S3 doesn't need this — its endpoint is already public; drop the field entirely.
- **Bucket CORS** — the browser uploads file bytes straight to RustFS with a cross-origin `PUT` to the presigned URL (Dev Center deploy, `puter.fs.upload`). That only works if the bucket answers the `OPTIONS` preflight with `Access-Control-Allow-*` headers. The `s3-init` container applies those rules on every boot (`put-bucket-cors`, origins `*` — the signed URL is the credential, and the SDK sends no cookies). If deploys fail with `No 'Access-Control-Allow-Origin' header is present`, re-run it: `docker compose run --rm s3-init`. Verify with:
```bash
curl -sk -X OPTIONS "https://s3.<domain>/<bucket>/test" \
-H "Origin: https://<domain>" \
-H "Access-Control-Request-Method: PUT" \
-H "Access-Control-Request-Headers: content-type" -D - -o /dev/null | grep -i access-control
```
You should see `access-control-allow-origin`, `allow-methods: GET, HEAD, PUT, POST, DELETE`, `allow-headers: *`, and `max-age: 3600`.
- `trust_proxy: 1` — Caddy terminates TLS and forwards `X-Forwarded-For`. Without this, `req.ip` is the docker-network address of the Caddy container instead of the real client IP, which breaks rate limiting and IP-based audit logs. `1` = one trusted hop (Caddy). Bump to `2` if you put Cloudflare in front of Caddy; never set `true` (it trusts every hop and makes XFF forgeable).
> If you ever change `MARIADB_PASSWORD` after first boot, `.env` alone won't update MariaDB — its credentials are baked into `./puter/data/mariadb/` on first init. Either rotate the password inside MariaDB by hand or `docker compose down && rm -rf ./puter/data/mariadb` to start fresh.
@@ -226,7 +195,7 @@ Drop the resulting `fullchain.pem` and `privkey.pem` into `./puter/tls/`.
```
…and update the S3 public endpoint:
```json
"s3": { "s3Config": { "publicEndpoint": "https://s3.puter.localhost", ... } }
"s3": { "s3Config": { "publicEndpoint": "https://s3.puter.local", ... } }
```
## Running behind your own reverse proxy
@@ -273,15 +242,15 @@ Healthy startup:
```
[config] override from /etc/puter/config.json
[mysql] running migrations from /opt/puter/dist/src/backend/clients/database/migrations/mysql: <n> file(s)
[mysql] running migrations from /opt/puter/dist/src/backend/clients/database/migrations/mysql: 2 file(s)
[mysql] applied mysql_mig_1.sql (...)
...
[mysql] applied mysql_mig_2.sql (9 statements)
```
Then open **<https://puter.localhost>** (or `http://` if you skipped TLS). Login is `admin`; its temporary password is printed in the puter container logs on every boot until you change it:
Then open **<https://puter.local>** (or `http://` if you skipped TLS). Login is `admin` — the temp password is printed once in the puter container logs on first boot:
```bash
docker compose logs puter | grep "password for admin"
docker compose logs puter | grep tmp_password
```
Change it in Settings after first login.
@@ -326,32 +295,6 @@ Used for password resets, email confirmation, and notifications. Without it thos
To require email confirmation before login, also set `"strict_email_verification_required": true`.
Account and notification mail is sent on a named stream: `auth` (password, address and 2FA changes), `signup` (address confirmation; a `puter.signup.validate` listener may name another: lowercase letters, digits, `:`, `_` and `-`, up to 64 characters, starting with a letter or digit) and `notify` (share, app feedback and contact form mail). Add `"streams"` inside `"email"` to configure one, for instance to let your mail provider route or report each kind of mail separately:
```json
"streams": {
"notify": {
"from": "\"Puter\" <[email protected]>",
"headers": { "X-Example-Stream": "notify" },
"tagsHeader": "X-Example-Tags"
}
}
```
`headers` are added to every message on the stream, `from` replaces the default sender, and `tagsHeader` names the header that carries a message's tags as `name=value` pairs joined by `, `. A stream that is not listed adds nothing: the message goes out with the default from address and the caller's headers. `streams` is never handed to the transport.
To read the mail yourself instead of sending it, point the transport at a local SMTP sink — [MailHog](https://github.com/mailhog/MailHog) catches everything and serves it at `http://localhost:8025`. Publish both ports; a container that only exposes them is unreachable from the host:
```sh
docker run -d -p 1025:1025 -p 8025:8025 mailhog/mailhog
```
```json
"email": { "from": "\"Puter\" <[email protected]>", "host": "127.0.0.1", "port": 1025, "secure": false, "ignoreTLS": true }
```
Share email is on once a transport exists: a recipient who already has an account is emailed as well as notified in-app, and they opt out with the unsubscribe link the mail carries. Set `"share_email_notifications": false` to keep those in the app only — an address with no account is still emailed, since there is nothing else to reach them with.
### Sign in with Google (or another OIDC provider)
```json
@@ -418,14 +361,14 @@ Full provider list (chat, image, video, TTS, OCR) is in the template.
### Per-user storage quota
Storage is unlimited by default (bounded by host disk). To cap each user:
Default is 100 MB per user.
```json
"storage_capacity": 5368709120, // 5 GB
"is_storage_limited": true
```
`storage_capacity` defaults to 100 MB.
Set `is_storage_limited: false` for unlimited (bounded by host disk).
### Usage metering and budgets
@@ -437,8 +380,8 @@ deletion, stay available so an account can always see what it has and clear it.
On a self-hosted install this is almost certainly not what you want. There is
nowhere to buy more, so accounts are held to the free monthly allowance
(US$0.50 of measured cost for a registered account, roughly 2.5 GiB of
downloads) and start being refused after that. Turn it off:
(US$0.25 of measured cost — roughly 2 GiB of downloads) and start being refused
after that. Turn it off:
```json
"unlimitedMetering": true
@@ -458,17 +401,6 @@ Calls driven by a deployed worker are exempt from enforcement by default,
because a worker has no prompt to show and nobody watching it fail. Set
`"meteringEnforcement": { "workers": true }` to include them.
Monthly usage records expire on their own once they're old enough to be of no
further use — `meteringRetentionMonths` full months past the month they
belong to, 3 by default. Set it to `0` to keep every month's records forever:
```json
"meteringRetentionMonths": 6
```
Changing it, including to `0`, applies to records first written after the
change; records already stamped keep their expiry.
### Captcha on signup / login
Built-in proof-of-work captcha — no external service needed.
@@ -488,10 +420,9 @@ temporary or permanent one.
"disable_user_signup": true
```
### Block disposable email domains
### Block disposable email TLDs
Addresses on these domains (suffix match, so subdomains too) can't sign up or be
set as an account's email.
Only enforced when `env: "prod"`.
```json
"blockedEmailDomains": ["mailinator.com", "tempmail.com", "guerrillamail.com"]
@@ -538,125 +469,14 @@ Without `--profile ai`, the `ollama` containers stay down and Puter (with `enabl
For GPU acceleration (NVIDIA), uncomment the `deploy:` block under the `ollama` service in [docker-compose.yml](../docker-compose.yml). Requires `nvidia-container-toolkit` on the host.
## Optional: receiving mail over SMTP
Puter can give each account a mailbox at a domain you control. Two pieces are
involved, and they are configured together under `userEmail`:
- an **ingress endpoint** (`POST /email/ingress`) that files a message into the
recipient's `~/.mail`. It exists whenever `userEmail.secret` is set.
- the **`puter-smtp` receiver**, which accepts mail from other mail servers and
hands each message to that endpoint. It runs as its own process, behind the
`smtp` compose profile.
> **Read this before you expose it.** An inbound mail server is unauthenticated
> by nature: anyone on the internet may hand it a message for any address in
> `localDomains`. Puter performs **no** spam filtering and **no** SPF, DKIM or
> DMARC checks — whatever is delivered is stored in the recipient's mailbox and
> counts against their storage. If that is not acceptable, put a filtering relay
> in front of it and point `localDomains` at that instead.
1. Configure it in `puter/config/config.json`:
```json
"userEmail": {
"secret": "a long random string",
"localServer": true,
"localDomains": ["example.com"]
}
```
`secret` is shared between the two halves; generate it with
`openssl rand -hex 32`. `localDomains` lists the domains this server accepts
mail for — anything else is refused at `RCPT TO`, which is what stops it
being used as an open relay. It has no default, and the receiver refuses to
start without it.
The receiver posts to `<api_base_url>/email/ingress` unless you set
`localIngressUrl`. The endpoint is served on the `api` subdomain, so if you
point it at an internal address to skip the reverse proxy, name that virtual
host too or the request will 404:
```json
"localIngressUrl": "http://puter:4100/email/ingress",
"localIngressHost": "api.example.com"
```
`localIngressHost` defaults to the host of `api_base_url`, which is usually
what you want.
2. Bring it up with the `smtp` profile:
```bash
docker compose --profile smtp up -d
docker compose logs -f puter-smtp
```
3. Point DNS at it:
```dns
mx.example.com. A 203.0.113.10
example.com. MX 10 mx.example.com.
```
Also set a `PTR` record for the address that resolves back to
`mx.example.com` — many senders refuse mail from a server without one.
Inbound TCP port 25 must reach the host. **Most cloud providers block port
25 by default and residential ISPs block it permanently**; if you cannot get
it unblocked, this will not work no matter how it is configured. The
container itself binds 2525 (it runs unprivileged) and compose publishes
`${SMTP_PORT:-25}` onto it, so you can move the host port if you need to.
A message is held in memory while it is being delivered, so peak memory is
bounded by `localMaxClients` (default 20) times the 25 MiB message ceiling.
Lower `localMaxClients` on a small host. There is no queue and no retry
schedule — if delivery fails, the sending server is told to try again later and
retrying is its job.
Mail is stored as `message/rfc822` objects under `~/.mail/objects/<date>/` and
read back through `puter.email` in the SDK.
Every setting is documented in
[config.template.jsonc](../config.template.jsonc). Two deliberate limits, both
in the name of keeping the self-hosted path simple: it does not offer STARTTLS,
so mail arrives in the clear, and it applies no per-sender rate limiting. Put a
filtering relay in front if either matters to you.
Note this is a *receiving* server only: it never sends, and it must not be used
for authenticated submission. Outbound transactional mail is the separate
`email` block above.
## Building from source instead of pulling
To run a local build against the full stack, use a source checkout and complete
the configuration steps above. Create `docker-compose.override.yml` in the
repository root, next to [docker-compose.yml](../docker-compose.yml):
```yaml
services:
puter:
pull_policy: never
build:
context: .
```
If that file already exists, merge these settings into its `puter` service.
Compose merges the override automatically: `build.context: .` builds this
checkout, and `pull_policy: never` stops it pulling the published image. Git
ignores the override, so local build settings stay out of pull requests and
don't conflict with `docker-compose.yml` when you pull updates.
From the repository root, build and start the stack:
If you want to test local Dockerfile changes against the full stack, uncomment the `build:` block in [docker-compose.yml](../docker-compose.yml) under the `puter` service, change `pull_policy: always` → `pull_policy: never`, then:
```bash
docker compose up -d --build
```
Run the same command after changing or pulling source code to rebuild the image.
To return to the published image, remove the local build settings (or the override
file if those are its only settings), then run `docker compose up -d`.
---
## Managing running backend
@@ -681,22 +501,14 @@ Migrations re-apply idempotently across pulls. Volumes are preserved.
## Troubleshooting
**`missing required command: docker` — the installer stops immediately.**
Docker isn't on your `PATH` yet. Puter self-hosting is a Docker Compose stack, so
nothing can run without it. Install [Docker Desktop](https://docs.docker.com/get-docker/)
on macOS/Windows or [Docker Engine](https://docs.docker.com/engine/install/) plus
`docker-compose-plugin` on Linux, then confirm `docker compose version` works in
the same shell and re-run the installer. On Linux, a fresh Docker install usually
needs a new login session before your user picks up the `docker` group.
**Site loads but I get a 502 / "Bad Gateway" from Caddy.**
The puter container failed to come up. `docker compose logs puter` will tell you which dependency rejected it (most often DB password mismatch between `.env` and `config.json`).
**Can't find the admin password.**
It's logged on every boot until you change it: `docker compose logs puter | grep "password for admin"`. After login, change it in Settings.
**Login screen says "admin password not set".**
First-boot temp password is logged once. Find it: `docker compose logs puter | grep "tmp_password"`. After login, change it in Settings.
**Healthcheck reports unhealthy but the site works.**
The compose healthcheck requests `http://localhost:4100/` from inside the container. If you changed `port` or set `allow_all_host_values: false`, the check fails while the site itself is fine.
The healthcheck hits `puter.localhost:4100/test` from inside the container. If you changed `domain` or `port`, the check still uses defaults. The site itself is fine.
**Nothing resolves at `puter.example.com` after DNS changes.**
DNS propagates slowly. `dig puter.example.com` and `dig api.puter.example.com` should both return your server IP. If not, give it 5–60 minutes.
+11 -52
View File
@@ -139,10 +139,8 @@ services:
start_period: 5s
s3-init:
# One-shot container that creates the `puter-local` bucket and
# applies its CORS rules on every boot (put-bucket-cors is
# idempotent). Without CORS, browser uploads to presigned URLs
# fail the preflight check.
# One-shot container that creates the `puter-local` bucket on first
# boot. Exits 0 once the bucket exists; stays exited 0 thereafter.
image: amazon/aws-cli:latest
container_name: puter-s3-init
depends_on:
@@ -165,26 +163,6 @@ services:
echo "creating bucket $$bucket"
aws --endpoint-url "$$endpoint" s3 mb "s3://$$bucket"
fi
# Browser uploads (Dev Center deploy, puter.fs.upload) PUT to
# presigned URLs cross-origin, so the bucket must answer the
# OPTIONS preflight. Origins are open because the presigned URL
# itself is the credential and the SDK sends no cookies.
cat > /tmp/cors.json <<'EOF'
{
"CORSRules": [
{
"AllowedOrigins": ["*"],
"AllowedMethods": ["GET", "HEAD", "PUT", "POST", "DELETE"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["ETag", "x-amz-request-id"],
"MaxAgeSeconds": 3600
}
]
}
EOF
aws --endpoint-url "$$endpoint" s3api put-bucket-cors \
--bucket "$$bucket" --cors-configuration file:///tmp/cors.json
echo "bucket CORS configured"
restart: "no"
# ── Optional: local LLM ───────────────────────────────────────────
@@ -252,8 +230,15 @@ services:
puter:
image: ghcr.io/heyputer/puter:main
pull_policy: always
# For local builds, create a docker-compose.override.yml;
# see doc/self-hosting.md#building-from-source-instead-of-pulling.
# Uncomment to build from this directory instead of pulling the published
# image. Also flip pull_policy to `never` so compose doesn't overwrite
# your local build by re-pulling :latest.
# build:
# context: .
# # buildx-only: cross-compile to both archs in a single push
# # platforms:
# # - linux/amd64
# # - linux/arm64
container_name: puter
restart: unless-stopped
depends_on:
@@ -286,32 +271,6 @@ services:
retries: 3
start_period: 30s
# Inbound mail. Behind the `smtp` profile, so it only starts when asked for:
# docker compose --profile smtp up -d
# Accepts SMTP from other mail servers and hands each message to the ingress
# endpoint, which files it into the recipient's mailbox. Needs `userEmail`
# configured — see doc/self-hosting.md#optional-receiving-mail-over-smtp.
puter-smtp:
profiles: ["smtp"]
image: ghcr.io/heyputer/puter:main
pull_policy: always
container_name: puter-smtp
restart: unless-stopped
command: ["node", "-r", "./dist/src/backend/telemetry.js", "./dist/src/backend/smtp/index.js"]
depends_on:
puter:
condition: service_healthy
ports:
# The container runs unprivileged and binds 2525; 25 is what an MX record
# means, and many providers block it.
- "${SMTP_PORT:-25}:2525"
environment:
PUID: 1000
PGID: 1000
volumes:
# The same config.json the backend reads; only `userEmail` applies here.
- ./puter/config:/etc/puter:z
caddy:
image: caddy:2.11-alpine
container_name: puter-caddy
+51 -23
View File
@@ -1,4 +1,5 @@
import { Context } from '@heyputer/backend/src/core';
import type { Actor } from '@heyputer/backend/src/core/actor';
import { HttpError } from '@heyputer/backend/src/core/http';
import { PuterDriver } from '@heyputer/backend/src/drivers/types';
import type {
@@ -132,30 +133,51 @@ export class AppTelemetryDriver extends PuterDriver {
const owner = await this.stores.user.getById(ownerId);
if (!owner?.uuid) throw new HttpError(404, 'App owner not found');
// `/drivers/call` requires auth, so there is always an actor.
const actor = Context.get('actor')!;
let ownsApp = false;
try {
ownsApp = await this.services.permission.check(
actor,
`apps-of-user:${owner.uuid}:write`,
);
} catch {
// A failed check is a denial.
}
const actor = Context.get('actor');
if (!actor) throw new HttpError(401, 'Authentication required');
const ownsApp = await this.services.permission
.check(actor as Actor, `apps-of-user:${owner.uuid}:write`)
.catch(() => false);
if (!ownsApp) throw new HttpError(403, 'Permission denied');
// An email surfaces only for a user who granted *this app*
// `user:<their-uuid>:email:read` — the grant `puter.perms.requestEmail()`
// obtains and `whoami` honours. Authenticating alone doesn't share it.
const users = await this.stores.permission.listAppAuthenticatedUsers(
(app as { id: number }).id,
{ limit: safeLimit, offset: safeOffset },
);
return users.map((u) =>
u.emailShared
? { user: u.username, user_uuid: u.uuid, user_email: u.email }
: { user: u.username, user_uuid: u.uuid },
const appId = (app as { id: number }).id;
const users = (await this.clients.db.read(
`SELECT u.id, u.username, u.uuid, u.email FROM user_to_app_permissions p
INNER JOIN ${this.clients.db.quoteIdentifier('user')} u ON p.user_id = u.id
WHERE p.permission = 'flag:app-is-authenticated' AND p.app_id = ?
ORDER BY (p.dt IS NOT NULL), p.dt, p.user_id
LIMIT ? OFFSET ?`,
[appId, safeLimit, safeOffset],
)) as Array<{
id: number;
username: string;
uuid: string;
email: string | null;
}>;
// Only surface a user's email if *that user* granted this app the
// `user:<their-uuid>:email:read` permission — the same grant
// `puter.perms.requestEmail()` obtains and `whoami` honours. This is a
// per-user check keyed on the app (not the calling owner-actor): a
// user may have authenticated into the app without sharing their
// email. Resolve the whole page in one query.
const emailPermitted = new Set<number>();
if (users.length > 0) {
const permStrings = users.map((u) => `user:${u.uuid}:email:read`);
const placeholders = permStrings.map(() => '?').join(', ');
const grants = (await this.clients.db.read(
`SELECT user_id FROM user_to_app_permissions
WHERE app_id = ? AND permission IN (${placeholders})`,
[appId, ...permStrings],
)) as Array<{ user_id: number }>;
for (const g of grants) emailPermitted.add(g.user_id);
}
return users.map((e) =>
emailPermitted.has(e.id)
? { user: e.username, user_uuid: e.uuid, user_email: e.email }
: { user: e.username, user_uuid: e.uuid },
);
}
@@ -168,7 +190,13 @@ export class AppTelemetryDriver extends PuterDriver {
const app = await this.stores.app.getByUid(app_uuid);
if (!app) throw new HttpError(404, 'App not found');
return this.stores.permission.countAppUsers((app as { id: number }).id);
const [row] = (await this.clients.db.read(
`SELECT COUNT(*) AS n FROM user_to_app_permissions
WHERE permission = 'flag:app-is-authenticated' AND app_id = ?`,
[(app as { id: number }).id],
)) as Array<{ n: number }>;
return row?.n ?? 0;
}
}
+10 -40
View File
@@ -9,7 +9,6 @@ import {
vi,
} from 'vitest';
import { runWithContext } from '../src/backend/core/context.ts';
import { extension } from '../src/backend/extensions.ts';
import { PuterServer } from '../src/backend/server.ts';
import { setupTestServer } from '../src/backend/testUtil.ts';
import { handleInstalledApps } from './installedApps.ts';
@@ -52,14 +51,13 @@ const seedUser = async () => {
});
};
const seedApp = async (ownerUserId: number, overrides = {}) => {
const seedApp = async (ownerUserId: number) => {
const slug = Math.random().toString(36).slice(2, 8);
return server.stores.app.create(
{
name: `iaapp_${slug}`,
title: `Installed App ${slug}`,
index_url: `https://example.com/${slug}`,
...overrides,
},
{ ownerUserId },
);
@@ -74,6 +72,15 @@ const grantInstalled = async (appId: number, userId: number) => {
};
describe('installedApps extension — handleInstalledApps', () => {
it('throws HttpError(401) when no actor is on the context', async () => {
const { res } = makeRes();
await expect(
runWithContext({ actor: undefined }, () =>
handleInstalledApps(makeReq({}), res),
),
).rejects.toMatchObject({ statusCode: 401 });
});
it('throws HttpError(400) when orderBy is not in the allowlist', async () => {
const user = await seedUser();
const { res } = makeRes();
@@ -138,43 +145,6 @@ describe('installedApps extension — handleInstalledApps', () => {
).toBe(false);
});
it('pairs iconUrl with the direct subdomain URL for a generated icon', async () => {
const user = await seedUser();
// The resize pipeline leaves an http(s) icon column behind, which is
// what says the sized PNG is on the icons subdomain.
const app = await seedApp(user.id as number, {
icon: 'https://api.puter.com/app-icon/x',
});
await grantInstalled(app!.id as number, user.id as number);
const { res, captured } = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleInstalledApps(makeReq({}), res),
);
const list = captured.body as Array<Record<string, unknown>>;
const { protocol, static_hosting_domain } = extension.config;
expect(list[0].iconCdnUrl).toBe(
`${protocol}://puter-app-icons.${static_hosting_domain}/${app!.uid}-256.png`,
);
});
it('reports no subdomain URL for an app with no generated icon', async () => {
const user = await seedUser();
const app = await seedApp(user.id as number);
await grantInstalled(app!.id as number, user.id as number);
const { res, captured } = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleInstalledApps(makeReq({}), res),
);
const list = captured.body as Array<Record<string, unknown>>;
expect(list[0].iconCdnUrl).toBeNull();
});
it('flags apps with no owner_user_id as external', async () => {
const user = await seedUser();
const slug = Math.random().toString(36).slice(2, 8);
+44 -22
View File
@@ -2,41 +2,65 @@ import type { Request, Response } from 'express';
import { Context } from '@heyputer/backend/src/core';
import { HttpError } from '@heyputer/backend/src/core/http';
import { extension } from '@heyputer/backend/src/extensions';
import {
INSTALLED_APPS_ORDERS,
type InstalledAppsOrder,
} from '@heyputer/backend/src/stores/permission/PermissionStore.js';
import {
getAppIconCdnUrl,
getAppIconUrl,
} from '@heyputer/backend/src/util/appIcon.js';
import { getAppIconUrl } from '@heyputer/backend/src/util/appIcon.js';
const stores = extension.import('store');
const clients = extension.import('client');
const ALLOWED_ORDER_BY = [
'id',
'name',
'uid',
'title',
'installed_at',
] as const;
const ORDER_BY_FIELD_MAP: Record<string, string> = {
id: 'apps.id',
name: 'apps.name',
uid: 'apps.uid',
title: 'apps.title',
installed_at: 'installed_at',
};
export const handleInstalledApps = async (
req: Request,
res: Response,
): Promise<void> => {
// Behind `requireUserActor`, so there is always an actor.
const actor = Context.get('actor')!;
const actor = Context.get('actor');
if (!actor?.user?.id) throw new HttpError(401, 'Authentication required');
const orderBy = String(
req.query.orderBy ?? 'installed_at',
) as InstalledAppsOrder;
if (!INSTALLED_APPS_ORDERS.includes(orderBy)) {
const orderBy = String(req.query.orderBy ?? 'installed_at');
if (!(ALLOWED_ORDER_BY as readonly string[]).includes(orderBy)) {
throw new HttpError(
400,
`Invalid orderBy. Allowed: ${INSTALLED_APPS_ORDERS.join(', ')}`,
`Invalid orderBy. Allowed: ${ALLOWED_ORDER_BY.join(', ')}`,
);
}
const page = Math.max(Number(req.query.page) || 1, 1);
const limit = Math.min(Math.max(Number(req.query.limit) || 100, 1), 100);
const offset = (page - 1) * limit;
const installedApps = await stores.permission.listAppsGrantedByUser(
actor.user.id!,
{ orderBy, descending: Boolean(req.query.desc), limit, offset },
);
const orderByField = ORDER_BY_FIELD_MAP[orderBy];
const sortDirection = req.query.desc ? 'DESC' : 'ASC';
const installedApps = (await clients.db.read(
`SELECT
apps.name,
apps.uid,
apps.title,
apps.description,
apps.icon,
apps.index_url,
apps.owner_user_id,
MIN(perm.dt) AS installed_at
FROM apps
LEFT JOIN user_to_app_permissions AS perm ON apps.id = perm.app_id
WHERE perm.user_id = ?
GROUP BY apps.id, apps.name, apps.uid, apps.title, apps.description
ORDER BY ${orderByField} ${sortDirection}
LIMIT ?
OFFSET ?`,
[actor.user.id, limit, offset],
)) as Array<Record<string, unknown>>;
const apiBaseUrl = extension.config.api_base_url as string | undefined;
res.json(
@@ -49,8 +73,6 @@ export const handleInstalledApps = async (
return {
...rest,
iconUrl: getAppIconUrl(app, { apiBaseUrl }),
// Direct subdomain URL for the client to try before iconUrl.
iconCdnUrl: getAppIconCdnUrl(app, extension.config),
external,
};
}),
+21 -202
View File
@@ -8,7 +8,6 @@ import {
it,
vi,
} from 'vitest';
import { makeActor, type Actor } from '../src/backend/core/actor.ts';
import { runWithContext } from '../src/backend/core/context.ts';
import { PuterServer } from '../src/backend/server.ts';
import { setupTestServer } from '../src/backend/testUtil.ts';
@@ -63,6 +62,15 @@ const seedUser = async () => {
};
describe('metering extension — handleMeteringUsage', () => {
it('throws HttpError(401) when no user actor is on the context', async () => {
const { res } = makeRes();
await expect(
runWithContext({ actor: undefined }, () =>
handleMeteringUsage(makeReq(), res),
),
).rejects.toMatchObject({ statusCode: 401 });
});
it('returns usage details merged with allowanceInfo for an authenticated user', async () => {
const user = await seedUser();
const { res, captured } = makeRes();
@@ -80,83 +88,21 @@ describe('metering extension — handleMeteringUsage', () => {
(captured.body as Record<string, unknown>).allowanceInfo,
).toBeDefined();
});
describe('usage split across apps', () => {
const seedUsage = async () => {
const user = await seedUser();
const owner = { uuid: user.uuid, id: user.id as number };
const metering = server.services.metering;
await metering.incrementUsage(
makeActor({ user: owner, app: { uid: 'app-mine' } }),
'kv:read',
1,
10,
);
await metering.incrementUsage(
makeActor({ user: owner, app: { uid: 'app-other' } }),
'kv:write',
1,
20,
);
await metering.incrementUsage(
makeActor({ user: owner }),
'kv:list',
1,
40,
);
await server.stores.meteringBuffer.flushCycle();
return owner;
};
const usageAs = async (actor: Actor) => {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleMeteringUsage(makeReq(), res),
);
return captured.body as {
usage: Record<string, unknown> & { total: number };
appTotals: Record<string, unknown>;
allowanceInfo: Record<string, unknown>;
};
};
it('reports only the calling app’s own usage to an app actor', async () => {
const owner = await seedUsage();
const body = await usageAs(
makeActor({ user: owner, app: { uid: 'app-mine' } }),
);
expect(body.usage.total).toBe(10);
expect(Object.keys(body.usage).sort()).toEqual([
'kv:read',
'total',
]);
expect(body.appTotals).toEqual({
'app-mine': { total: 10, count: 1 },
});
expect(body.allowanceInfo).toMatchObject({
remaining: expect.any(Number),
monthUsageAllowance: expect.any(Number),
});
});
it('still reports the whole account to the user', async () => {
const owner = await seedUsage();
const body = await usageAs(makeActor({ user: owner }));
expect(body.usage.total).toBe(70);
expect(body.usage).toHaveProperty('kv:read');
expect(body.usage).toHaveProperty('kv:write');
expect(body.usage).toHaveProperty('kv:list');
expect(body.appTotals).toMatchObject({
'app-mine': { total: 10, count: 1 },
'app-other': { total: 20, count: 1 },
});
});
});
});
describe('metering extension — handleMeteringUsageForApp', () => {
it('throws HttpError(401) when no user actor is on the context', async () => {
const { res } = makeRes();
await expect(
runWithContext({ actor: undefined }, () =>
handleMeteringUsageForApp(
makeReq({ params: { appIdOrName: 'any' } }),
res,
),
),
).rejects.toMatchObject({ statusCode: 401 });
});
it('throws HttpError(400) when no appId is supplied', async () => {
const user = await seedUser();
const { res } = makeRes();
@@ -186,96 +132,6 @@ describe('metering extension — handleMeteringUsageForApp', () => {
),
).rejects.toMatchObject({ statusCode: 404 });
});
describe('scoped to the calling app', () => {
const appUsageAs = async (actor: Actor, appIdOrName: string) => {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleMeteringUsageForApp(
makeReq({ params: { appIdOrName } }),
res,
),
);
return captured.body as { total: number };
};
const seedApps = async () => {
const user = await seedUser();
const owner = { uuid: user.uuid, id: user.id as number };
const mine = `app-${uuidv4()}`;
const other = `app-${uuidv4()}`;
await server.services.metering.incrementUsage(
makeActor({ user: owner, app: { uid: mine } }),
'kv:read',
1,
10,
);
await server.services.metering.incrementUsage(
makeActor({ user: owner, app: { uid: other } }),
'kv:write',
1,
20,
);
await server.stores.meteringBuffer.flushCycle();
return {
owner,
mine,
other,
asMine: makeActor({ user: owner, app: { uid: mine } }),
};
};
it('lets an app read its own usage', async () => {
const { mine, asMine } = await seedApps();
expect((await appUsageAs(asMine, mine)).total).toBe(10);
});
it('refuses an app another app’s usage', async () => {
const { other, asMine } = await seedApps();
await expect(appUsageAs(asMine, other)).rejects.toMatchObject({
statusCode: 403,
});
});
it('refuses an app another app’s usage by name', async () => {
const { owner, other, asMine } = await seedApps();
const name = `other-${uuidv4()}`;
await server.clients.db.write(
'INSERT INTO `apps` (`uid`, `name`, `title`, `index_url`, `owner_user_id`) VALUES (?, ?, ?, ?, ?)',
[other, name, name, `https://${name}.example/`, owner.id],
);
await expect(appUsageAs(asMine, name)).rejects.toMatchObject({
statusCode: 403,
});
});
it('refuses an app the usage outside any app', async () => {
const { asMine } = await seedApps();
await expect(appUsageAs(asMine, 'os-global')).rejects.toMatchObject(
{ statusCode: 403 },
);
});
it('still lets the user read any of their apps', async () => {
const { owner, mine, other } = await seedApps();
const asUser = makeActor({ user: owner });
expect((await appUsageAs(asUser, mine)).total).toBe(10);
expect((await appUsageAs(asUser, other)).total).toBe(20);
});
it('lets the user read their usage outside any app', async () => {
const { owner } = await seedApps();
const asUser = makeActor({ user: owner });
await server.services.metering.incrementUsage(
asUser,
'kv:read',
1,
5,
);
await server.stores.meteringBuffer.flushCycle();
expect((await appUsageAs(asUser, 'os-global')).total).toBe(5);
});
});
});
describe('metering extension — handleMeteringGlobalUsage', () => {
@@ -309,41 +165,4 @@ describe('metering extension — handleMeteringAllCosts', () => {
// per request).
expect(costsA).toBe(costsB);
});
it('refreshes a warm catalogue at the UTC price boundary', async () => {
vi.useFakeTimers({ toFake: ['Date'] });
try {
vi.setSystemTime(new Date('2026-11-30T23:59:59Z'));
const before = makeRes();
await handleMeteringAllCosts(makeReq(), before.res);
vi.setSystemTime(new Date('2026-12-01T00:00:00Z'));
const after = makeRes();
await handleMeteringAllCosts(makeReq(), after.res);
expect(
(after.captured.body as { costs: unknown[] }).costs,
).not.toBe((before.captured.body as { costs: unknown[] }).costs);
} finally {
vi.useRealTimers();
}
});
// A scheduled price change has to reach the catalogue without a restart.
it('collects the catalogue again once the cache expires', async () => {
vi.useFakeTimers({ toFake: ['Date'] });
try {
vi.setSystemTime(new Date('2026-12-02T00:00:00Z'));
const a = makeRes();
const b = makeRes();
await handleMeteringAllCosts(makeReq(), a.res);
vi.setSystemTime(Date.now() + 10 * 60 * 1000 + 1);
await handleMeteringAllCosts(makeReq(), b.res);
const costsA = (a.captured.body as { costs: unknown[] }).costs;
const costsB = (b.captured.body as { costs: unknown[] }).costs;
expect(costsB).not.toBe(costsA);
expect(costsB).toEqual(costsA);
} finally {
vi.useRealTimers();
}
});
});
+23 -127
View File
@@ -6,21 +6,14 @@ import {
servicesContainers,
} from '@heyputer/backend/src/exports';
import { extension } from '@heyputer/backend/src/extensions';
import { GLOBAL_APP_KEY } from '@heyputer/backend/src/services/metering/consts';
import {
creditMultiplierFrom,
toCredits,
} from '@heyputer/backend/src/services/metering/utils';
import type { Request, Response } from 'express';
const services = extension.import('service');
const stores = extension.import('store');
const clients = extension.import('client');
// Catalogues change on deploy or on a scheduled price change, so the cache
// expires rather than living as long as the process.
const ALL_COSTS_TTL_MS = 10 * 60 * 1000;
// Cached on first request — the underlying cost catalogues are baked into
// driver/controller source so they only change on deploy.
let cachedAllCosts: Record<string, unknown>[] | null = null;
let allCostsExpireAt = 0;
function collectAllCosts(): Record<string, unknown>[] {
const all: Record<string, unknown>[] = [];
@@ -57,109 +50,41 @@ function collectAllCosts(): Record<string, unknown>[] {
return all;
}
// -- Credits scaling ---------------------------------------------------
//
// When the deployment configures a credit multiplier, every monetary amount
// is scaled to display credits BEFORE it leaves the server — raw metered
// amounts are never reported. Counts, units, and byte figures pass through
// untouched. A response that was scaled says so (`unit: 'credits'`); without
// a configured multiplier amounts pass through raw and clients render them
// as dollars.
/** The per-usage-type records of a UsageByType map, costs scaled. */
const scaleUsageByType = (
usage: Record<string, unknown>,
multiplier: number,
): Record<string, unknown> => {
const out: Record<string, unknown> = {};
for (const [key, value] of Object.entries(usage)) {
// `allowanceUsed` is the month's allowance-charged spend — monetary,
// scalar, scaled like the total.
if (key === 'total' || key === 'allowanceUsed') {
out[key] = toCredits(Number(value) || 0, multiplier);
} else if (
value &&
typeof value === 'object' &&
typeof (value as { cost?: unknown }).cost === 'number'
) {
out[key] = {
...(value as Record<string, unknown>),
cost: toCredits((value as { cost: number }).cost, multiplier),
};
} else {
out[key] = value;
}
}
return out;
};
export const handleMeteringUsage = async (
_req: Request,
res: Response,
): Promise<void> => {
// Behind `requireAuth`, so there is always an actor.
const actor = Context.get('actor')!;
const actor = Context.get('actor');
if (!actor?.user) throw new HttpError(401, 'Authentication required');
const [actorUsage, allowanceInfo] = await Promise.all([
services.metering.getActorCurrentMonthUsageDetails(actor),
services.metering.getAllowedUsage(actor),
]);
const multiplier = creditMultiplierFrom(extension.config);
if (!multiplier) {
res.json({ ...actorUsage, allowanceInfo });
return;
}
res.json({
usage: scaleUsageByType(
actorUsage.usage as unknown as Record<string, unknown>,
multiplier,
),
appTotals: Object.fromEntries(
Object.entries(actorUsage.appTotals).map(([appId, totals]) => [
appId,
{ ...totals, total: toCredits(totals.total, multiplier) },
]),
),
allowanceInfo: {
...allowanceInfo,
remaining: toCredits(allowanceInfo.remaining, multiplier),
monthUsageAllowance: toCredits(
allowanceInfo.monthUsageAllowance,
multiplier,
),
// Monetary addon fields scale; absent ones stay absent rather
// than materializing as NaN/null. purchasedStorage is bytes.
addons: Object.fromEntries(
Object.entries(allowanceInfo.addons ?? {}).map(([k, v]) => [
k,
(k === 'purchasedCredits' ||
k === 'consumedPurchaseCredits') &&
typeof v === 'number'
? toCredits(v, multiplier)
: v,
]),
),
unit: 'credits',
},
});
res.json({ ...actorUsage, allowanceInfo });
};
export const handleMeteringUsageForApp = async (
req: Request,
res: Response,
): Promise<void> => {
// Behind `requireAuth`, so there is always an actor.
const actor = Context.get('actor')!;
const actor = Context.get('actor');
if (!actor?.user) throw new HttpError(401, 'Authentication required');
let appId = String(req.params.appIdOrName ?? '');
if (!appId) throw new HttpError(400, 'appId parameter is required');
// If not a UUID-shaped app UID or the global sentinel, look up by name.
// Which apps an actor may read is MeteringService's call, not this route's.
if (!appId.startsWith('app-') && appId !== GLOBAL_APP_KEY) {
const app = await stores.app.getByName(appId);
if (!app) throw new HttpError(404, 'App not found');
appId = app.uid;
// If not a UUID-shaped app UID, look up by name
if (!appId.startsWith('app-')) {
const appRows = (await clients.db.read(
'SELECT `uid` FROM `apps` WHERE `name` = ? LIMIT 1',
[appId],
)) as Array<{ uid: string }>;
if (appRows.length > 0) {
appId = appRows[0].uid;
} else {
throw new HttpError(404, 'App not found');
}
}
const appUsage =
@@ -167,18 +92,7 @@ export const handleMeteringUsageForApp = async (
actor,
appId,
);
const multiplier = creditMultiplierFrom(extension.config);
if (!multiplier) {
res.json(appUsage);
return;
}
res.json({
...scaleUsageByType(
appUsage as unknown as Record<string, unknown>,
multiplier,
),
unit: 'credits',
});
res.json(appUsage);
};
export const handleMeteringGlobalUsage = async (
@@ -189,31 +103,13 @@ export const handleMeteringGlobalUsage = async (
res.json(globalUsage);
};
// A hit walks the registries when the in-memory cache is missing or expired.
// First hit walks the registries; subsequent hits serve the in-memory cache.
export const handleMeteringAllCosts = async (
_req: Request,
res: Response,
): Promise<void> => {
if (!cachedAllCosts || Date.now() >= allCostsExpireAt) {
allCostsExpireAt =
(Math.floor(Date.now() / ALL_COSTS_TTL_MS) + 1) * ALL_COSTS_TTL_MS;
const raw = collectAllCosts();
const multiplier = creditMultiplierFrom(extension.config);
// Price the catalogue in the same unit the usage endpoints report:
// per-unit credits when a multiplier is configured, raw otherwise.
cachedAllCosts = multiplier
? raw.map(({ ucentsPerUnit, costs_currency: _cc, ...rest }) =>
typeof ucentsPerUnit === 'number'
? {
...rest,
creditsPerUnit: toCredits(
ucentsPerUnit,
multiplier,
),
}
: rest,
)
: raw;
if (!cachedAllCosts) {
cachedAllCosts = collectAllCosts();
}
res.json({ costs: cachedAllCosts });
};
File diff suppressed because it is too large. Load diff
+88 -211
View File
@@ -1,21 +1,15 @@
import {
CopyObjectCommand,
DeleteObjectCommand,
GetObjectCommand,
HeadObjectCommand,
PutObjectCommand,
S3Client,
} from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { extension } from '@heyputer/backend/src/extensions';
import type { FSEntryStore } from '@heyputer/backend/src/stores/fs/FSEntryStore';
import { isMissingObjectError } from '@heyputer/backend/src/stores/fs/S3ObjectStore';
import crypto from 'node:crypto';
import type { Readable } from 'node:stream';
import sharp from 'sharp';
const clients = extension.import('client');
const stores = extension.import('store');
type ThumbnailEntryStore = Pick<FSEntryStore, 'updateEntryThumbnailByUuid'>;
const MAX_THUMBNAIL_BYTES = 2 * 1024 * 1024;
const MAX_THUMBNAIL_PIXELS = 64e6;
@@ -28,13 +22,8 @@ const THUMBNAIL_KEY_PREFIX = 'thumbnails/';
const UUID_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
const isUuid = (value: unknown): value is string =>
typeof value === 'string' && UUID_PATTERN.test(value);
// `thumbnails/<entry uuid>/<random uuid>`: the entry segment binds the object
// to the entry it was minted for, which is what writes and deletes check.
const mintThumbnailKey = (entryUuid: string): string =>
`${THUMBNAIL_KEY_PREFIX}${entryUuid}/${crypto.randomUUID()}`;
const mintThumbnailKey = (): string =>
`${THUMBNAIL_KEY_PREFIX}${crypto.randomUUID()}`;
/**
* Extract the object key from a stored thumbnail pointer, or null when the
@@ -42,13 +31,12 @@ const mintThumbnailKey = (entryUuid: string): string =>
*
* `fsentries.thumbnail` is writable through the FS API, so neither half of the
* stored string is trusted: the bucket is discarded (callers always pass their
* own) and the key must sit under {@link THUMBNAIL_KEY_PREFIX} as either
* `<uuid>` (minted before keys were bound to entries) or `<entry uuid>/<uuid>`.
* Honouring an arbitrary key would lend this extension's storage credentials to
* whatever object the caller named — in the shared-bucket layout that is every
* user's file, since an fs object's key is its fsentry uuid. Bare-uuid keys
* from before the prefix fail the check and are treated as absent; they are
* indistinguishable from a planted pointer, so there is nothing safer to do
* own) and the key must sit under {@link THUMBNAIL_KEY_PREFIX} with a random
* uuid. Honouring an arbitrary key would lend this extension's storage
* credentials to whatever object the caller named — in the shared-bucket layout
* that is every user's file, since an fs object's key is its fsentry uuid.
* Legacy bare-uuid thumbnails fail the check and are treated as absent; they
* are indistinguishable from a planted pointer, so there is nothing safer to do
* with them than stop signing them.
*/
const resolveThumbnailKey = (pointer: string): string | null => {
@@ -70,26 +58,10 @@ const resolveThumbnailKey = (pointer: string): string | null => {
key = segments.join('/');
}
if (!key.startsWith(THUMBNAIL_KEY_PREFIX)) return null;
const segments = key.slice(THUMBNAIL_KEY_PREFIX.length).split('/');
return segments.length <= 2 && segments.every(isUuid) ? key : null;
const id = key.slice(THUMBNAIL_KEY_PREFIX.length);
return UUID_PATTERN.test(id) ? key : null;
};
// The entry a resolved key was minted for. Null for an unbound key, whose
// holder can't be told apart from a row that copied or planted the pointer.
const thumbnailKeyOwner = (key: string): string | null => {
const segments = key.slice(THUMBNAIL_KEY_PREFIX.length).split('/');
return segments.length === 2 ? segments[0] : null;
};
// Whether a stored thumbnail names an object in our storage. Anything else is
// an external image URL the client supplied: shown as given, never copied or
// deleted.
const isStoragePointer = (pointer: string, bucketEndpoint: string): boolean =>
pointer.startsWith('s3://') ||
// Legacy format — remove after full migration
(pointer.startsWith('https') &&
pointer.includes(new URL(bucketEndpoint).hostname));
// S3 client + bucket config — lazily resolved after boot from config.
let s3Client: S3Client | null = null;
let s3PresignClient: S3Client | null = null;
@@ -139,6 +111,13 @@ function getPresignClient(): S3Client {
return resolveClients().presign;
}
function base64ParseDataUrl(dataURL: string) {
dataURL = dataURL.slice(5);
const mimeType = dataURL.split(';')[0];
const data = Buffer.from(dataURL.split(',')[1], 'base64');
return { mimeType, data };
}
// Strictly decode a data: URL and validate the decoded image. Encoded-string
// length lies about decoded byte count (whitespace, padding) and says nothing
// about pixel count — a 2MB PNG can decompress to hundreds of MB of raster.
@@ -165,73 +144,16 @@ async function decodeAndValidateThumbnail(
return { mimeType, data };
}
// A storage pointer a client hands back after a presigned upload is kept only
// if it names a key minted for this entry and an object within the size bound.
async function vetUploadedThumbnail(
pointer: string,
entryUuid: unknown,
deps: { s3: S3Client; bucketName: string },
): Promise<string | null> {
const key = resolveThumbnailKey(pointer);
if (!key || thumbnailKeyOwner(key) !== entryUuid) return null;
let size: number | undefined;
try {
const head = await deps.s3.send(
new HeadObjectCommand({ Bucket: deps.bucketName, Key: key }),
);
size = head.ContentLength;
} catch (err) {
if (!isMissingObjectError(err)) {
console.warn(
'[thumbnails] failed to check uploaded thumbnail',
err,
);
}
return null;
}
if (typeof size !== 'number') return null;
if (size > MAX_THUMBNAIL_BYTES) {
// Only reachable on a store that doesn't enforce the signed length.
// The key is this entry's own, so removing it is safe.
try {
await deps.s3.send(
new DeleteObjectCommand({ Bucket: deps.bucketName, Key: key }),
);
} catch (err) {
console.warn(
'[thumbnails] failed to remove oversized thumbnail',
err,
);
}
return null;
}
return `s3://${deps.bucketName}/${key}`;
}
// -- thumbnail.created -----------------------------------------------
// Intercept data-URL thumbnails before they hit the DB: upload to S3
// and replace the URL with an s3:// pointer bound to the entry.
// and replace the URL with an s3:// pointer.
export async function handleThumbnailCreated(
event: Record<string, unknown>,
deps: { s3: S3Client; bucketName: string; bucketEndpoint: string },
deps: { s3: S3Client; bucketName: string },
): Promise<void> {
const url = event.url;
if (typeof url !== 'string') return;
if (!url.startsWith('data:')) {
if (isStoragePointer(url, deps.bucketEndpoint)) {
event.url = await vetUploadedThumbnail(url, event.uuid, deps);
}
return;
}
const entryUuid = event.uuid;
if (!isUuid(entryUuid)) {
event.url = null;
return;
}
if (typeof url !== 'string' || !url.startsWith('data:')) return;
const decoded = await decodeAndValidateThumbnail(url);
if (!decoded) {
@@ -239,7 +161,7 @@ export async function handleThumbnailCreated(
return;
}
const key = mintThumbnailKey(entryUuid);
const key = mintThumbnailKey();
event.url = `s3://${deps.bucketName}/${key}`;
await deps.s3.send(
@@ -268,24 +190,21 @@ export const handleThumbnailUploadPrepare = async (
typeof item.contentType === 'string' ? item.contentType.trim() : '';
if (!contentType) continue;
const size = item.size;
if (
typeof size !== 'number' ||
!Number.isInteger(size) ||
size < 0 ||
size > MAX_THUMBNAIL_BYTES
)
continue;
if (!isUuid(item.item_uid)) continue;
if (item.size !== undefined) {
const size = Number(item.size);
if (
!Number.isFinite(size) ||
size < 0 ||
size > MAX_THUMBNAIL_BYTES
)
continue;
}
const key = mintThumbnailKey(item.item_uid);
const key = mintThumbnailKey();
const command = new PutObjectCommand({
Bucket: deps.bucketName,
Key: key,
ContentType: contentType,
// Signing covers `content-length`, so a body of any other size
// fails the signature instead of landing.
ContentLength: size,
});
item.uploadUrl = await getSignedUrl(presignClient, command, {
expiresIn: 900,
@@ -301,17 +220,19 @@ export const handleThumbnailRead = async (
s3Presign: S3Client;
bucketName: string;
bucketEndpoint: string;
fsEntry: ThumbnailEntryStore;
db: { write: (sql: string, params: unknown[]) => Promise<unknown> };
},
): Promise<void> => {
const thumb = entry.thumbnail;
if (typeof thumb !== 'string' || !thumb) return;
const presignClient = deps.s3Presign;
const signingDate = new Date(
Math.floor(Date.now() / 3_600_000) * 3_600_000,
);
if (isStoragePointer(thumb, deps.bucketEndpoint)) {
if (
thumb.startsWith('s3://') ||
// Legacy format — remove after full migration
(thumb.startsWith('https') &&
thumb.includes(new URL(deps.bucketEndpoint).hostname))
) {
const key = resolveThumbnailKey(thumb);
if (!key) {
// Not a pointer we minted — refuse to sign it rather than hand
@@ -322,61 +243,40 @@ export const handleThumbnailRead = async (
entry.thumbnail = await getSignedUrl(
presignClient,
new GetObjectCommand({ Bucket: deps.bucketName, Key: key }),
{ expiresIn: 604800, signingDate },
{ expiresIn: 604800 },
);
} else if (thumb.startsWith('data')) {
// Move an inline thumbnail into storage. The row only takes the new
// pointer while it still holds this data URL, so of concurrent reads
// one wins and the rest discard their upload.
const uuid = entry.uuid ?? entry.uid;
if (!isUuid(uuid)) return;
// Same bar as an upload; anything else stays inline, unmigrated.
const decoded = await decodeAndValidateThumbnail(thumb);
if (!decoded) return;
const key = mintThumbnailKey(uuid);
// Inline data-URL migration: upload to S3 and update the DB entry.
const key = mintThumbnailKey();
const { mimeType, data } = base64ParseDataUrl(thumb);
const newUrl = `s3://${deps.bucketName}/${key}`;
await deps.s3.send(
new PutObjectCommand({
Bucket: deps.bucketName,
Key: key,
Body: decoded.data,
ContentType: decoded.mimeType,
Body: data,
ContentType: mimeType,
}),
);
let migrated: boolean;
try {
migrated = await deps.fsEntry.updateEntryThumbnailByUuid(
uuid,
`s3://${deps.bucketName}/${key}`,
thumb,
);
} catch (err) {
// The row may or may not point at the object now, so keep it.
console.warn('[thumbnails] inline migration failed', err);
return;
}
if (!migrated) {
try {
await deps.s3.send(
new DeleteObjectCommand({
Bucket: deps.bucketName,
Key: key,
}),
// Best-effort async DB update
const uuid = entry.uuid ?? entry.uid;
if (uuid) {
deps.db
.write(
'UPDATE `fsentries` SET `thumbnail` = ? WHERE `uuid` = ?',
[newUrl, uuid],
)
.catch((err: unknown) =>
console.warn('[thumbnails] inline migration failed', err),
);
} catch (err) {
console.warn(
'[thumbnails] failed to remove unused thumbnail',
err,
);
}
return;
}
entry.thumbnail = await getSignedUrl(
presignClient,
new GetObjectCommand({ Bucket: deps.bucketName, Key: key }),
{ expiresIn: 604800, signingDate },
{ expiresIn: 604800 },
);
}
};
@@ -386,83 +286,63 @@ export const handleFsCopyNodeThumbnail = async (
deps: {
s3: S3Client;
bucketName: string;
bucketEndpoint: string;
fsEntry: ThumbnailEntryStore;
db: { write: (sql: string, params: unknown[]) => Promise<unknown> };
},
): Promise<void> => {
const copy = payload.copy;
const thumbnailUrl = copy?.thumbnail;
if (!copy || !isUuid(copy.uuid) || typeof thumbnailUrl !== 'string') return;
if (!isStoragePointer(thumbnailUrl, deps.bucketEndpoint)) return;
if (!copy || !copy.uuid || typeof thumbnailUrl !== 'string') return;
// Same trust rule as the read and remove paths: only touch objects this
// extension minted.
const sourceKey = resolveThumbnailKey(thumbnailUrl);
if (!sourceKey) return;
// The copied row points at the SAME object as its source. Give the copy
// an object of its own, bound to it, so removing either entry leaves the
// other's thumbnail alone.
const newKey = mintThumbnailKey(copy.uuid);
// The copied row points at the SAME S3 object as its source, and
// fs.remove.node deletes the pointed-to object — so the first removal
// among the sharers (an overwrite, a trash purge) would break every
// other sharer's thumbnail. Give the copy an object of its own.
const newKey = mintThumbnailKey();
try {
// Nothing larger than an upload may produce gets duplicated; such an
// object came in through an unbounded upload URL.
const head = await deps.s3.send(
new HeadObjectCommand({ Bucket: deps.bucketName, Key: sourceKey }),
);
if (
typeof head.ContentLength !== 'number' ||
head.ContentLength > MAX_THUMBNAIL_BYTES
) {
throw new Error('thumbnail exceeds the size bound');
}
// Read and rewrite rather than CopyObject: a copy source skips any path
// in the client's endpoint, so it can miss an object Bucket/Key reach.
const source = await deps.s3.send(
new GetObjectCommand({ Bucket: deps.bucketName, Key: sourceKey }),
);
// The object can be replaced after the HEAD; bound what gets buffered.
if ((source.ContentLength ?? 0) > MAX_THUMBNAIL_BYTES) {
(source.Body as Readable | undefined)?.destroy();
throw new Error('thumbnail exceeds the size bound');
}
const body = await source.Body?.transformToByteArray();
if (!body) throw new Error('thumbnail has no body');
await deps.s3.send(
new PutObjectCommand({
new CopyObjectCommand({
Bucket: deps.bucketName,
CopySource: `${deps.bucketName}/${sourceKey}`,
Key: newKey,
Body: body,
ContentType: source.ContentType,
}),
);
} catch (err) {
// The shared object is gone or oversized — drop the pointer rather
// than leave the row advertising a thumbnail it doesn't have.
await deps.fsEntry.updateEntryThumbnailByUuid(copy.uuid, null);
// The shared object is already gone (e.g. a sharer was removed
// before this fix existed) — the pointer is dead either way, so
// drop it rather than leave the row advertising a thumbnail it
// doesn't have.
await deps.db.write(
'UPDATE `fsentries` SET `thumbnail` = NULL WHERE `uuid` = ?',
[copy.uuid],
);
console.warn('[thumbnails] failed to duplicate thumbnail on copy', err);
return;
}
await deps.fsEntry.updateEntryThumbnailByUuid(
copy.uuid,
`s3://${deps.bucketName}/${newKey}`,
await deps.db.write(
'UPDATE `fsentries` SET `thumbnail` = ? WHERE `uuid` = ?',
[`s3://${deps.bucketName}/${newKey}`, copy.uuid],
);
};
export const handleFsRemoveNodeThumbnail = async (
payload: { target: { thumbnail?: string | null; uuid?: string } },
deps: { s3: S3Client; bucketName: string; bucketEndpoint: string },
payload: { target: { thumbnail?: string | null } },
deps: { s3: S3Client; bucketName: string },
): Promise<void> => {
const { thumbnail: thumbnailUrl, uuid } = payload.target;
if (!thumbnailUrl || !isStoragePointer(thumbnailUrl, deps.bucketEndpoint))
return;
const thumbnailUrl = payload.target.thumbnail;
if (!thumbnailUrl) return;
// The pointer decides which object gets deleted, and a key is visible to
// anyone who can list the entry holding it. Only a key minted for this
// entry is deleted; an unbound one may be shared or planted, so it stays.
// Same trust rule as the read path, and load-bearing for the same reason:
// the pointer decides which object gets deleted, so a key we didn't mint
// would let the owner of one file destroy an object belonging to someone
// else just by naming it here.
const key = resolveThumbnailKey(thumbnailUrl);
if (!key || thumbnailKeyOwner(key) !== uuid) return;
if (!key) return;
await deps.s3.send(
new DeleteObjectCommand({ Bucket: deps.bucketName, Key: key }),
@@ -475,7 +355,6 @@ extension.on(
await handleThumbnailCreated(event, {
s3: getClient(),
bucketName: thumbnailBucketName,
bucketEndpoint: extensionBucketEndpoint,
});
},
);
@@ -502,7 +381,7 @@ extension.on('thumbnail.read', async (_key, entry: Record<string, unknown>) => {
s3Presign: getPresignClient(),
bucketName: thumbnailBucketName,
bucketEndpoint: extensionBucketEndpoint,
fsEntry: stores.fsEntry,
db: clients.db,
});
});
@@ -518,8 +397,7 @@ extension.on('fs.copy.node', async (_key, payload) => {
{
s3: getClient(),
bucketName: thumbnailBucketName,
bucketEndpoint: extensionBucketEndpoint,
fsEntry: stores.fsEntry,
db: clients.db,
},
);
});
@@ -531,6 +409,5 @@ extension.on('fs.remove.node', async (_key, payload) => {
await handleFsRemoveNodeThumbnail(payload, {
s3: getClient(),
bucketName: thumbnailBucketName,
bucketEndpoint: extensionBucketEndpoint,
});
});
-355
View File
@@ -1,355 +0,0 @@
import { EventEmitter } from 'node:events';
import { Readable } from 'node:stream';
import type { Request, Response } from 'express';
import { beforeEach, describe, expect, test, vi } from 'vitest';
const SECRET = 'ingress-secret';
// Only `extension` is stubbed; the secret comparison and the routing are the
// real code paths, and the fake service/store layer is the boundary.
const state = vi.hoisted(() => ({
config: { userEmail: { secret: 'ingress-secret' } } as Record<
string,
unknown
>,
routes: [] as Array<{
path: string;
handler: (req: unknown, res: unknown) => unknown;
}>,
users: new Map<string, Record<string, unknown>>(),
writes: [] as Array<{
userId: number;
path: string;
homeRegion?: string;
content?: unknown;
}>,
feedback: [] as Array<{ to: string; from: string | null; raw: Buffer }>,
takeFeedback: true,
}));
vi.mock('@heyputer/backend/src/extensions', () => ({
extension: {
get config() {
return state.config;
},
post(
path: string,
_opts: unknown,
handler: (req: unknown, res: unknown) => unknown,
) {
state.routes.push({ path, handler });
},
import(layer: string) {
if (layer === 'client') {
return {
event: {
async emitAndWait(
key: string,
data: {
to: string;
from: string | null;
raw: Buffer;
handled: boolean;
},
) {
if (key !== 'email.ingress.feedback') return;
state.feedback.push(data);
if (state.takeFeedback) data.handled = true;
},
},
};
}
if (layer === 'store') {
return {
user: {
getByUsername: async (username: string) =>
state.users.get(username) ?? null,
},
};
}
return {
fs: {
async write(
userId: number,
{
fileMetadata,
fileContent,
}: {
fileMetadata: { path: string };
fileContent: unknown;
},
_uploadTracker?: unknown,
_storageAllowanceMax?: number,
homeRegion?: string,
) {
state.writes.push({
userId,
path: fileMetadata.path,
homeRegion,
content: fileContent,
});
return { fsEntry: { path: fileMetadata.path } };
},
},
};
},
},
}));
import './index.js';
const permanent = (id: number, username: string) => ({
id,
username,
email: `${username}@example.com`,
password: '$2b$10$hash',
});
/** No password and no email is what makes an account temporary. */
const temp = (id: number, username: string) => ({
id,
username,
email: null,
password: null,
});
const ingress = async (
to: string,
over: { secret?: string; contentLength?: string | null } = {},
) => {
const route = state.routes.find((r) => r.path === '/email/ingress');
if (!route) throw new Error('route not registered');
const body = Buffer.from('From: [email protected]\r\n\r\nhello\r\n');
const req = Readable.from([body], { objectMode: false }) as unknown as {
headers: Record<string, string>;
query: Record<string, string>;
};
// `contentLength: null` sends no length header at all, which is the one
// framing the HTTP parser does not bound for us.
req.headers =
over.contentLength === null
? {}
: {
'content-length': over.contentLength ?? String(body.length),
};
req.query = {
SECRET: over.secret ?? SECRET,
to,
from: '[email protected]',
subject: 'hi',
};
let status = 200;
let ended = false;
// `end` takes (chunk, encoding) and no callback, the way the compression
// middleware leaves it, and emits `finish` — which is what the refusal
// paths hang their drain/destroy on. A stub that accepts a callback
// instead hides the bug that took the 413 path to production as a 500.
const emitter = new EventEmitter();
const res = {
status(code: number) {
status = code;
return res;
},
end(chunk?: unknown, encoding?: BufferEncoding) {
if (chunk) Buffer.byteLength(chunk as string, encoding);
ended = true;
emitter.emit('finish');
},
once(event: string, listener: () => void) {
emitter.once(event, listener);
return res;
},
} as unknown as Response;
await route.handler(req as unknown as Request, res);
return {
status,
ended,
drained: (req as unknown as Readable).readableFlowing,
destroyed: (req as unknown as Readable).destroyed,
};
};
beforeEach(() => {
state.users.clear();
state.writes.length = 0;
state.feedback.length = 0;
state.takeFeedback = true;
});
describe('temporary accounts cannot receive', () => {
test('mail for a temp account is accepted and dropped', async () => {
state.users.set('tempy', temp(1, 'tempy'));
const { status, ended, drained } = await ingress('[email protected]');
// Accepted, so the relay neither retries nor bounces to the sender.
expect(status).toBe(200);
expect(ended).toBe(true);
// The body is read off the wire rather than left unconsumed.
expect(drained).toBe(true);
expect(state.writes).toEqual([]);
});
test('mail for a permanent account still lands in the mailbox', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status } = await ingress('[email protected]');
expect(status).toBe(200);
expect(state.writes).toHaveLength(1);
expect(state.writes[0].userId).toBe(2);
expect(state.writes[0].path).toMatch(/^\/bob\/\.mail\/objects\//);
});
test('an unknown address is still rejected outright', async () => {
const { status } = await ingress('[email protected]');
expect(status).toBe(404);
expect(state.writes).toEqual([]);
});
});
describe('inbound mail is placed in the recipient home region', () => {
/** A permanent account with explicit placement columns. */
const placed = (
id: number,
username: string,
over: { home?: string | null; signup_server?: string | null },
) => ({
...permanent(id, username),
home: null,
signup_server: null,
...over,
});
test("uses the recipient's own home region when they have one", async () => {
state.users.set('bob', placed(2, 'bob', { home: 'frankfurt' }));
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('frankfurt');
});
test('falls back to the server that served their signup', async () => {
// Every account that predates the `home` column is this case.
state.users.set(
'bob',
placed(2, 'bob', { home: null, signup_server: 'london' }),
);
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('london');
});
test('falls back to the primary region when neither is set', async () => {
state.users.set('bob', placed(2, 'bob', {}));
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('oregon');
});
test('home wins over signup_server when both are set', async () => {
state.users.set(
'bob',
placed(2, 'bob', { home: 'sydney', signup_server: 'london' }),
);
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('sydney');
});
});
describe('refusals dispose of the body nobody read', () => {
test('an unknown recipient drains it and keeps the connection', async () => {
// The secret checked out and the length is under the cap, so the body
// was ours to read: discard it rather than resetting a usable socket.
const { status, drained, destroyed } =
await ingress('[email protected]');
expect(status).toBe(404);
expect(drained).toBe(true);
expect(destroyed).toBe(false);
});
test('a bad secret closes rather than reading the upload', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status, destroyed } = await ingress('[email protected]', {
secret: 'not-the-secret',
});
expect(status).toBe(403);
// Nothing authenticated this caller; their bytes are not worth reading.
expect(destroyed).toBe(true);
expect(state.writes).toEqual([]);
});
test('no content-length is refused with 411', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status, destroyed } = await ingress('[email protected]', {
contentLength: null,
});
expect(status).toBe(411);
expect(destroyed).toBe(true);
expect(state.writes).toEqual([]);
});
test('a declared length over the cap is refused with 413', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status, destroyed } = await ingress('[email protected]', {
contentLength: String(25 * 1024 * 1024 + 1),
});
expect(status).toBe(413);
expect(destroyed).toBe(true);
expect(state.writes).toEqual([]);
});
});
describe('feedback addresses', () => {
test('are handed to a listener and reach no mailbox', async () => {
const { status, ended } = await ingress('[email protected]');
expect(status).toBe(200);
expect(ended).toBe(true);
expect(state.feedback).toHaveLength(1);
expect(state.feedback[0].to).toBe('[email protected]');
expect(state.feedback[0].from).toBe('[email protected]');
expect(state.feedback[0].raw.toString()).toContain('hello');
expect(state.writes).toEqual([]);
});
test('match the local part case-insensitively', async () => {
await ingress('[email protected]');
expect(state.feedback).toHaveLength(1);
});
test('fall through to normal routing when nobody takes them', async () => {
state.takeFeedback = false;
state.users.set('abuse', permanent(3, 'abuse'));
const { status } = await ingress('[email protected]');
expect(status).toBe(200);
expect(state.feedback).toHaveLength(1);
// The body was already read, so the buffered copy is what is stored.
expect(state.writes).toHaveLength(1);
expect(Buffer.isBuffer(state.writes[0].content)).toBe(true);
});
test('an unclaimed one with no such account is still a 404', async () => {
state.takeFeedback = false;
const { status } = await ingress('[email protected]');
expect(status).toBe(404);
});
test('only count on a Puter mail domain', async () => {
state.users.set('abuse', permanent(3, 'abuse'));
await ingress('[email protected]');
expect(state.feedback).toEqual([]);
});
test('are refused over the feedback size cap without being read', async () => {
const { status, destroyed } = await ingress('[email protected]', {
contentLength: String(5 * 1024 * 1024 + 1),
});
expect(status).toBe(413);
expect(destroyed).toBe(true);
expect(state.feedback).toEqual([]);
});
});
-161
View File
@@ -1,161 +0,0 @@
import { extension } from '@heyputer/backend/src/extensions';
import { createHmac, randomBytes, timingSafeEqual } from 'node:crypto';
import type { Request } from 'express';
import {
MAX_MESSAGE_BYTES,
isPuterEmailAddress,
isTempUser,
puterEmailUsername,
storeInboxMessage,
} from '@heyputer/backend/src/services/email/mailbox.js';
// copied form peer secrets checker
const COMPARE_KEY = randomBytes(32);
const secretsEqual = (a: string, b: string): boolean =>
timingSafeEqual(
createHmac('sha256', COMPARE_KEY).update(a).digest(),
createHmac('sha256', COMPARE_KEY).update(b).digest(),
);
const USER_EMAIL_CONFIG = (extension.config as Record<string, unknown>)
.userEmail as { secret?: string; feedbackAddresses?: unknown } | undefined;
const INGRESS_SECRET = USER_EMAIL_CONFIG?.secret;
/**
* Local parts that take complaint feedback rather than naming a mailbox. The
* same names are reserved at signup, so no account can be handed them.
*/
const DEFAULT_FEEDBACK_ADDRESSES = ['fbl', 'abuse', 'postmaster'];
const FEEDBACK_ADDRESSES = new Set(
(Array.isArray(USER_EMAIL_CONFIG?.feedbackAddresses)
? USER_EMAIL_CONFIG.feedbackAddresses
: DEFAULT_FEEDBACK_ADDRESSES
).map((local) => String(local).toLowerCase()),
);
/**
* Feedback is handed over whole, so it is read into memory; a complaint report
* quotes one message and has no business near the mailbox ceiling.
*/
const MAX_FEEDBACK_BYTES = 5 * 1024 * 1024;
const isFeedbackAddress = (to: string): boolean =>
isPuterEmailAddress(to) &&
FEEDBACK_ADDRESSES.has(puterEmailUsername(to).toLowerCase());
const readBody = async (req: Request): Promise<Buffer> => {
const chunks: Buffer[] = [];
for await (const chunk of req) chunks.push(chunk as Buffer);
return Buffer.concat(chunks);
};
if (!INGRESS_SECRET) {
console.warn(
'[useremailrx] config.userEmail.secret unset - email ingress disabled',
);
} else {
extension.post(
'/email/ingress',
{
subdomain: 'api',
},
async (req, res) => {
// Answer first, then deal with the body nobody read. Touching the
// request before the response has flushed takes the response down
// with the connection and the caller sees a reset with no status,
// so this hangs off `finish` — and it has to be the event rather
// than `res.end(cb)`, because the compression middleware replaces
// `res.end` with an `(chunk, encoding)` signature that has no
// callback parameter, so a callback lands in `chunk` and dies in
// `Buffer.byteLength`.
//
// `drain` discards the rest of a body we were entitled to read and
// leaves the connection reusable. `close` is for bodies we do not
// want to read at all: an unauthenticated caller's, or one whose
// declared length is missing or over the cap.
const refuse = (status: number, how: 'drain' | 'close') => {
res.once('finish', () =>
how === 'drain' ? req.resume() : req.destroy(),
);
res.status(status).end();
};
const presented = req.query.SECRET;
if (
typeof presented !== 'string' ||
!secretsEqual(presented, INGRESS_SECRET)
) {
// Nothing has authenticated this caller, so refuse to spend
// bandwidth reading whatever they were uploading.
refuse(403, 'close');
return;
}
const size = Number(req.headers['content-length']);
if (!Number.isInteger(size) || size <= 0) {
refuse(411, 'close');
return;
}
if (size > MAX_MESSAGE_BYTES) {
refuse(413, 'close');
return;
}
const stores = extension.import('store');
const services = extension.import('service');
const to = String(req.query.to ?? '');
let content: Buffer | Request = req;
if (isFeedbackAddress(to)) {
if (size > MAX_FEEDBACK_BYTES) {
refuse(413, 'close');
return;
}
const raw = await readBody(req);
const event = {
to,
from:
typeof req.query.from === 'string'
? req.query.from
: null,
raw,
handled: false,
};
await extension
.import('client')
.event.emitAndWait('email.ingress.feedback', event, {});
if (event.handled) {
res.end();
return;
}
// Nobody took it: route it like any other message.
content = raw;
}
const user = await stores.user.getByUsername(
puterEmailUsername(to),
);
if (!user) {
refuse(404, 'drain');
return;
}
if (isTempUser(user)) {
if (content === req) req.resume();
res.end();
return;
}
await storeInboxMessage(services.fs, user, {
subject: req.query.subject,
content,
size: Buffer.isBuffer(content) ? content.length : size,
});
const { SECRET: _secret, ..._loggableQuery } = req.query;
// console.log('got email: ', puterUser, loggableQuery, size);
res.end();
},
);
}
-12
View File
@@ -1,12 +0,0 @@
{
"name": "emailrx",
"version": "1.0.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "emailrx",
"version": "1.0.0"
}
}
}
-6
View File
@@ -1,6 +0,0 @@
{
"name": "emailrx",
"version": "1.0.0",
"main": "index.ts",
"description": "Internal service which handles routing puter emails to users"
}
+18 -409
View File
@@ -1,10 +1,14 @@
import type { Request, Response } from 'express';
import { installSeatStub } from '@heyputer/backend/testFixtures/seatStub';
import { v4 as uuidv4 } from 'uuid';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { makeActor, type Actor } from '../src/backend/core/actor.ts';
import {
afterAll,
beforeAll,
describe,
expect,
it,
vi,
} from 'vitest';
import { runWithContext } from '../src/backend/core/context.ts';
import { configContainer } from '../src/backend/exports.ts';
import { PuterServer } from '../src/backend/server.ts';
import { setupTestServer } from '../src/backend/testUtil.ts';
import { handleWhoami } from './whoami.ts';
@@ -61,78 +65,6 @@ const seedUser = async () => {
};
describe('whoami extension — handleWhoami', () => {
// The sidebar label needs this at boot, which is why it rides whoami
// rather than a call of its own.
describe('the team an account belongs to', () => {
// A seat is created, never adopted, so it must have no password.
const seedSeat = async () => {
const slug = Math.random().toString(36).slice(2, 8);
return server.stores.user.create({
username: `wseat_${slug}`,
uuid: uuidv4(),
password: null,
email: null,
});
};
const seatOf = async (teamName: string) => {
const seat = await seedSeat();
const team = {
uid: `wt-${Math.random().toString(36).slice(2, 9)}`,
name: teamName,
};
installSeatStub(server as never, [seat.id as number], {
teamUid: team.uid,
teamName: team.name,
});
return { seat, team };
};
it('names no team where teams is off', async () => {
const { seat } = await seatOf('Acme Corp');
const { res, captured } = makeRes();
await runWithContext(
{ actor: { user: { uuid: seat.uuid, id: seat.id as number } } },
() => handleWhoami(makeReq(), res),
);
// Off, so the seat is never looked up.
expect((captured.body as { team?: unknown }).team).toBeUndefined();
});
it('says nothing for an account that is not a seat', async () => {
const user = await seedUser();
const { res, captured } = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleWhoami(makeReq(), res),
);
expect(captured.body).not.toHaveProperty('team');
});
it('withholds it from an app actor', async () => {
// Same class as the phone number: a seat's employer is not an
// app's business.
const { seat } = await seatOf('Acme Corp');
const { res, captured } = makeRes();
await runWithContext(
{
actor: makeActor({
user: { uuid: seat.uuid, id: seat.id as number },
app: { uid: 'app-1' },
}),
},
() => handleWhoami(makeReq(), res),
);
expect(captured.body).not.toHaveProperty('team');
});
});
it('returns 401 when no actor is on the context', async () => {
const { res, captured } = makeRes();
@@ -160,81 +92,6 @@ describe('whoami extension — handleWhoami', () => {
expect(captured.body).toEqual({ error: 'User not found' });
});
describe("an app's view of the email", () => {
const appActorFor = async (user: { uuid: string; id: unknown }) => {
const app = await (
server.stores.app.create as unknown as (
fields: Record<string, unknown>,
opts: { ownerUserId: number },
) => Promise<{ uid: string; id: number }>
)(
{
name: `we-${uuidv4()}`,
title: 'Email reader',
index_url: `https://${uuidv4()}.example/`,
},
{ ownerUserId: user.id as number },
);
return {
app,
actor: makeActor({
user: { uuid: user.uuid, id: user.id as number },
app: { uid: app.uid, id: app.id },
}),
};
};
const whoamiAs = async (actor: Actor) => {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleWhoami(makeReq(), res),
);
return captured;
};
it('is withheld until the user grants it, then shown', async () => {
const user = await seedUser();
const { app, actor } = await appActorFor(user);
const before = (await whoamiAs(actor)).body as Record<
string,
unknown
>;
expect(before).not.toHaveProperty('email');
expect(before).not.toHaveProperty('unconfirmed_email');
await server.services.permission.grantUserAppPermission(
makeActor({ user: { uuid: user.uuid, id: user.id as number } }),
app.uid,
`user:${user.uuid}:email:read`,
);
const after = (await whoamiAs(actor)).body as Record<
string,
unknown
>;
expect(after.email).toBe(user.email);
expect(after.unconfirmed_email).toBe(user.email);
});
it('answers 404 for a missing account even when the check fails', async () => {
const check = vi
.spyOn(server.services.permission, 'check')
.mockRejectedValue(new Error('scan failed'));
try {
const captured = await whoamiAs(
makeActor({
user: { uuid: 'ghost-uuid', id: 99_999_998 },
app: { uid: 'app-ghost', id: 1 },
}),
);
expect(captured.statusCode).toBe(404);
expect(captured.body).toEqual({ error: 'User not found' });
} finally {
check.mockRestore();
}
});
});
it('returns full user details for a user actor', async () => {
const user = await seedUser();
const { res, captured } = makeRes();
@@ -257,85 +114,8 @@ describe('whoami extension — handleWhoami', () => {
expect(body.created_ts).toBeGreaterThan(0);
// `directories` is only sent to user actors — confirm it’s present.
expect(body.directories).toBeDefined();
});
it('leaves out taskbar items saved on the account', async () => {
const user = await seedUser();
await server.stores.user.update(user.id as number, {
taskbar_items: JSON.stringify([{ type: 'app', name: 'editor' }]),
});
const { res, captured } = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleWhoami(makeReq(), res),
);
expect(captured.body).not.toHaveProperty('taskbar_items');
});
it('leaves out a desktop background saved on the account', async () => {
const user = await seedUser();
await server.stores.user.update(user.id as number, {
desktop_bg_url: 'https://cdn.test/wallpaper.png',
desktop_bg_color: '#000000',
desktop_bg_fit: 'cover',
});
const { res, captured } = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleWhoami(makeReq(), res),
);
const body = captured.body as Record<string, unknown>;
expect(body).not.toHaveProperty('desktop_bg_url');
expect(body).not.toHaveProperty('desktop_bg_color');
expect(body).not.toHaveProperty('desktop_bg_fit');
});
it('reports the SMS-to-card fallback only once a send has opened it', async () => {
const user = await server.stores.user.create({
username: `wuser_${Math.random().toString(36).slice(2, 8)}`,
uuid: uuidv4(),
password: 'hashedpw',
email: `${Math.random().toString(36).slice(2, 8)}@example.com`,
requires_phone_verification: true,
} as never);
const prev = configContainer.phone_verification_card_fallback;
configContainer.phone_verification_card_fallback = {
enabled: true,
} as never;
try {
const before = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleWhoami(makeReq(), before.res),
);
// Phone-gated, but the user has attempts left — no offer yet.
expect(
before.captured.body as Record<string, unknown>,
).toMatchObject({ card_fallback_available: false });
// Exhausting SMS attempts stamps this flag; whoami is then the only
// thing that can still tell a reloading GUI about the offer, since
// further sends are rejected by the route's own rate limit.
await server.stores.kv.set({
key: `card-fallback-open:${user.id}`,
value: true,
});
const after = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleWhoami(makeReq(), after.res),
);
expect(
after.captured.body as Record<string, unknown>,
).toMatchObject({ card_fallback_available: true });
} finally {
configContainer.phone_verification_card_fallback = prev;
}
// taskbar_items is only sent to user actors.
expect(body).toHaveProperty('taskbar_items');
});
it('only forwards allow-listed feature flags', async () => {
@@ -355,16 +135,16 @@ describe('whoami extension — handleWhoami', () => {
expect(flags.payment_bypass).toBeUndefined();
});
it('strips created_ts and human_readable_age fields for app actors', async () => {
it('strips desktop_bg_*, created_ts and human_readable_age fields for app actors', async () => {
const user = await seedUser();
const { res, captured } = makeRes();
await runWithContext(
{
actor: makeActor({
actor: {
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
},
},
() => handleWhoami(makeReq(), res),
);
@@ -373,6 +153,9 @@ describe('whoami extension — handleWhoami', () => {
expect(body.app_name).toBe('app-test-actor');
// is_user_token is stripped for app actors.
expect(body.is_user_token).toBeUndefined();
expect(body.desktop_bg_url).toBeUndefined();
expect(body.desktop_bg_color).toBeUndefined();
expect(body.desktop_bg_fit).toBeUndefined();
expect(body.human_readable_age).toBeUndefined();
// Account age, in either form, is not exposed to apps.
expect(body.created_ts).toBeUndefined();
@@ -380,104 +163,6 @@ describe('whoami extension — handleWhoami', () => {
expect(body.directories).toBeUndefined();
});
describe('fields added by whoami.details listeners', () => {
const addAccountDetails = (
_key: unknown,
event: { details: Record<string, unknown> },
) => {
Object.assign(event.details, {
subscribed: true,
paid_storage: 1024,
subscription: { tier: 'pro', status: 'trialing' },
accountEligible: true,
});
};
const whoamiWithListener = async (
actor: Actor,
listener: typeof addAccountDetails = addAccountDetails,
) => {
const { res, captured } = makeRes();
server.clients.event.on('whoami.details', listener);
try {
await runWithContext({ actor }, () =>
handleWhoami(makeReq(), res),
);
} finally {
server.clients.event.off('whoami.details', listener);
}
return captured.body as Record<string, unknown>;
};
it('reach an app actor only as the documented subscribed / paid_storage', async () => {
const user = await seedUser();
const body = await whoamiWithListener(
makeActor({
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
);
expect(body.subscribed).toBe(true);
expect(body.paid_storage).toBe(1024);
expect(body).not.toHaveProperty('subscription');
expect(body).not.toHaveProperty('accountEligible');
// Fields the handler built itself still ship.
expect(body.username).toBe(user.username);
expect(body.app_name).toBe('app-test-actor');
});
it('are withheld from an app reached through an access token it issued', async () => {
const user = await seedUser();
const owner = { uuid: user.uuid, id: user.id as number };
const body = await whoamiWithListener(
makeActor({
user: owner,
accessToken: {
uid: 'tok-1',
issuer: makeActor({
user: owner,
app: { uid: 'app-issuer' },
}),
},
}),
);
expect(body.subscribed).toBe(true);
expect(body).not.toHaveProperty('subscription');
expect(body).not.toHaveProperty('accountEligible');
});
it('all reach a user actor', async () => {
const user = await seedUser();
const body = await whoamiWithListener(
makeActor({ user: { uuid: user.uuid, id: user.id as number } }),
);
expect(body).toMatchObject({
subscribed: true,
paid_storage: 1024,
subscription: { tier: 'pro', status: 'trialing' },
accountEligible: true,
});
});
it('cannot add directories for an app actor, which core omits', async () => {
const user = await seedUser();
const body = await whoamiWithListener(
makeActor({
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
(_key, event) => {
event.details.directories = { '/injected': 'uuid' };
},
);
expect(body).not.toHaveProperty('directories');
});
});
it('redacts tmp_password from metadata for user actors', async () => {
const user = await seedUser();
await server.stores.user.updateMetadata(user.id as number, {
@@ -508,10 +193,10 @@ describe('whoami extension — handleWhoami', () => {
const { res, captured } = makeRes();
await runWithContext(
{
actor: makeActor({
actor: {
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
},
},
() => handleWhoami(makeReq(), res),
);
@@ -521,82 +206,6 @@ describe('whoami extension — handleWhoami', () => {
expect(JSON.stringify(body)).not.toContain('bootstrap-secret');
});
it('never exposes phone, card fingerprint or signup identity', async () => {
const user = await seedUser();
await server.stores.user.update(user.id as number, {
phone: '+15551234567',
card_fingerprint: 'fp_ABC123',
requires_phone_verification: false,
requires_card_verification: false,
});
// Guard against a vacuous assertion below: the columns really do hold
// the values we then expect never to see on the wire.
const stored = await server.stores.user.getById(user.id as number, {
cached: false,
force: true,
});
expect(stored?.phone).toBe('+15551234567');
expect(stored?.card_fingerprint).toBe('fp_ABC123');
for (const actor of [
makeActor({ user: { uuid: user.uuid, id: user.id as number } }),
makeActor({
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
]) {
const { res, captured } = makeRes();
await runWithContext({ actor }, () => handleWhoami(makeReq(), res));
const body = captured.body as Record<string, unknown>;
expect(body.phone).toBeUndefined();
expect(body.card_fingerprint).toBeUndefined();
expect(body.password).toBeUndefined();
expect(body.otp_secret).toBeUndefined();
expect(body.signup_ip).toBeUndefined();
// Not just absent as a key — the values must not appear anywhere
// in the payload (nested under metadata, …).
const serialized = JSON.stringify(body);
expect(serialized).not.toContain('5551234567');
expect(serialized).not.toContain('fp_ABC123');
// The verification flags, which the GUI acts on, still ship.
expect(body).toHaveProperty('requires_phone_verification');
expect(body).toHaveProperty('requires_card_verification');
}
});
it('scrubs sensitive keys added to metadata, and leaves the cached row intact', async () => {
const user = await seedUser();
await server.stores.user.updateMetadata(user.id as number, {
tmp_password: 'bootstrap-secret',
billing: { card_fingerprint: 'fp_NESTED', tier: 'pro' },
});
const { res, captured } = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleWhoami(makeReq(), res),
);
const body = captured.body as Record<string, unknown>;
const metadata = body.metadata as Record<string, unknown>;
expect(metadata.tmp_password).toBeUndefined();
// Nested sensitive keys are removed too; siblings survive.
expect(metadata.billing).toEqual({ tier: 'pro' });
expect(JSON.stringify(body)).not.toContain('fp_NESTED');
// The scrub works on a copy — server-side state is untouched.
const fresh = await server.stores.user.getById(user.id as number, {
cached: false,
force: true,
});
expect(fresh?.metadata?.tmp_password).toBe('bootstrap-secret');
expect(
(fresh?.metadata?.billing as Record<string, unknown>)
?.card_fingerprint,
).toBe('fp_NESTED');
});
it('marks the user as oidc_only when password is null', async () => {
const slug = Math.random().toString(36).slice(2, 8);
const oidcUser = await server.stores.user.create({
+133 -55
View File
@@ -1,9 +1,6 @@
import { Context } from '@heyputer/backend/src/core';
import { extension } from '@heyputer/backend/src/extensions';
import {
buildUserDetails,
scrubSensitive,
} from '@heyputer/backend/src/util/userDetails.js';
import { getTaskbarItems } from '@heyputer/backend/src/util/taskbarItems.js';
import type { Request, Response } from 'express';
import TimeAgo from 'javascript-time-ago';
import localeEn from 'javascript-time-ago/locale/en';
@@ -17,11 +14,23 @@ const timeago = (() => {
return new TimeAgo('en-US');
})();
// What an app actor may receive from `whoami.details` listeners. Anything
// else a listener adds describes the account, and only ships to user actors.
const APP_VISIBLE_LISTENER_KEYS: ReadonlySet<string> = new Set([
'subscribed',
'paid_storage',
// User timestamps come off the DB as SQL datetime strings; the wire format
// for all of them is unix seconds. Unparseable values are dropped rather
// than sent as NaN.
const toUnixSeconds = (value: unknown): number | undefined => {
if (!value) return undefined;
const ms = new Date(value as string | number | Date).getTime();
return Number.isNaN(ms) ? undefined : Math.round(ms / 1000);
};
// Allowlist of `config.feature_flags` keys safe to surface via /whoami.
// Anything not listed here stays server-side, so internal flags
// (payment_bypass, staff_only_*, etc.) cannot leak by accident. Add a
// flag here when, and only when, the client actually needs to read it.
const CLIENT_VISIBLE_FEATURE_FLAGS: ReadonlySet<string> = new Set([
'create_shortcut',
'download_directory',
'prompt_user_when_navigation_away_from_puter',
]);
export const handleWhoami = async (
@@ -34,62 +43,141 @@ export const handleWhoami = async (
return;
}
const isUser = !actor.effectiveApp;
const canReadEmail = async (uuid: unknown) => {
try {
return await services.permission.check(
actor,
`user:${uuid}:email:read`,
);
} catch {
return false;
}
};
// Started beside the row read: the actor carries the row's uuid.
const emailCheck = isUser ? null : canReadEmail(actor.user.uuid);
const isUser = !actor.app;
const user = await stores.user.getById(actor.user.id);
if (!user) {
res.status(404).json({ error: 'User not found' });
return;
}
const details = await buildUserDetails(
user,
{ config: extension.config, clients, stores, services },
{ isUser },
);
details.human_readable_age = user.timestamp
? timeago.format(new Date(user.timestamp as string))
: null;
const oidcOnly = user.password === null;
const ALLOWED_ICON_SIZES = new Set([16, 32, 64, 128, 256, 512]);
const rawIconSize =
typeof req.query?.icon_size === 'string'
? Number(req.query.icon_size)
: undefined;
const iconSize =
rawIconSize !== undefined && ALLOWED_ICON_SIZES.has(rawIconSize)
? rawIconSize
: undefined;
const noIcons = !iconSize;
// Feature flags come from `config.feature_flags`. We only forward keys
// listed in CLIENT_VISIBLE_FEATURE_FLAGS so internal flags can't leak.
// Non-boolean values (e.g. `"true"` as a string) are coerced so the
// client never has to guess.
const rawFlags = extension.config.feature_flags ?? {};
const feature_flags: Record<string, boolean> = {};
for (const [k, v] of Object.entries(rawFlags)) {
if (CLIENT_VISIBLE_FEATURE_FLAGS.has(k)) {
feature_flags[k] = Boolean(v);
}
}
const metadata = user.metadata ? { ...user.metadata } : user.metadata;
if (metadata) delete metadata.tmp_password;
const details: Record<string, unknown> = {
username: user.username,
uuid: user.uuid,
email: user.email,
unconfirmed_email: user.email,
email_confirmed: user.email_confirmed || user.username === 'admin',
requires_email_confirmation: user.requires_email_confirmation,
phone: user.phone,
requires_phone_verification: user.requires_phone_verification,
requires_card_verification: user.requires_card_verification,
desktop_bg_url: user.desktop_bg_url,
desktop_bg_color: user.desktop_bg_color,
desktop_bg_fit: user.desktop_bg_fit,
is_temp: user.password === null && user.email === null,
is_user_token: true,
oidc_only: oidcOnly,
taskbar_items: isUser
? await getTaskbarItems(
user,
{
clients,
stores,
services,
apiBaseUrl: String(extension.config.api_base_url ?? ''),
},
{ iconSize, noIcons },
)
: undefined,
otp: !!user.otp_enabled,
feature_flags,
created_ts: toUnixSeconds(user.timestamp),
human_readable_age: user.timestamp
? timeago.format(new Date(user.timestamp as string))
: null,
metadata,
hasDevAccountAccess: !!user.metadata?.hasDevAccountAccess,
};
// OIDC revalidate URL for password-less accounts
if (oidcOnly) {
try {
const provider = await services.oidc.getLinkedProviderForUser(
user.id as number,
);
if (provider) {
const origin = (extension.config.origin ?? '').replace(
/\/$/,
'',
);
details.oidc_revalidate_url = `${origin}/auth/oidc/${provider}/start?flow=revalidate&user_uuid=${encodeURIComponent(user.uuid)}`;
}
} catch {
// OIDC not configured
}
}
// Directories — only sent to user actors
if (isUser) {
const directories: Record<string, unknown> = {};
const nameToProp: Record<string, string> = {
desktop_uuid: `/${user.username}/Desktop`,
appdata_uuid: `/${user.username}/AppData`,
documents_uuid: `/${user.username}/Documents`,
pictures_uuid: `/${user.username}/Pictures`,
videos_uuid: `/${user.username}/Videos`,
trash_uuid: `/${user.username}/Trash`,
};
for (const k in nameToProp) {
directories[nameToProp[k]] = user[k];
}
details.directories = directories;
}
// Last activity
const lastActivityTs = toUnixSeconds(user.last_activity_ts);
if (lastActivityTs !== undefined) {
details.last_activity_ts = lastActivityTs;
}
// Strip sensitive fields for app actors
if (!isUser) {
const emailReadable = await (user.uuid === actor.user.uuid
? emailCheck
: canReadEmail(user.uuid));
if (!emailReadable) {
const canReadEmail = await services.permission
.check(actor, `user:${user.uuid}:email:read`)
.catch(() => false);
if (!canReadEmail) {
delete details.email;
delete details.unconfirmed_email;
}
delete details.desktop_bg_url;
delete details.desktop_bg_color;
delete details.desktop_bg_fit;
delete details.human_readable_age;
delete details.created_ts;
delete details.is_user_token;
delete details.metadata;
// An app has no business reading the code its user earns credit with.
delete details.referral_code;
}
const app = actor.effectiveApp;
if (app) {
details.app_name = app.uid;
if (actor.app) {
details.app_name = actor.app.uid;
}
// A key core left undefined isn't built, so a listener can't fill it for an app.
const builtKeys = isUser
? null
: new Set(
Object.keys(details).filter((key) => details[key] !== undefined),
);
try {
await clients.event.emitAndWait(
'whoami.details',
@@ -99,13 +187,6 @@ export const handleWhoami = async (
} catch {
/* best-effort */
}
if (builtKeys) {
for (const key of Object.keys(details)) {
if (!builtKeys.has(key) && !APP_VISIBLE_LISTENER_KEYS.has(key)) {
delete details[key];
}
}
}
const subscription = details.subscription as
{ offering?: Record<string, unknown> } | undefined;
@@ -115,9 +196,6 @@ export const handleWhoami = async (
delete subscription.offering.price_id;
}
// Last word on what ships, after every listener has had its say.
scrubSensitive(details);
res.json(details);
};
-7
View File
@@ -190,13 +190,6 @@ S3_BUCKET=puter-local
providers = [ordered]@{
ollama = [ordered]@{ enabled = $false }
}
# Off because a self-hosted install has no paid plans: every account
# is on the free policy, so the plan gates would 402 the surfaces that
# declare one (the OpenAI/Anthropic-compatible AI endpoints) for
# everybody. Same as install.sh.
meteringEnforcement = [ordered]@{
subscriptions = $false
}
trust_proxy = 1
}
$configJson = $config | ConvertTo-Json -Depth 10
+1 -30
View File
@@ -53,29 +53,9 @@ need() {
command -v "$1" >/dev/null 2>&1 || die "missing required command: $1"
}
check_docker() {
if command -v docker >/dev/null 2>&1; then
return 0
fi
echo >&2 ""
echo >&2 "[puter-install] missing required command: docker"
echo >&2 ""
echo >&2 "Docker is required to run Puter self-hosted."
echo >&2 ""
echo >&2 "Installation instructions:"
echo >&2 " - macOS (Apple Silicon & Intel): https://docs.docker.com/desktop/setup/install/mac-install/"
echo >&2 " (Docker Desktop is recommended. Alternatives: OrbStack, Colima)"
echo >&2 " - Linux: https://docs.docker.com/engine/install/"
echo >&2 " - Windows: https://docs.docker.com/desktop/setup/install/windows-install/"
echo >&2 ""
echo >&2 "After installing Docker, make sure it's running and try again."
echo >&2 ""
exit 1
}
# ── Step 1: dependency check ────────────────────────────────────────
log "checking dependencies"
check_docker
need docker
need curl
need openssl
docker compose version >/dev/null 2>&1 \
@@ -151,11 +131,6 @@ S3_SECRET_KEY=$S3_SECRET_KEY
S3_BUCKET=puter-local
EOF
# `meteringEnforcement.subscriptions` is off here because a self-hosted
# install has no paid plans to be on: every account resolves to the free
# policy, so leaving the plan gates armed would 402 the surfaces that
# declare one (the OpenAI/Anthropic-compatible AI endpoints) for everybody.
# The code default is on — this is a property of the install, not of Puter.
log "writing puter/config/config.json"
cat > puter/config/config.json <<EOF
{
@@ -216,10 +191,6 @@ EOF
"ollama": { "enabled": false }
},
"meteringEnforcement": {
"subscriptions": false
},
"trust_proxy": $PUTER_TRUST_PROXY
}
EOF
+4072 -2946
View File
File diff suppressed because it is too large. Load diff
+28 -18
View File
@@ -13,9 +13,14 @@
"@babel/core": "^7.29.7",
"@eslint/js": "^9.35.0",
"@playwright/test": "^1.56.1",
"@stylistic/eslint-plugin": "^5.3.1",
"@types/better-sqlite3": "^7.6.13",
"@types/express": "^5.0.0",
"@types/mime-types": "^3.0.1",
"@typescript-eslint/eslint-plugin": "^8.46.1",
"@typescript-eslint/parser": "^8.46.1",
"@vitest/coverage-v8": "^4.0.14",
"@vitest/ui": "^4.0.14",
"babel-loader": "^10.1.1",
"babel-plugin-istanbul": "^8.0.0",
"chalk": "^4.1.0",
@@ -25,19 +30,25 @@
"eslint": "^9.35.0",
"eslint-config-prettier": "^10.1.8",
"eslint-plugin-prettier": "^5.5.5",
"eslint-rule-composer": "^0.3.0",
"globals": "^15.15.0",
"html-webpack-plugin": "^5.6.0",
"husky": "^9.1.7",
"istanbul-lib-coverage": "^3.2.2",
"istanbul-lib-report": "^3.0.1",
"istanbul-reports": "^3.2.0",
"license-check-and-add": "^4.0.5",
"nodemon": "^3.1.0",
"prettier": "^3.8.3",
"prettier-plugin-jsdoc": "^1.8.1",
"simple-git": "^3.32.3",
"typescript": "^5.4.5",
"uglify-js": "^3.17.4",
"vite-plugin-static-copy": "^3.3.0",
"vitest": "^4.1.5",
"webpack": "^5.88.2",
"webpack-cli": "^5.1.1"
"webpack-cli": "^5.1.1",
"yaml": "^2.8.1"
},
"scripts": {
"test:backend": "npm run setupExtensions && npm run build:workerLib && vitest run --config src/backend/vitest.config.ts ",
@@ -50,10 +61,10 @@
"build:workerLib:coverage": "cd src/puter-js && npm run build:coverage && cd ../worker && npm run build",
"start:gui": "nodemon --exec \"node dev-server.js\" ",
"start": "node ./tools/start.mjs",
"start:smtp": "node ./tools/start.mjs --smtp",
"dev": "npm start",
"build": "npm run setupExtensions && npm run build:ts && cd src/gui && node ./build.js && cd ../puter-js && npm run build",
"build:workerLib": "cd src/puter-js && npm run build && cd ../worker && npm run build",
"check-translations": "node tools/check-translations.js",
"prepare": "husky",
"build:ts": "tsc -p tsconfig.build.json && node ./tools/write-dist-package-json.mjs",
"check:puterjs:types": "node tools/checkPuterjsTypes.mjs",
@@ -62,7 +73,9 @@
"setupExtensions": "node ./tools/extensionSetup.mjs"
},
"workspaces": [
"src/*"
"src/*",
"tools/*",
"experiments/js-parse-and-output"
],
"nodemonConfig": {
"ext": "js, json, mjs, jsx, svg, css",
@@ -72,31 +85,28 @@
]
},
"dependencies": {
"@ai-sdk/openai": "^3.0.25",
"@aws-sdk/client-s3": "^3.1020.0",
"@aws-sdk/s3-request-presigner": "^3.1028.0",
"@heyputer/putility": "^1.0.2",
"ai": "^6.0.73",
"dedent": "^1.5.3",
"javascript-time-ago": "^2.5.11",
"libphonenumber-js": "1.13.15",
"libphonenumber-js": "1.13.6",
"miniflare": "^4.20260617.1",
"open": "^10.1.0",
"sharp": "^0.35.5"
"open": "^10.1.0"
},
"engines": {
"node": ">=24.0.0"
},
"allowScripts": {
"@google/genai": false,
"bcrypt": true,
"better-sqlite3": true,
"browser-tabs-lock": true,
"classic-level": true,
"esbuild": true,
"fsevents": true,
"protobufjs": true,
"sharp": true,
"workerd": true
},
"overrides": {
"undici": "^7.30.0"
"bcrypt@5.1.1": true,
"better-sqlite3@12.11.1": true,
"browser-tabs-lock@1.3.0": true,
"classic-level@3.0.0": true,
"[email protected]": true,
"[email protected]": true,
"[email protected]": true
}
}
-134
View File
@@ -1,134 +0,0 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { v4 as uuidv4 } from 'uuid';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import type { PuterServer } from './server.ts';
import { createTestUser, setupTestServer } from './testUtil.ts';
/**
* Teams is optional, so core has to work when neither the service nor the store
* is registered, which is how this test server boots.
*/
describe('core without a teams registration', () => {
let server: PuterServer;
const makeUser = async (): Promise<{ id: number; username: string }> => {
const username = `nt_${Math.random().toString(36).slice(2, 10)}`;
const created = (await server.stores.user.create({
username,
uuid: uuidv4(),
password: null,
email: `${username}@test.local`,
})) as unknown as { id: number };
return { id: created.id, username };
};
beforeAll(async () => {
server = await setupTestServer({
// Without this the allowlist check returns before asking Teams.
teams_allowed_email_domains: ['test.local'],
} as never);
});
afterAll(async () => {
await server?.shutdown();
});
it('has no teams in the registry', () => {
expect(server.services.team).toBeUndefined();
expect(server.stores.team).toBeUndefined();
});
it('refuses to boot when the config asks for teams', async () => {
await expect(
setupTestServer({ teams_enabled: true } as never),
).rejects.toThrow(/teams_enabled/);
});
it('deletes an account without asking what it was charged for', async () => {
const user = await makeUser();
await expect(
server.services.userAccount.cascadeDelete(user.id),
).resolves.not.toThrow();
expect(await server.stores.user.getById(user.id)).toBeFalsy();
});
it('answers the address-claim question signup asks', async () => {
const user = await makeUser();
const row = (await server.stores.user.getById(user.id))!;
// Nothing is a seat, so the holder is returned as-is.
await expect(
server.services.signup.signupEmailHolder(row.email as string, {
releaseSeat: true,
}),
).resolves.toMatchObject({ id: user.id });
});
it('lists what is shared with an account, folding in no team shares', async () => {
const user = await makeUser();
await expect(
server.services.share.listSharedWithMe({
user: { id: user.id },
} as never),
).resolves.toMatchObject({ items: [] });
});
it('refuses a share aimed at a team, rather than failing', async () => {
// A real home directory, so the refusal is the recipient's and not
// the entry's.
const username = `nt_${Math.random().toString(36).slice(2, 10)}`;
const creds = await createTestUser(server, {
username,
password: 'Nt-Pw-12345!',
});
const row = (await server.stores.user.getByUsername(username))!;
const user = { id: row.id, username };
const actor = { user: row } as never;
expect(creds).toBeTruthy();
// The recipient cannot exist, so this is a 404 and not a crash.
await expect(
server.services.share.share(actor, {
recipient: { teamHandle: 'nobody' },
mode: 'read',
path: `/${user.username}`,
} as never),
).rejects.toMatchObject({ statusCode: 404 });
});
it('serves the homepage shell with the teams tab off', async () => {
const user = await makeUser();
const row = (await server.stores.user.getById(user.id))!;
let html = '';
const res = { send: (out: string) => (html = out) };
await server.services.homepage.send(
{
req: { headers: {}, query: {}, get: () => undefined },
res,
actor: { user: row },
} as never,
{ title: 'Puter' } as never,
);
const params = JSON.parse(
/gui\((\{.*?\})\);/s.exec(html)![1].replaceAll('\\u003c', '<'),
);
expect(params.teams_ui).toBe(false);
});
});
+6 -177
View File
@@ -1,9 +1,5 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import {
ALARM_IDLE_TTL_MS,
AlarmClient,
MAX_TRACKED_ALARMS,
} from './AlarmClient';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { AlarmClient } from './AlarmClient';
import type { IConfig, IPagerConfig } from '../../types';
import type { AlertPayload } from './types';
@@ -346,19 +342,6 @@ describe('AlarmClient alarm registry', () => {
expect(client.get(shortId)).toBeUndefined();
});
it('keeps the short id of another alarm that shares it when one is cleared', () => {
const client = makeClient();
// These two ids hash to the same short id.
client.create('alarm:274', 'first');
const shortId = client.get('alarm:274')!.shortId;
client.create('alarm:643', 'second');
expect(client.get('alarm:643')!.shortId).toBe(shortId);
client.clear('alarm:274');
expect(client.get(shortId)?.id).toBe('alarm:643');
});
it('ignores a clear for an alarm that is not active', () => {
const client = makeClient();
expect(() => client.clear('never-raised')).not.toThrow();
@@ -392,18 +375,18 @@ describe('AlarmClient alarm registry', () => {
const client = makeClient();
const seen = capture(client);
for (let i = 0; i < 500; i++) {
for (let i = 0; i < 50; i++) {
client.create('hot', `occurrence ${i}`, { i });
}
const alarm = client.get('hot')!;
expect(alarm.count).toBe(500);
expect(alarm.count).toBe(50);
expect(alarm.occurrences).toHaveLength(20);
expect(alarm.timestamps).toHaveLength(20);
// The window kept is the most recent one, not the oldest.
expect(alarm.occurrences[19].message).toBe('occurrence 499');
expect(alarm.occurrences[19].message).toBe('occurrence 49');
// Trimming history must not rewind what the transports are told.
expect(seen.at(-1)).toMatchObject({ repeatCount: 500, isRepeat: true });
expect(seen[49]).toMatchObject({ repeatCount: 50, isRepeat: true });
});
it('names anonymous handlers by their registration order', () => {
@@ -445,110 +428,6 @@ describe('AlarmClient alarm registry', () => {
});
});
describe('AlarmClient alarm eviction', () => {
beforeEach(() => {
vi.useFakeTimers();
vi.spyOn(console, 'error').mockImplementation(() => {});
vi.spyOn(console, 'warn').mockImplementation(() => {});
vi.spyOn(console, 'log').mockImplementation(() => {});
});
afterEach(() => {
vi.useRealTimers();
});
it('forgets alarms that stay quiet past the idle window, under both ids', () => {
const client = makeClient();
const ids = Array.from(
{ length: 50 },
(_, i) => `metering_usage_dropped:user-${i}`,
);
for (const id of ids) client.create(id, 'usage dropped');
const shortIds = ids.map((id) => client.get(id)!.shortId);
vi.advanceTimersByTime(ALARM_IDLE_TTL_MS + 1);
client.create('unrelated', 'the next alarm sweeps the registry');
expect(ids.filter((id) => client.get(id))).toEqual([]);
expect(shortIds.filter((shortId) => client.get(shortId))).toEqual([]);
expect(client.get('unrelated')).toBeDefined();
});
it('treats a recurrence after the idle window as a new alarm', () => {
const client = makeClient();
const seen = capture(client);
client.create('scan:failed', 'first');
client.create('scan:failed', 'second');
vi.advanceTimersByTime(ALARM_IDLE_TTL_MS + 1);
client.create('scan:failed', 'back again');
expect(seen.at(-1)).toMatchObject({ repeatCount: 1, isRepeat: false });
// A fresh start keeps per-occurrence incident keys from colliding.
expect(seen.at(-1)!.dedupKey).not.toBe(seen[0].dedupKey);
});
it('still treats a repeat inside the window as a repeat after a sweep', () => {
const client = makeClient();
const seen = capture(client);
for (let i = 0; i < 10; i++) client.create('flap', 'same fault');
vi.advanceTimersByTime(ALARM_IDLE_TTL_MS - 1000);
client.create('other', 'sweeps the registry');
client.create('flap', 'same fault');
// Occurrence 11 is past the burst, so nothing new goes out for it.
expect(seen.map((alert) => alert.id)).toEqual([
...Array(10).fill('flap'),
'other',
]);
expect(client.get('flap')?.count).toBe(11);
});
it('keeps an alarm that keeps recurring for longer than the window', () => {
const client = makeClient();
for (let i = 0; i < 6; i++) {
client.create('hot', 'still broken');
vi.advanceTimersByTime(ALARM_IDLE_TTL_MS / 2);
client.create(`noise:${i}`, 'sweeps the registry');
}
expect(client.get('hot')?.count).toBe(6);
});
it('drops the least recently raised alarm once full', () => {
const client = makeClient();
for (let i = 0; i < MAX_TRACKED_ALARMS; i++) {
client.create(`user:${i}`, 'per-user alarm');
vi.advanceTimersByTime(1);
}
// Raised again, so user:1 is now the one seen longest ago.
client.create('user:0', 'per-user alarm');
client.create('one-more', 'over the cap');
expect(client.get('user:1')).toBeUndefined();
expect(client.get('user:0')?.count).toBe(2);
expect(client.get('user:2')).toBeDefined();
expect(client.get('one-more')).toBeDefined();
});
it('never holds more than the cap', () => {
const client = makeClient();
const ids = Array.from(
{ length: MAX_TRACKED_ALARMS * 3 },
(_, i) => `user:${i}`,
);
for (const id of ids) client.create(id, 'per-user alarm');
const held = ids.filter((id) => client.get(id));
expect(held).toHaveLength(MAX_TRACKED_ALARMS);
expect(held[0]).toBe(`user:${MAX_TRACKED_ALARMS * 2}`);
});
});
describe('AlarmClient known-error rules', () => {
beforeEach(() => {
vi.spyOn(console, 'error').mockImplementation(() => {});
@@ -640,53 +519,3 @@ describe('AlarmClient known-error rules', () => {
expect(chat[0].severity).toBe('info');
});
});
describe('AlarmClient repeat reporting', () => {
beforeEach(() => {
vi.spyOn(console, 'error').mockImplementation(() => {});
vi.spyOn(console, 'warn').mockImplementation(() => {});
vi.spyOn(console, 'log').mockImplementation(() => {});
pdEvent.mockClear();
});
it('reports every occurrence up to the burst, then backs off', () => {
const client = makeClient();
const seen = capture(client);
for (let i = 0; i < 12; i++) client.create('flap', 'same fault');
expect(seen).toHaveLength(10);
expect(seen[9].repeatCount).toBe(10);
});
it('reports again once the interval is reached', () => {
const client = makeClient();
const seen = capture(client);
for (let i = 0; i < 500; i++) client.create('flap', 'same fault');
expect(seen).toHaveLength(11);
expect(seen[10].repeatCount).toBe(500);
});
it('keeps a fast-repeating fault out of the log after the burst', () => {
const client = makeClient();
capture(client);
const warn = vi.mocked(console.warn);
warn.mockClear();
for (let i = 0; i < 100; i++) client.create('flap', 'same fault');
// Occurrences 2-10 only; the first is an ACTIVE line on console.error.
expect(warn.mock.calls).toHaveLength(9);
});
it('skips the payload entirely when no transport accepts the severity', () => {
const client = makeClient();
const seen = capture(client, 'critical');
client.create('quiet', 'nobody is listening', {}, 'info');
expect(seen).toHaveLength(0);
});
});
+9 -77
View File
@@ -61,33 +61,14 @@ interface RegisteredHandler {
/** Severity used when neither the call site nor config picks one. */
const FALLBACK_SEVERITY: PagerSeverity = 'critical';
/**
* How many recent occurrences an alarm keeps. A hot alarm repeats for as long
* as the fault lasts, so retaining every occurrence means retaining every
* message and field set it was ever raised with, request bodies and actors
* included. The last few are what a human reads; the rest is only a count, and
* `count` keeps that.
* How many recent occurrences an alarm keeps. An alarm is never cleared unless
* something calls `clear`, and a hot one repeats for as long as the fault lasts
* — so retaining every occurrence means retaining every message and field set
* it was ever raised with, request bodies and actors included, for the life of
* the process. The last few are what a human reads; the rest is only a count,
* and `count` keeps that.
*/
const OCCURRENCE_HISTORY_LIMIT = 20;
/**
* An alarm that hasn't recurred for this long is forgotten; its next occurrence
* is a new alarm with a fresh count. Kept far past the chat transport's repeat
* throttle so a slowly recurring fault still counts as one alarm.
*/
export const ALARM_IDLE_TTL_MS = 24 * 60 * 60 * 1000;
/**
* Most alarms held at once. Ids that embed a user or key can outrun the idle
* window; past this the least recently raised alarm is dropped.
*/
export const MAX_TRACKED_ALARMS = 1000;
// A fast-recurring fault repeats far faster than it is useful to report, and
// every report allocates a log line and a payload. Report every occurrence up
// to the burst, then one per interval; `count` still tracks all of them.
const REPEAT_REPORT_BURST = 10;
const REPEAT_REPORT_INTERVAL = 500;
const shouldReportRepeat = (count: number): boolean =>
count <= REPEAT_REPORT_BURST || count % REPEAT_REPORT_INTERVAL === 0;
/** Keeps `info` alarms out of the paging system unless config says otherwise. */
const DEFAULT_PAGERDUTY_MIN_SEVERITY: PagerSeverity = 'warning';
/** Slack's ceiling once a pager exists: chat gets what doesn't page. */
@@ -197,7 +178,6 @@ function cleanFields(fields: AlarmFields): Record<string, string> {
* `pager.severityOverrides` in {@link IConfig}.
*/
export class AlarmClient extends PuterClient {
/** Ordered by last occurrence: a repeat moves its alarm to the back. */
private alarms = new Map<string, Alarm>();
private aliases = new Map<string, Alarm>();
private alertHandlers: RegisteredHandler[] = [];
@@ -301,8 +281,7 @@ export class AlarmClient extends PuterClient {
/**
* Create or update an alarm. If the alarm ID already exists, the occurrence
* count is incremented and a repeat alert is dispatched. An alarm idle for
* {@link ALARM_IDLE_TTL_MS} is forgotten, so its next occurrence is new.
* count is incremented and a repeat alert is dispatched.
*
* `severity` decides where the alarm lands:
*
@@ -333,12 +312,9 @@ export class AlarmClient extends PuterClient {
return;
}
this.evict();
const existing = this.alarms.get(id);
if (existing) {
this.alarms.delete(id);
this.alarms.set(id, existing);
this.recordOccurrence(existing, message, fields);
this.handleRepeat(existing);
return;
@@ -363,7 +339,6 @@ export class AlarmClient extends PuterClient {
this.alarms.set(id, alarm);
this.aliases.set(alarm.shortId, alarm);
this.recordOccurrence(alarm, message, fields);
this.evict();
this.handleNew(alarm);
}
@@ -372,7 +347,8 @@ export class AlarmClient extends PuterClient {
const alarm = this.alarms.get(id);
if (!alarm) return;
this.forget(alarm);
this.alarms.delete(id);
this.aliases.delete(alarm.shortId);
console.log(`[alarm] CLEAR ${displayId(alarm)} :: ${alarm.message}`);
}
@@ -410,32 +386,6 @@ export class AlarmClient extends PuterClient {
// -- Internals ----------------------------------------------------
/**
* Drop idle alarms, then the least recently raised ones past the cap. The
* map's order puts both at the front, so this stops at the first keeper.
*/
private evict(): void {
const now = Date.now();
for (const alarm of this.alarms.values()) {
const lastSeen = alarm.timestamps.at(-1) ?? alarm.started;
if (
this.alarms.size <= MAX_TRACKED_ALARMS &&
now - lastSeen < ALARM_IDLE_TTL_MS
) {
return;
}
this.forget(alarm);
}
}
private forget(alarm: Alarm): void {
this.alarms.delete(alarm.id);
// Short ids can collide; leave an alias that now names another alarm.
if (this.aliases.get(alarm.shortId) === alarm) {
this.aliases.delete(alarm.shortId);
}
}
private recordOccurrence(
alarm: Alarm,
message: string,
@@ -506,8 +456,6 @@ export class AlarmClient extends PuterClient {
private handleRepeat(alarm: Alarm): void {
this.applyKnownErrors(alarm);
if (!shouldReportRepeat(alarm.count)) return;
console.warn(
`[alarm] REPEAT ${displayId(alarm)} :: ${alarm.message} (${alarm.count})`,
);
@@ -535,18 +483,6 @@ export class AlarmClient extends PuterClient {
);
}
private hasHandlerFor(severity: PagerSeverity): boolean {
for (const { minSeverity, maxSeverity } of this.alertHandlers) {
if (
meetsMinSeverity(severity, minSeverity) &&
withinMaxSeverity(severity, maxSeverity)
) {
return true;
}
}
return false;
}
private dispatchAlert(alarm: Alarm): void {
const resolved = this.resolveSeverity(alarm);
if (resolved === 'mute') {
@@ -558,10 +494,6 @@ export class AlarmClient extends PuterClient {
}
alarm.severity = resolved;
// Build nothing for a severity no transport accepts — the payload
// (cleaned fields plus the stack, twice) is the expensive part.
if (!this.hasHandlerFor(resolved)) return;
const fieldsClean = cleanFields(alarm.fields);
const repeatCount = alarm.count;
+22 -7
View File
@@ -18,7 +18,6 @@
*/
import type { ISlackAlertConfig, PagerSeverity } from '../../types';
import { BoundedTtlMap } from '../../util/boundedTtlMap';
import type { AlertHandler, AlertPayload } from './types';
const REQUEST_TIMEOUT_MS = 5000;
@@ -124,14 +123,30 @@ export function createSlackAlertHandler(
): AlertHandler {
const webhookUrl = conf.webhookUrl as string;
const throttleMs = conf.repeatThrottleMs ?? DEFAULT_REPEAT_THROTTLE_MS;
// A zero or negative throttle expires every entry on arrival.
const lastPosted = new BoundedTtlMap<string, true>({
maxEntries: MAX_THROTTLE_ENTRIES,
ttlMs: throttleMs,
});
const lastPosted = new Map<string, number>();
const shouldPost = (alert: AlertPayload): boolean => {
if (throttleMs <= 0) return true;
const now = Date.now();
const previous = lastPosted.get(alert.id);
if (previous !== undefined && now - previous < throttleMs) return false;
if (lastPosted.size >= MAX_THROTTLE_ENTRIES) {
for (const [id, at] of lastPosted) {
if (now - at >= throttleMs) lastPosted.delete(id);
}
// Still full of live entries — drop the oldest to stay bounded.
if (lastPosted.size >= MAX_THROTTLE_ENTRIES) {
const oldest = lastPosted.keys().next().value;
if (oldest !== undefined) lastPosted.delete(oldest);
}
}
lastPosted.set(alert.id, now);
return true;
};
return async (alert) => {
if (!lastPosted.shouldEmit(alert.id)) return;
if (!shouldPost(alert)) return;
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
@@ -17,9 +17,24 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
// Optional analytics client, registered by an extension. Only the surface the
// app-stats path consumes is declared; extend it rather than widening to `any`.
/**
* Optional ClickHouse client.
*
* ClickHouse is NOT a hard dependency. Core Puter runs without it — the app
* stats read path (open / unique-user counts) falls back to the primary SQL
* database. `this.clients.clickhouse` is therefore typed as optional and is
* `undefined` in default and self-hosted setups; callers MUST branch on its
* presence and fall back to SQL.
*
* A production deployment can register a real ClickHouse client via an
* extension (`extension.registerClient('clickhouse', client)`) to offload the
* analytics queries off the primary database and keep the stats path fast at
* scale. When registered, the instance flows into `this.clients.clickhouse`
* everywhere it's typed.
*
* Only the surface the stats path actually consumes is declared here. Extend
* this interface (don't widen to `any`) when a new query shape is needed.
*/
export interface ClickhouseQueryResult {
json<T = Record<string, unknown>>(): Promise<T[]>;
}
@@ -45,15 +45,10 @@ class FixtureDatabaseClient extends AbstractDatabaseClient {
constructor(
private readonly replicaRows: Record<string, unknown>[] | Error,
private readonly primaryRows: Record<string, unknown>[] = [],
private readonly replica = true,
) {
super(config('fixture'));
}
protected override hasReadReplica(): boolean {
return this.replica;
}
override async read(query: string): Promise<Record<string, unknown>[]> {
this.reads.push(query);
if (this.replicaRows instanceof Error) throw this.replicaRows;
@@ -108,25 +103,6 @@ describe('AbstractDatabaseClient — unimplemented surface', () => {
});
describe('AbstractDatabaseClient — replica-aware reads', () => {
it('avoids a primary read when the replica has rows in sequential mode', async () => {
const client = new FixtureDatabaseClient([{ id: 1 }], [{ id: 2 }]);
expect(
await client.tryHardRead('SELECT 1', [], { race: false }),
).toEqual([{ id: 1 }]);
expect(client.preads).toEqual([]);
});
it.each([[], new Error('replica unavailable')])(
'falls back once in sequential mode',
async (replica) => {
const client = new FixtureDatabaseClient(replica, [{ id: 2 }]);
expect(
await client.tryHardRead('SELECT 1', [], { race: false }),
).toEqual([{ id: 2 }]);
expect(client.preads).toEqual(['SELECT 1']);
},
);
it('returns replica rows without waiting on the primary', async () => {
const client = new FixtureDatabaseClient([{ id: 1 }], [{ id: 2 }]);
await expect(client.tryHardRead('SELECT 1')).resolves.toEqual([
@@ -151,54 +127,14 @@ describe('AbstractDatabaseClient — replica-aware reads', () => {
]);
});
it('surfaces the primary error when both reads fail', async () => {
const client = new FixtureDatabaseClient(new Error('replica down'));
client.pread = async () => {
throw new Error('primary down');
};
await expect(client.tryHardRead('SELECT 1')).rejects.toThrow(
'primary down',
it('names the failing query when a required read finds nothing', async () => {
const client = new FixtureDatabaseClient([], []);
await expect(
client.requireRead('SELECT * FROM `user` WHERE `id` = ?'),
).rejects.toThrow(
'required read returned no rows: SELECT * FROM `user` WHERE `id` = ?',
);
});
it('issues a single read when there is no replica', async () => {
const client = new FixtureDatabaseClient([], [{ id: 2 }], false);
await expect(client.tryHardRead('SELECT 1')).resolves.toEqual([]);
expect(client.reads).toEqual(['SELECT 1']);
expect(client.preads).toEqual([]);
});
describe('tryHardReadAll', () => {
const reads = [{ query: 'SELECT 1' }, { query: 'SELECT 2' }];
it('keeps replica rows without reading the primary', async () => {
const client = new FixtureDatabaseClient([{ id: 1 }], [{ id: 2 }]);
expect(await client.tryHardReadAll(reads)).toEqual([
[{ id: 1 }],
[{ id: 1 }],
]);
expect(client.preads).toEqual([]);
});
it.each([[], new Error('replica unavailable')])(
'reads every query on the primary when the replica has none',
async (replica) => {
const client = new FixtureDatabaseClient(replica, [{ id: 2 }]);
expect(await client.tryHardReadAll(reads)).toEqual([
[{ id: 2 }],
[{ id: 2 }],
]);
expect(client.preads).toEqual(['SELECT 1', 'SELECT 2']);
},
);
it('issues one read per query when there is no replica', async () => {
const client = new FixtureDatabaseClient([], [{ id: 2 }], false);
expect(await client.tryHardReadAll(reads)).toEqual([[], []]);
expect(client.reads).toEqual(['SELECT 1', 'SELECT 2']);
expect(client.preads).toEqual([]);
});
});
});
describe('AbstractDatabaseClient — SQL generation', () => {
@@ -299,47 +235,6 @@ describe('AbstractDatabaseClient — SQL generation', () => {
expect(mysql().returningIdClause()).toBe('');
});
it('stamps a timestamp from unix seconds in each dialect', () => {
expect(sqlite().timestampFromUnixSql()).toBe(
"datetime(?, 'unixepoch')",
);
expect(mysql().timestampFromUnixSql()).toBe('FROM_UNIXTIME(?)');
expect(postgres().timestampFromUnixSql()).toBe('to_timestamp(?)');
});
it('reads a timestamp column back as unix seconds in each engine', () => {
expect(sqlite().unixSecondsSql('`t`')).toBe(
"CAST(strftime('%s', `t`) AS INTEGER)",
);
expect(mysql().unixSecondsSql('`t`')).toBe('UNIX_TIMESTAMP(`t`)');
expect(postgres().unixSecondsSql('`t`')).toBe(
'EXTRACT(EPOCH FROM `t`)::bigint',
);
});
it('reads the database clock as epoch milliseconds in each dialect', () => {
expect(sqlite().nowMsExpr()).toBe(
"CAST((julianday('now') - 2440587.5) * 86400000 AS INTEGER)",
);
expect(mysql().nowMsExpr()).toBe(
"(TIMESTAMPDIFF(MICROSECOND, '1970-01-01 00:00:00', UTC_TIMESTAMP(6)) DIV 1000)",
);
expect(postgres().nowMsExpr()).toBe(
'CAST(EXTRACT(EPOCH FROM clock_timestamp()) * 1000 AS BIGINT)',
);
});
it('describes a stored timestamp the way each driver reads it back', () => {
const seconds = Date.UTC(2026, 9, 11, 8, 5, 9) / 1000;
expect(sqlite().timestampValue(seconds)).toBe('2026-10-11 08:05:09');
expect(mysql().timestampValue(seconds)).toEqual(
new Date(seconds * 1000),
);
expect(postgres().timestampValue(seconds)).toEqual(
new Date(seconds * 1000),
);
});
it('falls back to `otherwise` for an engine with no explicit choice', () => {
const client = new FixtureDatabaseClient([]);
expect(
@@ -350,13 +245,3 @@ describe('AbstractDatabaseClient — SQL generation', () => {
);
});
});
it('builds an insert for batchWrite and shares it with insert', async () => {
const client = new FixtureDatabaseClient([]);
const entry = client.insertEntry('widget', { name: "reader's", enabled: true });
expect(entry.values).toEqual(["reader's", true]);
expect(entry.statement).toContain('VALUES (?, ?)');
await client.insert('widget', { name: "reader's", enabled: true });
expect(client.writes).toEqual([{ query: entry.statement, params: entry.values }]);
});
+26 -263
View File
@@ -17,23 +17,13 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { readdirSync, readFileSync } from 'fs';
import { isAbsolute, resolve as resolvePath } from 'path';
import type { IConfig } from '../../types';
import { Span } from '../../util/span.js';
import { PuterClient } from '../types';
import { compareMigrationFilenames } from './migrationFilenames.js';
export interface WriteResult {
insertId: number | bigint;
affectedRows: number;
anyRowsAffected: boolean;
/**
* Warnings the statement raised; nonzero means a value may have been stored
* differently from how it was sent. Absent when the engine can't vouch that
* a clean insert stored exactly what was sent, on a fresh id.
*/
warningStatus?: number;
}
export interface BatchEntry {
@@ -43,12 +33,6 @@ export interface BatchEntry {
type SqlJsonPath = readonly [string, ...string[]];
/** The slice of a pooled connection that migrations need. */
export interface MigrationConnection {
query(sql: string): Promise<unknown>;
release(): void;
}
/**
* Base database client. Subclasses must override every method that throws here.
*
@@ -110,182 +94,49 @@ export class AbstractDatabaseClient extends PuterClient {
tableName: string,
data: Record<string, unknown>,
): Promise<WriteResult> {
const entry = this.insertEntry(tableName, data);
return this.write(
entry.statement + this.returningIdClause(),
entry.values,
);
}
/** An insert that can join other writes in a transaction. */
insertEntry(tableName: string, data: Record<string, unknown>): BatchEntry {
const cols = Object.keys(data);
return {
statement:
`INSERT INTO ${this.quoteIdentifier(tableName)} ` +
`(${cols.map((c) => this.quoteIdentifier(c)).join(', ')}) ` +
`VALUES (${cols.map(() => '?').join(', ')})`,
values: Object.values(data),
};
}
/** Whether `read()` goes to a separate replica rather than the primary. */
protected hasReadReplica(): boolean {
return false;
const values = Object.values(data);
const sql =
`INSERT INTO ${this.quoteIdentifier(tableName)} ` +
`(${cols.map((c) => this.quoteIdentifier(c)).join(', ')}) ` +
`VALUES (${cols.map(() => '?').join(', ')})` +
this.returningIdClause();
return this.write(sql, values);
}
/**
* The replica leg of `tryHardRead`. Override when `read()` falls back to
* the primary by itself, or a replica failure reads the primary twice.
* Like `read()` but falls back to the primary when read-replicas are in
* use. Subclasses may override with replica-aware logic; the default
* delegates to `pread()`.
*/
protected replicaRead(
query: string,
params: unknown[],
): Promise<Record<string, unknown>[]> {
return this.read(query, params);
}
/**
* Like `read()`, but an empty or failed replica read falls back to the
* primary, so replication lag can't hide a row that was just written. By
* default the primary read starts alongside the replica one; `race: false`
* reads the primary only after the replica returns empty or fails. Without
* a replica this is a single `read()`.
*/
@Span('db.tryHardRead', (query: string) => ({ 'db.statement': query }))
async tryHardRead(
query: string,
params: unknown[] = [],
{ race = true }: { race?: boolean } = {},
): Promise<Record<string, unknown>[]> {
if (!this.hasReadReplica()) return this.read(query, params);
const primary = this.pread(query, params);
primary.catch(() => {});
const primary = race ? settle(this.pread(query, params)) : null;
try {
const rows = await this.replicaRead(query, params);
if (rows.length > 0) return rows;
const rows = await this.read(query, params);
if (rows.length > 0) {
return rows;
}
} catch {
// replica failed — fall through to primary
}
const result = await (primary ?? settle(this.pread(query, params)));
if ('error' in result) throw result.error;
return result.value;
return primary;
}
/** Reads the primary only when every replica read is empty or fails. */
@Span('db.tryHardReadAll', (reads: unknown[]) => ({
'db.batch_size': reads.length,
}))
async tryHardReadAll(
reads: { query: string; params?: unknown[] }[],
): Promise<Record<string, unknown>[][]> {
if (!this.hasReadReplica()) {
return Promise.all(
reads.map(({ query, params = [] }) => this.read(query, params)),
);
}
const onReplica = await Promise.all(
reads.map(async ({ query, params = [] }) => {
try {
return await this.replicaRead(query, params);
} catch {
// A failed replica read counts as a miss.
return [];
}
}),
);
if (onReplica.some((rows) => rows.length > 0)) return onReplica;
return Promise.all(
reads.map(({ query, params = [] }) => this.pread(query, params)),
);
}
/**
* Apply the `.sql` files named `<engineName>*` in each configured migration
* directory, in filename order. Files must be idempotent: nothing records
* which ones already ran. A failure aborts startup. `transactional` wraps
* each file in BEGIN/COMMIT, for engines whose DDL can roll back.
*/
protected async applyMigrationPaths(
connect: () => Promise<MigrationConnection>,
splitStatements: (contents: string) => string[],
{ transactional }: { transactional: boolean },
): Promise<void> {
const paths = this.config.database?.migrationPaths;
if (!paths || paths.length === 0) return;
const tag = `[${this.engineName}]`;
const conn = await connect();
try {
for (const rawPath of paths) {
const dir = isAbsolute(rawPath)
? rawPath
: resolvePath(process.cwd(), rawPath);
let files: string[];
try {
files = readdirSync(dir)
.filter(
(f) =>
f.endsWith('.sql') &&
f.startsWith(this.engineName),
)
.sort(compareMigrationFilenames);
} catch (e) {
throw new Error(
`${tag} migration path is unreadable: ${dir}`,
{
cause: e,
},
);
}
if (files.length === 0) {
console.log(`${tag} no migrations in ${dir}`);
continue;
}
console.log(
`${tag} running migrations from ${dir}: ${files.length} file(s)`,
);
for (const file of files) {
const contents = readFileSync(
resolvePath(dir, file),
'utf8',
);
const statements = splitStatements(contents);
if (transactional) await conn.query('BEGIN');
try {
for (let i = 0; i < statements.length; i++) {
try {
await conn.query(statements[i]);
} catch (e) {
throw new Error(
`${tag} failed to apply ${file} at statement ${i}`,
{ cause: e },
);
}
}
if (transactional) await conn.query('COMMIT');
} catch (e) {
if (transactional) {
try {
await conn.query('ROLLBACK');
} catch {
// the original failure is the one worth surfacing
}
}
throw e;
}
console.log(
`${tag} applied ${file} (${statements.length} statements)`,
);
}
}
} finally {
conn.release();
/** Like `tryHardRead()` but throws when the result set is empty. */
async requireRead(
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
const rows = await this.tryHardRead(query, params);
if (rows.length === 0) {
throw new Error(`required read returned no rows: ${query}`);
}
return rows;
}
/**
@@ -375,30 +226,6 @@ export class AbstractDatabaseClient extends PuterClient {
});
}
/**
* Whole `periodSeconds` buckets since the epoch, from a unix-seconds
* column. MySQL's `/` yields a decimal, which would split a bucket.
*/
periodBucket(secondsExpression: string, periodSeconds: number): string {
return this.case({
mysql: `(${secondsExpression} DIV ${periodSeconds})`,
otherwise: `(${secondsExpression} / ${periodSeconds})`,
});
}
/**
* Counts each (`userExpression`, bucket) pair once. `COUNT(DISTINCT a, b)`
* is MySQL-only, so the pair folds into one number — injective while the
* bucket stays under the multiplier, which hourly reaches in the 2980s.
*/
countDistinctUserPeriods(
userExpression: string,
secondsExpression: string,
periodSeconds: number,
): string {
return `COUNT(DISTINCT ${userExpression} * 10000000 + ${this.periodBucket(secondsExpression, periodSeconds)})`;
}
nullCoalesce(...expressions: readonly string[]): string {
if (expressions.length === 0) {
throw new Error('nullCoalesce requires at least one expression');
@@ -413,71 +240,7 @@ export class AbstractDatabaseClient extends PuterClient {
});
}
/**
* A timestamp column value from one bound parameter in unix seconds, stored
* the way `CURRENT_TIMESTAMP` would store that instant.
*/
timestampFromUnixSql(): string {
return this.case({
sqlite: "datetime(?, 'unixepoch')",
postgres: 'to_timestamp(?)',
otherwise: 'FROM_UNIXTIME(?)',
});
}
/**
* A timestamp column as unix seconds, computed by the database: the mysql
* driver reads a stored UTC timestamp as local time, so parsing it in JS
* can be off by the server's UTC offset.
*/
unixSecondsSql(column: string): string {
return this.case({
postgres: `EXTRACT(EPOCH FROM ${column})::bigint`,
mysql: `UNIX_TIMESTAMP(${column})`,
otherwise: `CAST(strftime('%s', ${column}) AS INTEGER)`,
});
}
/**
* The database's own clock as epoch milliseconds, so nodes whose clocks
* disagree still agree on it. The mysql form ignores the session time zone,
* so a DST change can't shift it.
*/
nowMsExpr(): string {
return this.case({
sqlite: "CAST((julianday('now') - 2440587.5) * 86400000 AS INTEGER)",
postgres:
'CAST(EXTRACT(EPOCH FROM clock_timestamp()) * 1000 AS BIGINT)',
otherwise:
"(TIMESTAMPDIFF(MICROSECOND, '1970-01-01 00:00:00', UTC_TIMESTAMP(6)) DIV 1000)",
});
}
/**
* What a read of a timestamp column holding `seconds` returns: sqlite gives
* back the UTC text it stored, mysql2 and pg a `Date` read in node's time
* zone (so this matches when node and the database session share one).
*/
timestampValue(seconds: number): Date | string {
const date = new Date(seconds * 1000);
return this.case<Date | string>({
sqlite: date.toISOString().slice(0, 19).replace('T', ' '),
otherwise: date,
});
}
protected sqlStringLiteral(value: string): string {
return `'${value.replaceAll("'", "''")}'`;
}
}
/** Await later without an unhandled rejection in the meantime. */
const settle = async <T>(
promise: Promise<T>,
): Promise<{ value: T } | { error: unknown }> => {
try {
return { value: await promise };
} catch (error) {
return { error };
}
};
@@ -184,8 +184,7 @@ describe('compareMigrationFilenames', () => {
//
// `read()` normally goes to the replica batcher; when the replica side is
// degraded (batcher load-shed or a transient connection error) and a real
// replica is configured, the read retries once on the primary read batcher.
// The write batcher is left unset, so a failover that reached it would throw.
// replica is configured, the read retries once on the primary batcher.
type Batcher = { execute: ReturnType<typeof vi.fn> };
@@ -201,7 +200,7 @@ const makeClient = (opts: {
// inject the batchers directly. Configuration enum: SINGLE=0, REPLICA=1.
Object.assign(client as unknown as Record<string, unknown>, {
dbReplica: opts.replica,
dbPrimaryRead: opts.primary,
db: opts.primary,
configuration: opts.multiNode === false ? 0 : 1,
});
return client;
@@ -285,10 +284,6 @@ const startClient = async (
// must answer with one row-set per statement plus one for the SELECT 1.
const rowSets = (...sets: unknown[][]) => [[...sets, [{ 1: 1 }]], undefined];
// Writes go out one statement at a time, so the driver answers with just the
// result header.
const writeHeader = (header: Record<string, unknown>) => [header, undefined];
beforeEach(() => {
createdPools.length = 0;
createPoolMock.mockClear();
@@ -308,9 +303,6 @@ describe('MySQLDatabaseClient — pool construction', () => {
expect(createdPools[0].poolConfig).toEqual({
maxPreparedStatements: 900,
connectionLimit: 30,
maxIdle: 29,
idleTimeout: 60_000,
gracefulEnd: true,
enableKeepAlive: true,
host: 'db.internal',
port: 3307,
@@ -321,32 +313,6 @@ describe('MySQLDatabaseClient — pool construction', () => {
});
});
// mysql2 skips its idle reaper unless maxIdle < connectionLimit, and an
// unreaped connection is closed by the server under the pool's feet.
it('keeps the idle reaper on when a pool sets its own connection limit', async () => {
await startClient({
replica: { host: 'replica.internal', connectionLimit: 10 },
});
expect(createdPools[1].poolConfig).toMatchObject({
connectionLimit: 10,
maxIdle: 9,
idleTimeout: 60_000,
});
});
// mysql2 reads 0 as unlimited; a maxIdle of -1 would crash its reaper.
it('leaves maxIdle to mysql2 for an unlimited pool', async () => {
await startClient({
replica: { host: 'replica.internal', connectionLimit: 0 },
});
expect(createdPools[1].poolConfig).toMatchObject({
connectionLimit: 0,
});
expect(createdPools[1].poolConfig).not.toHaveProperty('maxIdle');
});
it('falls back to loopback defaults when the endpoint is unspecified', async () => {
await startClient();
@@ -435,129 +401,10 @@ describe('MySQLDatabaseClient — query interface', () => {
]);
});
it.each([
{ method: 'pread', poolIndex: 0, maxTimeInQueue: 30 },
{ method: 'read', poolIndex: 1, maxTimeInQueue: 10 },
] as const)(
'$method: flushes lone reads at once until 8 are in flight, then waits $maxTimeInQueue ms and coalesces',
async ({ method, poolIndex, maxTimeInQueue }) => {
const client = await startClient({ replica: { host: 'replica' } });
const pool = createdPools[poolIndex];
const held: Array<() => void> = [];
pool.respond = (sql) =>
new Promise((resolve) => {
const sets = sql
.split(';')
.slice(0, -1)
.map((stmt) => [{ stmt }]);
held.push(() => resolve(rowSets(...sets)));
});
vi.useFakeTimers({
toFake: [
'setTimeout',
'clearTimeout',
'setImmediate',
'clearImmediate',
],
});
try {
const reads: Promise<unknown>[] = [];
for (let i = 0; i < 8; i++) {
reads.push(client[method](`SELECT ${i}`));
await vi.advanceTimersByTimeAsync(0);
}
expect(pool.calls).toHaveLength(8);
reads.push(client[method]('SELECT 8'));
await vi.advanceTimersByTimeAsync(maxTimeInQueue - 1);
reads.push(client[method]('SELECT 9'));
expect(pool.calls).toHaveLength(8);
await vi.advanceTimersByTimeAsync(1);
expect(pool.calls.map((c) => c.sql).at(-1)).toBe(
'SELECT 8;SELECT 9; SELECT 1',
);
for (const release of held.splice(0)) release();
const rows = await Promise.all(reads);
expect(rows.at(-2)).toEqual([{ stmt: 'SELECT 8' }]);
expect(rows.at(-1)).toEqual([{ stmt: 'SELECT 9' }]);
} finally {
vi.useRealTimers();
}
},
);
it.each([
{ setup: 'single node', replica: undefined, waits: true },
{ setup: 'replica', replica: { host: 'replica' }, waits: false },
])(
'$setup: 8 primary reads in flight hold back a lone read: $waits',
async ({ replica, waits }) => {
const client = await startClient(replica ? { replica } : {});
const held: Array<() => void> = [];
for (const pool of createdPools) {
pool.respond = (sql) =>
new Promise((resolve) => {
const sets = sql
.split(';')
.slice(0, -1)
.map((stmt) => [{ stmt }]);
held.push(() => resolve(rowSets(...sets)));
});
}
const sent = () => createdPools.flatMap((pool) => pool.calls);
vi.useFakeTimers({
toFake: [
'setTimeout',
'clearTimeout',
'setImmediate',
'clearImmediate',
],
});
try {
const reads: Promise<unknown>[] = [];
for (let i = 0; i < 8; i++) {
reads.push(client.pread(`SELECT ${i}`));
await vi.advanceTimersByTimeAsync(0);
}
reads.push(client.read('SELECT 8'));
await vi.advanceTimersByTimeAsync(0);
expect(sent()).toHaveLength(waits ? 8 : 9);
await vi.advanceTimersByTimeAsync(10);
expect(sent()).toHaveLength(9);
for (const release of held.splice(0)) release();
await Promise.all(reads);
} finally {
vi.useRealTimers();
}
},
);
it('sends a write on its own, outside a transaction', async () => {
const client = await startClient();
const pool = createdPools[0];
pool.respond = () => writeHeader({ affectedRows: 1 });
await client.write('UPDATE `user` SET `username` = ? WHERE `id` = ?', [
'ada',
1,
]);
expect(pool.calls).toEqual([
{
sql: 'UPDATE `user` SET `username` = ? WHERE `id` = ?',
values: ['ada', 1],
},
]);
expect(pool.connections[0].beginTransaction).not.toHaveBeenCalled();
expect(pool.connections[0].commit).not.toHaveBeenCalled();
});
it('maps the mysql result header onto the shared write result', async () => {
const client = await startClient();
createdPools[0].respond = () =>
writeHeader({ insertId: 42, affectedRows: 2 });
rowSets({ insertId: 42, affectedRows: 2 } as unknown as unknown[]);
await expect(
client.write('UPDATE `user` SET `username` = ? WHERE `id` = ?', [
@@ -571,26 +418,9 @@ describe('MySQLDatabaseClient — query interface', () => {
});
});
it('passes the warning count through, so callers can tell a clean write', async () => {
const client = await startClient();
createdPools[0].respond = () =>
writeHeader({ insertId: 9, affectedRows: 1, warningStatus: 1 });
await expect(
client.write('INSERT IGNORE INTO `user` (`username`) VALUES (?)', [
'ada',
]),
).resolves.toEqual({
insertId: 9,
affectedRows: 1,
anyRowsAffected: true,
warningStatus: 1,
});
});
it('reports no rows affected when the header omits the counters', async () => {
const client = await startClient();
createdPools[0].respond = () => writeHeader({});
createdPools[0].respond = () => rowSets({} as unknown as unknown[]);
await expect(client.write('DELETE FROM `user`')).resolves.toEqual({
insertId: 0,
@@ -602,14 +432,17 @@ describe('MySQLDatabaseClient — query interface', () => {
it('builds an INSERT with quoted identifiers and positional params', async () => {
const client = await startClient();
const pool = createdPools[0];
pool.respond = () => writeHeader({ insertId: 7, affectedRows: 1 });
pool.respond = () =>
rowSets({ insertId: 7, affectedRows: 1 } as unknown as unknown[]);
await expect(
client.insert('user', { username: 'ada', email: null }),
).resolves.toMatchObject({ insertId: 7 });
expect(pool.calls[0]).toEqual({
sql: 'INSERT INTO `user` (`username`, `email`) VALUES (?, ?)',
sql:
'INSERT INTO `user` (`username`, `email`) VALUES (?, ?); ' +
'SELECT 1',
values: ['ada', null],
});
});
@@ -735,21 +568,6 @@ describe('MySQLDatabaseClient — tryHardRead', () => {
{ from: 'primary' },
]);
});
it('reads the primary once when the replica fails in a way read() fails over on', async () => {
const replica = {
execute: vi.fn().mockRejectedValue(codedError('dbBatchFailed')),
};
const primary = {
execute: vi.fn().mockResolvedValue([[{ from: 'primary' }]]),
};
const client = makeClient({ replica, primary });
await expect(client.tryHardRead('SELECT 1')).resolves.toEqual([
{ from: 'primary' },
]);
expect(primary.execute).toHaveBeenCalledTimes(1);
});
});
describe('MySQLDatabaseClient — migrations', () => {
@@ -827,6 +645,46 @@ describe('MySQLDatabaseClient — migrations', () => {
});
describe('MySQLDatabaseClient — pool lifecycle', () => {
it('replaces and closes the previous primary pool on reinit', async () => {
const client = await startClient();
const original = createdPools[0];
client.reinitPrimary();
expect(createdPools).toHaveLength(2);
expect(original.ended).toBe(true);
// Single-node: the replica batcher must follow the new primary.
createdPools[1].respond = () => rowSets([{ from: 'new-primary' }]);
await expect(client.read('SELECT 1')).resolves.toEqual([
{ from: 'new-primary' },
]);
});
it('replaces only the replica pool on replica reinit', async () => {
const client = await startClient({ replica: { host: 'replica' } });
const [primary, originalReplica] = createdPools;
client.reinitReplica();
expect(createdPools).toHaveLength(3);
expect(originalReplica.ended).toBe(true);
expect(primary.ended).toBe(false);
});
it('ignores a replica reinit when no replica is configured', async () => {
const client = await startClient();
client.reinitReplica();
expect(createdPools).toHaveLength(1);
});
it('ignores reinit once shutdown has started', async () => {
const client = await startClient();
await client.onServerPrepareShutdown();
client.reinitPrimary();
expect(createdPools).toHaveLength(1);
await client.onServerShutdown();
});
it('closes both pools on shutdown when a replica is configured', async () => {
const client = await startClient({ replica: { host: 'replica' } });
await client.onServerShutdown();
@@ -835,19 +693,31 @@ describe('MySQLDatabaseClient — pool lifecycle', () => {
expect(createdPools[1].ended).toBe(true);
});
it('leaves the pools open through prepare-shutdown; only onServerShutdown closes them', async () => {
it('closes the pools once the drain window elapses', async () => {
vi.useFakeTimers();
try {
const client = await startClient();
await client.onServerPrepareShutdown();
expect(createdPools[0].ended).toBe(false);
// No timer closes the pools on its own — the layers above are
// still draining through them for as long as the process drains.
await vi.advanceTimersByTimeAsync(60_000);
expect(createdPools[0].ended).toBe(false);
expect(createdPools[0].ended).toBe(true);
} finally {
vi.useRealTimers();
}
});
it('cancels the drain timer when shutdown arrives first', async () => {
vi.useFakeTimers();
try {
const client = await startClient();
await client.onServerPrepareShutdown();
// A second prepare is a no-op — only one drain timer may exist.
await client.onServerPrepareShutdown();
expect(vi.getTimerCount()).toBe(1);
await client.onServerShutdown();
expect(vi.getTimerCount()).toBe(0);
expect(createdPools[0].ended).toBe(true);
} finally {
vi.useRealTimers();
@@ -870,3 +740,108 @@ describe('MySQLDatabaseClient — pool lifecycle', () => {
await expect(client.onServerShutdown()).rejects.toThrow('end exploded');
});
});
describe('MySQLDatabaseClient.readWithRetry', () => {
const codedFailure = (code: string) => {
const err = new Error(code) as Error & { code: string };
err.code = code;
return err;
};
it('classifies transient connection failures as retriable', () => {
expect(
MySQLDatabaseClient.isRetriableError(codedFailure('ECONNRESET')),
).toBe(true);
expect(
MySQLDatabaseClient.isRetriableError(codedFailure('ER_DUP_ENTRY')),
).toBe(false);
expect(
MySQLDatabaseClient.isRetriableError(new Error('Connection lost')),
).toBe(true);
});
it('retries a transient failure and returns the eventual result', async () => {
const client = await startClient();
let attempts = 0;
const operation = vi.fn(async () => {
attempts += 1;
if (attempts < 3) throw codedFailure('PROTOCOL_CONNECTION_LOST');
return [{ ok: 1 }];
});
await expect(
client.readWithRetry('health', operation, { baseBackoffMs: 0 }),
).resolves.toEqual([{ ok: 1 }]);
expect(operation).toHaveBeenCalledTimes(3);
});
it('gives up after the attempt budget is spent', async () => {
const client = await startClient();
const operation = vi.fn(async () => {
throw codedFailure('ETIMEDOUT');
});
await expect(
client.readWithRetry('health', operation, {
maxAttempts: 2,
baseBackoffMs: 0,
}),
).rejects.toMatchObject({ code: 'ETIMEDOUT' });
expect(operation).toHaveBeenCalledTimes(2);
});
it('never retries a deterministic SQL error', async () => {
const client = await startClient();
const operation = vi.fn(async () => {
throw codedFailure('ER_PARSE_ERROR');
});
await expect(
client.readWithRetry('health', operation, { baseBackoffMs: 0 }),
).rejects.toMatchObject({ code: 'ER_PARSE_ERROR' });
expect(operation).toHaveBeenCalledTimes(1);
});
it('stops retrying once shutdown has started', async () => {
const client = await startClient();
await client.onServerPrepareShutdown();
const operation = vi.fn(async () => {
throw codedFailure('ECONNRESET');
});
await expect(
client.readWithRetry('health', operation, { baseBackoffMs: 0 }),
).rejects.toMatchObject({ code: 'ECONNRESET' });
expect(operation).toHaveBeenCalledTimes(1);
await client.onServerShutdown();
});
it('applies jittered exponential backoff capped at maxBackoffMs', async () => {
const client = await startClient();
const delays: number[] = [];
const realSetTimeout = globalThis.setTimeout;
const timeoutSpy = vi
.spyOn(globalThis, 'setTimeout')
.mockImplementation(((fn: () => void, ms?: number) => {
delays.push(ms ?? 0);
return realSetTimeout(fn, 0);
}) as typeof setTimeout);
try {
const operation = vi.fn(async () => {
throw codedFailure('ECONNRESET');
});
await expect(
client.readWithRetry('health', operation, {
maxAttempts: 4,
baseBackoffMs: 100,
maxBackoffMs: 150,
jitterRatio: 0,
}),
).rejects.toMatchObject({ code: 'ECONNRESET' });
expect(delays).toEqual([100, 150, 150]);
} finally {
timeoutSpy.mockRestore();
}
});
});
@@ -17,18 +17,19 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { readdirSync, readFileSync } from 'fs';
import { isAbsolute, resolve as resolvePath } from 'path';
import { metrics } from '@opentelemetry/api';
import { createPool, ExecuteValues, type Pool } from 'mysql2';
import { createPool, type Pool } from 'mysql2';
import { Span } from '../../util/span.js';
import { AbstractDatabaseClient, type WriteResult } from './DatabaseClient';
import { SQLBatcher } from './SQLBatcher.js';
import { isRetriableError } from './retriableErrors.js';
import { splitMysqlStatements } from './splitMysqlStatements.js';
import { compareMigrationFilenames } from './migrationFilenames.js';
import type { IConfig } from '../../types';
const DEFAULT_SELECT_TIMEOUT_MS = 30_000;
const DEFAULT_CONNECTION_LIMIT = 30;
const IDLE_TIMEOUT_MS = 60_000;
const replicaFailoverCounter = metrics
.getMeter('puter-backend')
@@ -37,7 +38,7 @@ const replicaFailoverCounter = metrics
'Reads that failed on the replica batcher and were retried on the primary',
});
export { compareMigrationFilenames } from './migrationFilenames.js';
export { compareMigrationFilenames };
type PoolConfig = Parameters<typeof createPool>[0];
@@ -53,10 +54,9 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
private replicaPool!: Pool;
private db!: SQLBatcher;
private dbReplica!: SQLBatcher;
/** Primary pool, SELECT-only: same rows as `db`, coalesced like the replica. */
private dbPrimaryRead!: SQLBatcher;
private configuration = Configuration.SINGLE;
private shutdownStarted = false;
private shutdownTimer: ReturnType<typeof setTimeout> | null = null;
constructor(config: IConfig) {
super(config);
@@ -78,13 +78,7 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
});
console.log('[mysql] connected to primary');
// Read batchers on one pool share the cap on immediate flushes.
const primaryReadFlushes = { count: 0 };
this.db = this.createPrimaryBatcher(this.primaryPool);
this.dbPrimaryRead = this.createPrimaryReadBatcher(
this.primaryPool,
primaryReadFlushes,
);
if (dbConf.replica) {
this.replicaPool = this.createPool(dbConf.replica);
@@ -95,30 +89,38 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
this.configuration = Configuration.SINGLE;
}
this.dbReplica = this.createReplicaBatcher(
this.replicaPool,
this.configuration === Configuration.SINGLE
? primaryReadFlushes
: { count: 0 },
);
this.dbReplica = this.createReplicaBatcher(this.replicaPool);
await this.applyMigrationPaths(
() => this.primaryPool.promise().getConnection(),
splitMysqlStatements,
{ transactional: false },
);
await this.runMigrations();
}
override async onServerPrepareShutdown(): Promise<void> {
if (this.shutdownStarted) return;
this.shutdownStarted = true;
// Pools stay open; onServerShutdown closes them after the layers
// above have drained.
console.log('[mysql] entering drain mode');
// Allow in-flight queries to drain before closing pools
const drainMs = 60_000;
console.log(
`[mysql] draining in-flight queries (${drainMs}ms) before closing pools`,
);
this.shutdownTimer = setTimeout(() => {
this.shutdownTimer = null;
this.closeCurrentPools('drain').catch((e) =>
console.error('[mysql] error closing pools after drain', e),
);
}, drainMs);
if (typeof this.shutdownTimer.unref === 'function') {
this.shutdownTimer.unref();
}
}
override async onServerShutdown(): Promise<void> {
if (this.shutdownTimer) {
clearTimeout(this.shutdownTimer);
this.shutdownTimer = null;
}
await this.closeCurrentPools('shutdown');
}
@@ -149,7 +151,7 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
throw error;
}
replicaFailoverCounter.add(1);
result = await this.dbPrimaryRead.execute(query, params);
result = await this.db.execute(query, params);
}
if (!result) return [];
return (result[0] as Record<string, unknown>[]) ?? [];
@@ -160,7 +162,7 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
const result = await this.dbPrimaryRead.execute(query, params);
const result = await this.db.execute(query, params);
if (!result) return [];
return (result[0] as Record<string, unknown>[]) ?? [];
}
@@ -174,14 +176,12 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
const header = result[0] as {
insertId?: number;
affectedRows?: number;
warningStatus?: number;
};
const affectedRows = header.affectedRows ?? 0;
return {
insertId: header.insertId ?? 0,
affectedRows,
anyRowsAffected: affectedRows > 0,
warningStatus: header.warningStatus,
};
}
@@ -201,7 +201,7 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
await conn.beginTransaction();
try {
for (const { statement, values } of entries) {
await conn.execute(statement, values as ExecuteValues);
await conn.execute(statement, values);
}
await conn.commit();
} catch (err) {
@@ -213,19 +213,101 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
}
}
protected override hasReadReplica(): boolean {
return this.configuration === Configuration.REPLICA;
@Span('db.tryHardRead', (query: string) => ({ 'db.statement': query }))
override async tryHardRead(
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
if (this.configuration === Configuration.SINGLE) {
return this.read(query, params);
}
// Run both reads in parallel — prefer replica when it returns rows,
// otherwise fall back to primary to handle replication lag.
const primaryPromise = this.db.execute(query, params);
try {
const replicaResult = await this.dbReplica.execute(query, params);
if (
Array.isArray(replicaResult?.[0]) &&
(replicaResult[0] as unknown[]).length > 0
) {
primaryPromise.catch(() => {}); // suppress unhandled rejection
return replicaResult[0] as Record<string, unknown>[];
}
} catch {
// fall through to primary
}
const primaryResult = await primaryPromise;
return (primaryResult?.[0] as Record<string, unknown>[]) ?? [];
}
// No failover: `tryHardRead` falls back to the primary on an empty result
// or error (raced alongside by default, after the replica with `race: false`).
protected override async replicaRead(
query: string,
params: unknown[],
): Promise<Record<string, unknown>[]> {
const result = await this.dbReplica.execute(query, params);
if (!result) return [];
return (result[0] as Record<string, unknown>[]) ?? [];
// ------------------------------------------------------------------
// Migrations
// ------------------------------------------------------------------
/**
* Apply `.sql` files from each configured migration directory in order.
* Files within a directory are sorted lexically. Files MUST be idempotent —
* there is no per-file applied-state tracking. Failures abort startup so
* operators see schema problems loud.
*/
private async runMigrations(): Promise<void> {
const paths = this.config.database?.migrationPaths;
if (!paths || paths.length === 0) return;
const conn = await this.primaryPool.promise().getConnection();
try {
for (const rawPath of paths) {
const dir = isAbsolute(rawPath)
? rawPath
: resolvePath(process.cwd(), rawPath);
let files: string[];
try {
files = readdirSync(dir)
.filter(
(f) => f.endsWith('.sql') && f.startsWith('mysql'),
)
.sort(compareMigrationFilenames);
} catch (e) {
throw new Error(
`[mysql] migration path is unreadable: ${dir}`,
{ cause: e },
);
}
if (files.length === 0) {
console.log(`[mysql] no migrations in ${dir}`);
continue;
}
console.log(
`[mysql] running migrations from ${dir}: ${files.length} file(s)`,
);
for (const file of files) {
const filePath = resolvePath(dir, file);
const contents = readFileSync(filePath, 'utf8');
const statements = splitMysqlStatements(contents);
for (let i = 0; i < statements.length; i++) {
try {
await conn.query(statements[i]);
} catch (e) {
throw new Error(
`[mysql] failed to apply ${file} at statement ${i}`,
{ cause: e },
);
}
}
console.log(
`[mysql] applied ${file} (${statements.length} statements)`,
);
}
}
} finally {
conn.release();
}
}
// ------------------------------------------------------------------
@@ -233,19 +315,9 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
// ------------------------------------------------------------------
private createPool(poolConfig: PoolConfig): Pool {
const connectionLimit =
poolConfig.connectionLimit ?? DEFAULT_CONNECTION_LIMIT;
const pool = createPool({
maxPreparedStatements: 900,
connectionLimit,
// mysql2 only reaps idle connections when maxIdle < connectionLimit.
// Unreaped, they sit until the server's wait_timeout closes them,
// and the next statement sent on one fails. 0 means unlimited,
// and a negative maxIdle would crash the reaper.
...(connectionLimit > 1 && { maxIdle: connectionLimit - 1 }),
idleTimeout: IDLE_TIMEOUT_MS,
// Reaped connections say goodbye instead of dropping the socket.
gracefulEnd: true,
connectionLimit: 30,
enableKeepAlive: true,
...poolConfig,
multipleStatements: true,
@@ -273,43 +345,18 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
return pool;
}
/**
* One write per flush. Coalesced writes share a transaction, so unrelated
* callers hold each other's row locks until COMMIT and deadlock; a lone
* statement needs no BEGIN/COMMIT round trips.
*/
private createPrimaryBatcher(pool: Pool): SQLBatcher {
return new SQLBatcher(pool, {
maxBatchSize: 1,
poolLabel: 'primary',
acquireTimeoutMs: this.config.database?.acquireTimeoutMs,
});
}
/** Reads that must see the primary, still coalesced. */
private createPrimaryReadBatcher(
pool: Pool,
inFlight: { count: number },
): SQLBatcher {
return new SQLBatcher(pool, {
maxTimeInQueue: 30,
maxInFlightImmediate: 8,
inFlight,
maxBatchSize: 5,
poolLabel: 'primary',
readOnly: true,
acquireTimeoutMs: this.config.database?.acquireTimeoutMs,
});
}
private createReplicaBatcher(
pool: Pool,
inFlight: { count: number },
): SQLBatcher {
private createReplicaBatcher(pool: Pool): SQLBatcher {
return new SQLBatcher(pool, {
maxTimeInQueue: 10,
maxInFlightImmediate: 8,
inFlight,
maxBatchSize: 5,
poolLabel: 'replica',
readOnly: true,
@@ -317,6 +364,56 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
});
}
/** Reinitialize the primary pool (e.g. after a health-check failure). */
reinitPrimary(): void {
if (this.shutdownStarted) return;
const dbConf = this.config.database!;
const previous = this.primaryPool;
this.primaryPool = this.createPool({
host: dbConf.host ?? '127.0.0.1',
port: dbConf.port ?? 3306,
user: dbConf.user ?? 'root',
password: dbConf.password ?? '',
database: dbConf.database ?? 'puter',
});
this.db = this.createPrimaryBatcher(this.primaryPool);
if (this.configuration === Configuration.SINGLE) {
this.replicaPool = this.primaryPool;
this.dbReplica = this.createReplicaBatcher(this.primaryPool);
}
if (previous && previous !== this.primaryPool) {
this.closePool(previous, 'reinit:primary').catch(() => {});
}
}
/** Reinitialize the replica pool. */
reinitReplica(): void {
if (this.shutdownStarted || !this.config.database?.replica) return;
const previous = this.replicaPool;
this.replicaPool = this.createPool(this.config.database.replica);
this.dbReplica = this.createReplicaBatcher(this.replicaPool);
if (
previous &&
previous !== this.replicaPool &&
previous !== this.primaryPool
) {
this.closePool(previous, 'reinit:replica').catch(() => {});
}
}
// ------------------------------------------------------------------
// Retry helpers (for health checks or resilient reads)
// ------------------------------------------------------------------
static isRetriableError(error: unknown): boolean {
return isRetriableError(error);
}
/**
* Replica failures worth retrying on the primary: batcher load-shed or
* transient connection errors — never deterministic SQL errors.
@@ -326,6 +423,52 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
return code === 'dbBatchFailed' || isRetriableError(error);
}
async readWithRetry(
label: string,
operation: () => Promise<unknown[]>,
opts?: {
maxAttempts?: number;
baseBackoffMs?: number;
maxBackoffMs?: number;
jitterRatio?: number;
},
): Promise<unknown[]> {
const maxAttempts = opts?.maxAttempts ?? 3;
const baseBackoffMs = opts?.baseBackoffMs ?? 100;
const maxBackoffMs = opts?.maxBackoffMs ?? 500;
const jitterRatio = opts?.jitterRatio ?? 0.2;
let attempt = 1;
while (true) {
try {
return await operation();
} catch (error) {
if (this.shutdownStarted) throw error;
if (
attempt >= maxAttempts ||
!MySQLDatabaseClient.isRetriableError(error)
)
throw error;
const raw = baseBackoffMs * 2 ** (attempt - 1);
const capped = Math.min(maxBackoffMs, raw);
const window = Math.round(capped * jitterRatio);
const jitter =
window === 0
? 0
: Math.floor(Math.random() * (window * 2 + 1)) - window;
const delay = Math.max(0, capped + jitter);
console.warn(
`[${label}] transient mysql error (${(error as { code?: string })?.code ?? 'unknown'}); retry ${attempt + 1}/${maxAttempts} in ${delay}ms`,
);
await new Promise((r) => setTimeout(r, delay));
attempt++;
}
}
}
// ------------------------------------------------------------------
// Internal pool lifecycle
// ------------------------------------------------------------------
@@ -18,7 +18,7 @@
*/
import type { FieldDef, QueryResult } from 'pg';
import { describe, expect, it, vi } from 'vitest';
import { describe, expect, it } from 'vitest';
import type { IConfig } from '../../types';
import { DatabaseClientFactory } from './index.js';
import {
@@ -28,20 +28,6 @@ import {
type PostgresPoolClient,
} from './PostgresDatabaseClient.js';
// Pass-through spy: the rewrite is pure, so its call count shows the memo.
const { rewriteSpy } = vi.hoisted(() => ({ rewriteSpy: vi.fn() }));
vi.mock('./preparePostgresSql.js', async (importOriginal) => {
const actual =
await importOriginal<typeof import('./preparePostgresSql.js')>();
return {
...actual,
preparePostgresSql: (sql: string) => {
rewriteSpy(sql);
return actual.preparePostgresSql(sql);
},
};
});
type QueryCall = {
text: string;
values?: unknown[];
@@ -181,23 +167,6 @@ describe('PostgresDatabaseClient', () => {
});
});
it('rewrites a repeated query once', async () => {
const pool = new RecordingPool();
const client = new PostgresDatabaseClient(postgresConfig(), () => pool);
await client.onServerStart();
const query = 'SELECT `name` FROM `apps` WHERE `id` = ?';
rewriteSpy.mockClear();
await client.read(query, [1]);
await client.pread(query, [2]);
await client.write(query, [3]);
expect(rewriteSpy).toHaveBeenCalledTimes(1);
expect(pool.calls.slice(-3).map((c) => c.text)).toEqual(
Array(3).fill('SELECT "name" FROM "apps" WHERE "id" = $1'),
);
});
it('normalizes int8 fields on read and primary read rows', async () => {
const pool = new RecordingPool(
new RecordingPoolClient(),
@@ -388,33 +357,4 @@ describe('PostgresDatabaseClient', () => {
]);
expect(conn.released).toBe(true);
});
it('leaves the pool open through prepare-shutdown; only onServerShutdown closes it', async () => {
vi.useFakeTimers();
try {
const pool = new RecordingPool();
let ended = false;
pool.end = async () => {
ended = true;
};
const client = new PostgresDatabaseClient(
postgresConfig(),
() => pool,
);
await client.onServerStart();
await client.onServerPrepareShutdown();
expect(ended).toBe(false);
// No timer closes the pool on its own — the layers above are
// still draining through it for as long as the process drains.
await vi.advanceTimersByTimeAsync(60_000);
expect(ended).toBe(false);
await client.onServerShutdown();
expect(ended).toBe(true);
} finally {
vi.useRealTimers();
}
});
});
@@ -17,14 +17,16 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { readdirSync, readFileSync } from 'fs';
import { isAbsolute, resolve as resolvePath } from 'path';
import { Pool, type PoolConfig, type QueryResult } from 'pg';
import { BoundedTtlMap } from '../../util/boundedTtlMap.js';
import { Span } from '../../util/span.js';
import {
AbstractDatabaseClient,
type BatchEntry,
type WriteResult,
} from './DatabaseClient';
import { compareMigrationFilenames } from './migrationFilenames.js';
import { preparePostgresSql } from './preparePostgresSql.js';
import { splitPostgresStatements } from './splitPostgresStatements.js';
import type { IConfig } from '../../types';
@@ -60,7 +62,6 @@ enum Configuration {
}
const POSTGRES_INT8_OID = 20;
const PREPARED_SQL_CACHE_SIZE = 1000;
const INTEGER_TEXT_PATTERN = /^-?\d+$/u;
const normalizePostgresInt8 = (
@@ -137,10 +138,7 @@ export class PostgresDatabaseClient extends AbstractDatabaseClient {
private replicaPool!: PostgresPool;
private configuration = Configuration.SINGLE;
private shutdownStarted = false;
/** Query text as written → its Postgres rewrite, which is pure. */
private readonly preparedSql = new BoundedTtlMap<string, string>({
maxEntries: PREPARED_SQL_CACHE_SIZE,
});
private shutdownTimer: ReturnType<typeof setTimeout> | null = null;
constructor(
config: IConfig,
@@ -167,23 +165,35 @@ export class PostgresDatabaseClient extends AbstractDatabaseClient {
this.configuration = Configuration.SINGLE;
}
await this.applyMigrationPaths(
() => this.primaryPool.connect(),
splitPostgresStatements,
{ transactional: true },
);
await this.runMigrations();
}
override async onServerPrepareShutdown(): Promise<void> {
if (this.shutdownStarted) return;
this.shutdownStarted = true;
// Pools stay open; onServerShutdown closes them after the layers above
// have drained through them.
console.log('[postgres] entering drain mode');
const drainMs = 60_000;
console.log(
`[postgres] draining in-flight queries (${drainMs}ms) before closing pools`,
);
this.shutdownTimer = setTimeout(() => {
this.shutdownTimer = null;
this.closeCurrentPools().catch((e) =>
console.error('[postgres] error closing pools after drain', e),
);
}, drainMs);
if (typeof this.shutdownTimer.unref === 'function') {
this.shutdownTimer.unref();
}
}
override async onServerShutdown(): Promise<void> {
if (this.shutdownTimer) {
clearTimeout(this.shutdownTimer);
this.shutdownTimer = null;
}
await this.closeCurrentPools();
}
@@ -255,8 +265,98 @@ export class PostgresDatabaseClient extends AbstractDatabaseClient {
}
}
protected override hasReadReplica(): boolean {
return this.configuration === Configuration.REPLICA;
@Span('db.tryHardRead', (query: string) => ({ 'db.statement': query }))
override async tryHardRead(
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
if (this.configuration === Configuration.SINGLE) {
return this.read(query, params);
}
const primaryPromise = this.query(this.primaryPool, query, params);
try {
const replicaResult = await this.query(
this.replicaPool,
query,
params,
);
if (replicaResult.rows.length > 0) {
primaryPromise.catch(() => {});
return normalizePostgresRows(replicaResult);
}
} catch {
// fall through to primary
}
const primaryResult = await primaryPromise;
return normalizePostgresRows(primaryResult);
}
private async runMigrations(): Promise<void> {
const paths = this.config.database?.migrationPaths;
if (!paths || paths.length === 0) return;
const conn = await this.primaryPool.connect();
try {
for (const rawPath of paths) {
const dir = isAbsolute(rawPath)
? rawPath
: resolvePath(process.cwd(), rawPath);
let files: string[];
try {
files = readdirSync(dir)
.filter(
(f) =>
f.endsWith('.sql') && f.startsWith('postgres'),
)
.sort(compareMigrationFilenames);
} catch (e) {
throw new Error(
`[postgres] migration path is unreadable: ${dir}`,
{ cause: e },
);
}
if (files.length === 0) {
console.log(`[postgres] no migrations in ${dir}`);
continue;
}
console.log(
`[postgres] running migrations from ${dir}: ${files.length} file(s)`,
);
for (const file of files) {
const filePath = resolvePath(dir, file);
const contents = readFileSync(filePath, 'utf8');
const statements = splitPostgresStatements(contents);
await conn.query('BEGIN');
try {
for (let i = 0; i < statements.length; i++) {
try {
await conn.query(statements[i]);
} catch (e) {
throw new Error(
`[postgres] failed to apply ${file} at statement ${i}`,
{ cause: e },
);
}
}
await conn.query('COMMIT');
} catch (e) {
await conn.query('ROLLBACK').catch(() => {});
throw e;
}
console.log(
`[postgres] applied ${file} (${statements.length} statements)`,
);
}
}
} finally {
conn.release();
}
}
private createPool(dbConf: PostgresEndpointConfig): PostgresPool {
@@ -283,12 +383,8 @@ export class PostgresDatabaseClient extends AbstractDatabaseClient {
query: string,
params: unknown[] = [],
): Promise<QueryResult> {
let text = this.preparedSql.get(query);
if (text === undefined) {
text = preparePostgresSql(query).text;
this.preparedSql.set(query, text);
}
return target.query(text, params);
const prepared = preparePostgresSql(query);
return target.query(prepared.text, params);
}
private async closeCurrentPools(): Promise<void> {
+44 -121
View File
@@ -76,7 +76,6 @@ const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
export class SQLBatcher {
dbPool;
maxTimeInQueue;
maxInFlightImmediate;
maxBatchSize;
maxQueueSize;
failureThreshold;
@@ -85,9 +84,7 @@ export class SQLBatcher {
readOnly;
acquireTimeoutMs;
queue = [];
#flushTimer = null;
#flushTimerIsImmediate = false;
#inFlight;
timeouts = [];
#consecutiveFailures = 0;
#lastFailureAt = 0;
#metricAttrs;
@@ -96,11 +93,6 @@ export class SQLBatcher {
* @param {object} dbPool Mysql2 pool
* @param {object} [opts]
* @param {number} [opts.maxTimeInQueue] Ms an item may wait before flush
* @param {number} [opts.maxInFlightImmediate] While fewer flushes than this
* are in flight, a lone item flushes on the next turn instead of waiting
* maxTimeInQueue; past it, items wait and coalesce to spare connections
* @param {{ count: number }} [opts.inFlight] In-flight flush counter the
* cap reads; batchers sharing a pool can share one
* @param {number} [opts.maxBatchSize] Items coalesced per flush
* @param {number} [opts.maxQueueSize] Drop-oldest high-water mark
* @param {number} [opts.failureThreshold] Consecutive failures to open the
@@ -119,8 +111,6 @@ export class SQLBatcher {
dbPool,
{
maxTimeInQueue = 20,
maxInFlightImmediate = 0,
inFlight = { count: 0 },
maxBatchSize = 50,
maxQueueSize = DEFAULT_MAX_QUEUE_SIZE,
failureThreshold = DEFAULT_FAILURE_THRESHOLD,
@@ -132,8 +122,6 @@ export class SQLBatcher {
) {
this.dbPool = dbPool;
this.maxTimeInQueue = maxTimeInQueue;
this.maxInFlightImmediate = maxInFlightImmediate;
this.#inFlight = inFlight;
this.maxBatchSize = maxBatchSize;
this.maxQueueSize = maxQueueSize;
this.failureThreshold = failureThreshold;
@@ -197,34 +185,17 @@ export class SQLBatcher {
if (this.queue.length >= this.maxBatchSize) {
this.flush(this.queue.splice(0, this.maxBatchSize));
} else if (this.#flushTimer === null) {
this.#armFlushTimer();
} else if (this.queue.length === 1) {
this.timeouts.push(
setTimeout(() => {
this.flush(this.queue.splice(0, this.queue.length));
}, this.maxTimeInQueue),
);
}
return promise;
}
// A lone flush holds a pooled connection for a full round trip, so only
// the first maxInFlightImmediate flushes skip the wait.
#armFlushTimer() {
const run = () => {
this.#flushTimer = null;
this.flush(this.queue.splice(0, this.queue.length));
};
this.#flushTimerIsImmediate =
this.#inFlight.count < this.maxInFlightImmediate;
this.#flushTimer = this.#flushTimerIsImmediate
? setImmediate(run)
: setTimeout(run, this.maxTimeInQueue);
}
#clearFlushTimer() {
if (this.#flushTimer === null) return;
if (this.#flushTimerIsImmediate) clearImmediate(this.#flushTimer);
else clearTimeout(this.#flushTimer);
this.#flushTimer = null;
}
// Bounded wait for a pooled connection. Without a bound, a stalled
// database turns every flush into an indefinite hang — nothing fails,
// so neither the breaker nor callers' own timeouts ever engage.
@@ -281,23 +252,22 @@ export class SQLBatcher {
}
async flush(batch) {
this.#clearFlushTimer();
const timeout = this.timeouts.shift();
if (timeout && !timeout._destroyed) {
clearTimeout(timeout);
}
if (batch.length === 0) return;
this.#inFlight.count++;
try {
await this.#flushBatch(batch);
} finally {
this.#inFlight.count--;
}
}
const query = `${batch.map((b) => b.sql.replace(/;+\s*$/, '')).join(';')}; SELECT 1`; // SELECT 1 forces mysql2 to return array
const values = batch.map((b) => b.values ?? []).flat();
async #flushBatch(batch) {
let connection;
try {
connection = await this.#acquireConnection();
} catch (error) {
this.#recordFlushFailure();
this.#consecutiveFailures++;
this.#lastFailureAt = Date.now();
flushFailureCounter.add(1, this.#metricAttrs);
console.warn(
'SQLBatcher could not acquire connection for flush:',
error,
@@ -308,30 +278,17 @@ export class SQLBatcher {
return;
}
if (batch.length === 1 && !this.readOnly) {
await this.#flushLoneWrite(batch[0], connection);
return;
}
const query = `${batch.map((b) => b.sql.replace(/;+\s*$/, '')).join(';')}; SELECT 1`; // SELECT 1 forces mysql2 to return array
const values = batch.map((b) => b.values ?? []).flat();
// Run the coalesced multi-statement inside an explicit transaction so
// a single bad statement (e.g. a duplicate-key INSERT) rolls back the
// whole batch atomically, leaving us free to re-run each item
// individually below. Without this, MySQL would commit every
// statement up to the failure point and a per-item retry would
// misreport already-committed inserts as duplicate-key failures.
//
// A read-only batch has nothing to roll back, and the wrapper is not
// free: BEGIN and COMMIT are round trips of their own, which triples
// the cost of a read against a pool in another region.
const wrapInTransaction = !this.readOnly;
let batchSucceeded = false;
try {
if (wrapInTransaction) await connection.beginTransaction();
await connection.beginTransaction();
const [results, fields] = await connection.query(query, values);
if (wrapInTransaction) await connection.commit();
await connection.commit();
batchSucceeded = true;
this.#consecutiveFailures = 0;
for (let i = 0; i < batch.length; i++) {
@@ -339,12 +296,10 @@ export class SQLBatcher {
b.resolve([results[i], fields?.[i]]);
}
} catch (batchError) {
if (wrapInTransaction) {
try {
await connection.rollback();
} catch (rollbackError) {
console.warn('SQLBatcher rollback failed:', rollbackError);
}
try {
await connection.rollback();
} catch (rollbackError) {
console.warn('SQLBatcher rollback failed:', rollbackError);
}
console.warn(
'SQLBatcher batch failed; retrying items individually:',
@@ -356,11 +311,10 @@ export class SQLBatcher {
if (batchSucceeded) return;
// Per-item fallback. Nothing from the batch is live — it was rolled
// back, or carried only SELECTs — so re-running each item
// independently produces clean success/failure outcomes for each
// caller. Concurrency is capped to avoid briefly saturating the pool
// when a large batch fails.
// Per-item fallback. The transaction was rolled back so no statement
// committed; re-running each item independently produces clean
// success/failure outcomes for each caller. Concurrency is capped to
// avoid briefly saturating the pool when a large batch fails.
flushFailureCounter.add(1, this.#metricAttrs);
fallbackInvocationsCounter.add(1, this.#metricAttrs);
@@ -371,7 +325,7 @@ export class SQLBatcher {
async () => {
while (cursor < batch.length) {
const i = cursor++;
settled[i] = await this.#runItem(batch[i]);
settled[i] = await this.#runFallbackItem(batch[i]);
}
},
);
@@ -405,34 +359,7 @@ export class SQLBatcher {
}
}
// A single statement is atomic under autocommit, so it skips BEGIN/COMMIT
// and their round trips. With no transaction to roll back, a failure is
// not re-run wholesale; only the per-item retry policy applies.
async #flushLoneWrite(item, connection) {
const r = await this.#runItem(item, connection);
if (r.ok) {
this.#consecutiveFailures = 0;
item.resolve(r.value);
return;
}
// An error the server answered with (duplicate key, bad SQL) means it
// is up; only failing to reach it counts toward the breaker.
if (isRetriableError(r.error)) {
this.#recordFlushFailure();
} else {
this.#consecutiveFailures = 0;
flushFailureCounter.add(1, this.#metricAttrs);
}
item.reject(r.error);
}
#recordFlushFailure() {
this.#consecutiveFailures++;
this.#lastFailureAt = Date.now();
flushFailureCounter.add(1, this.#metricAttrs);
}
// Run one item on its own, retrying transient failures with backoff.
// Run one fallback item, retrying transient failures with backoff.
// A read-only batcher may retry anything transient; a batcher that
// carries writes only retries failures where the statement provably
// did not apply — either it never reached the server, or the server
@@ -443,37 +370,33 @@ export class SQLBatcher {
// than surfacing: an item is a single statement, so a deadlock victim has
// been fully undone, and the caller sees an unhandled 500 for what the
// database is telling us to just run again.
async #runItem(b, connection) {
async #runFallbackItem(b) {
let attempt = 0;
while (true) {
if (!connection) {
try {
connection = await this.#acquireConnection();
} catch (error) {
return { ok: false, error };
}
let connection;
try {
connection = await this.#acquireConnection();
} catch (error) {
return { ok: false, error };
}
let error;
try {
return {
ok: true,
value: await connection.query(b.sql, b.values ?? []),
};
} catch (e) {
error = e;
} catch (error) {
const canRetry = this.readOnly
? isRetriableError(error) || isRolledBackError(error)
: isNeverSentError(error) || isRolledBackError(error);
if (!canRetry || attempt >= ITEM_RETRY_ATTEMPTS) {
return { ok: false, error };
}
attempt++;
fallbackItemRetriesCounter.add(1, this.#metricAttrs);
await sleep(RETRY_BASE_BACKOFF_MS * attempt);
} finally {
connection.release();
connection = undefined;
}
const canRetry = this.readOnly
? isRetriableError(error) || isRolledBackError(error)
: isNeverSentError(error) || isRolledBackError(error);
if (!canRetry || attempt >= ITEM_RETRY_ATTEMPTS) {
return { ok: false, error };
}
attempt++;
fallbackItemRetriesCounter.add(1, this.#metricAttrs);
await sleep(RETRY_BASE_BACKOFF_MS * attempt);
}
}
}
+1 -449
View File
@@ -69,356 +69,6 @@ const happyBatch = (sql: string) => {
];
};
// Each query on this connection waits until the test releases it, so its
// flush stays in flight.
const makeHeldConnection = () => {
const held: Array<() => void> = [];
const conn = makeConnection(
(sql) =>
new Promise((resolve) => {
held.push(() => resolve(happyBatch(sql)));
}),
);
const releaseAll = () => {
for (const release of held.splice(0)) release();
};
const sent = () => conn.query.mock.calls.map(([sql]) => sql);
return { conn, releaseAll, sent };
};
const fakeFlushTimers = () =>
vi.useFakeTimers({
toFake: [
'setTimeout',
'clearTimeout',
'setImmediate',
'clearImmediate',
],
});
describe('SQLBatcher flush timing', () => {
it('flushes a lone query on the next turn while under the in-flight cap', async () => {
const conn = makeConnection(happyBatch);
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxTimeInQueue: 60_000,
maxInFlightImmediate: 2,
});
const pending = batcher.query('SELECT a', []);
expect(getConnection).not.toHaveBeenCalled();
await new Promise((resolve) => setImmediate(resolve));
expect(getConnection).toHaveBeenCalledTimes(1);
expect((await pending)[0]).toEqual([{ n: 0 }]);
});
it('holds a lone query for maxTimeInQueue without an in-flight cap', async () => {
fakeFlushTimers();
try {
const conn = makeConnection(happyBatch);
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxTimeInQueue: 30,
});
const pending = batcher.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(29);
expect(getConnection).not.toHaveBeenCalled();
await vi.advanceTimersByTimeAsync(1);
expect(getConnection).toHaveBeenCalledTimes(1);
expect((await pending)[0]).toEqual([{ n: 0 }]);
} finally {
vi.useRealTimers();
}
});
it('batches queries from the same turn and sends later ones separately', async () => {
const conn = makeConnection(happyBatch);
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxInFlightImmediate: 8,
});
const sameTurn = [
batcher.query('SELECT a', []),
batcher.query('SELECT b', []),
batcher.query('SELECT c', []),
];
await new Promise((resolve) => setImmediate(resolve));
const nextTurn = batcher.query('SELECT d', []);
const results = await Promise.all([...sameTurn, nextTurn]);
expect(results.map((r) => r[0])).toEqual([
[{ n: 0 }],
[{ n: 1 }],
[{ n: 2 }],
[{ n: 0 }],
]);
expect(getConnection).toHaveBeenCalledTimes(2);
expect(conn.query.mock.calls.map(([sql]) => sql)).toEqual([
'SELECT a;SELECT b;SELECT c; SELECT 1',
'SELECT d; SELECT 1',
]);
});
it('gives a query enqueued during an in-flight flush its own flush while under the cap', async () => {
const { conn, releaseAll, sent } = makeHeldConnection();
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxInFlightImmediate: 2,
});
const turn = () => new Promise((resolve) => setImmediate(resolve));
const a = batcher.query('SELECT a', []);
await turn();
const b = batcher.query('SELECT b', []);
await turn();
expect(sent()).toEqual(['SELECT a; SELECT 1', 'SELECT b; SELECT 1']);
releaseAll();
expect((await a)[0]).toEqual([{ n: 0 }]);
expect((await b)[0]).toEqual([{ n: 0 }]);
});
it('holds a lone query for maxTimeInQueue at the in-flight cap and coalesces what follows', async () => {
fakeFlushTimers();
try {
const { conn, releaseAll, sent } = makeHeldConnection();
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxTimeInQueue: 30,
maxInFlightImmediate: 1,
});
const a = batcher.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(0);
expect(sent()).toEqual(['SELECT a; SELECT 1']);
const b = batcher.query('SELECT b', []);
await vi.advanceTimersByTimeAsync(29);
const c = batcher.query('SELECT c', []);
expect(sent()).toHaveLength(1);
await vi.advanceTimersByTimeAsync(1);
expect(sent()).toEqual([
'SELECT a; SELECT 1',
'SELECT b;SELECT c; SELECT 1',
]);
releaseAll();
expect((await a)[0]).toEqual([{ n: 0 }]);
expect((await b)[0]).toEqual([{ n: 0 }]);
expect((await c)[0]).toEqual([{ n: 1 }]);
// Settled flushes free the cap again.
const d = batcher.query('SELECT d', []);
await vi.advanceTimersByTimeAsync(0);
expect(sent()).toHaveLength(3);
releaseAll();
expect((await d)[0]).toEqual([{ n: 0 }]);
} finally {
vi.useRealTimers();
}
});
it('shares the in-flight cap between batchers given one counter', async () => {
fakeFlushTimers();
try {
const { conn, releaseAll, sent } = makeHeldConnection();
const { pool } = makePool(conn);
const inFlight = { count: 0 };
const opts = {
readOnly: true,
maxTimeInQueue: 30,
maxInFlightImmediate: 1,
inFlight,
};
const first = new SQLBatcher(pool, opts);
const second = new SQLBatcher(pool, opts);
const own = new SQLBatcher(pool, { ...opts, inFlight: undefined });
const a = first.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(0);
const b = second.query('SELECT b', []);
const c = own.query('SELECT c', []);
await vi.advanceTimersByTimeAsync(0);
expect(sent()).toEqual([
'SELECT a; SELECT 1',
'SELECT c; SELECT 1',
]);
await vi.advanceTimersByTimeAsync(30);
expect(sent()).toHaveLength(3);
expect(sent()[2]).toBe('SELECT b; SELECT 1');
releaseAll();
await Promise.all([a, b, c]);
expect(inFlight.count).toBe(0);
} finally {
vi.useRealTimers();
}
});
it('counts a flush as in flight until its per-item fallback settles', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
fakeFlushTimers();
try {
const held: Array<() => void> = [];
const conn = makeConnection((sql) => {
if (isBatchQuery(sql)) throw makeError('ECONNRESET');
return new Promise((resolve) => {
held.push(() => resolve([[{ ok: sql }], undefined]));
});
});
const { pool } = makePool(conn);
const sent = () => conn.query.mock.calls.map(([sql]) => sql);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxTimeInQueue: 30,
maxInFlightImmediate: 1,
acquireTimeoutMs: 0,
});
const a = batcher.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(0);
// The batch failed; the per-item retry of `a` is still running.
expect(sent()).toEqual(['SELECT a; SELECT 1', 'SELECT a']);
const b = batcher.query('SELECT b', []);
await vi.advanceTimersByTimeAsync(29);
expect(sent()).toHaveLength(2);
await vi.advanceTimersByTimeAsync(1);
expect(sent()).toEqual([
'SELECT a; SELECT 1',
'SELECT a',
'SELECT b; SELECT 1',
'SELECT b',
]);
for (const release of held.splice(0)) release();
expect((await a)[0]).toEqual([{ ok: 'SELECT a' }]);
expect((await b)[0]).toEqual([{ ok: 'SELECT b' }]);
} finally {
vi.useRealTimers();
warnSpy.mockRestore();
}
});
it('flushes a full batch at once past the cap and cancels the armed wait', async () => {
fakeFlushTimers();
try {
const { conn, releaseAll, sent } = makeHeldConnection();
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxBatchSize: 2,
maxTimeInQueue: 30,
maxInFlightImmediate: 1,
acquireTimeoutMs: 0,
});
const a = batcher.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(0);
const b = batcher.query('SELECT b', []);
expect(vi.getTimerCount()).toBe(1);
const c = batcher.query('SELECT c', []);
expect(getConnection).toHaveBeenCalledTimes(2);
expect(vi.getTimerCount()).toBe(0);
// A later lone query waits its own full maxTimeInQueue.
await vi.advanceTimersByTimeAsync(10);
const d = batcher.query('SELECT d', []);
await vi.advanceTimersByTimeAsync(29);
expect(getConnection).toHaveBeenCalledTimes(2);
await vi.advanceTimersByTimeAsync(1);
expect(getConnection).toHaveBeenCalledTimes(3);
expect(sent()).toEqual([
'SELECT a; SELECT 1',
'SELECT b;SELECT c; SELECT 1',
'SELECT d; SELECT 1',
]);
releaseAll();
await Promise.all([a, b, c, d]);
} finally {
vi.useRealTimers();
}
});
it('cancels the pending next-turn flush when a batch fills up', async () => {
fakeFlushTimers();
try {
const conn = makeConnection(happyBatch);
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxBatchSize: 2,
readOnly: true,
maxInFlightImmediate: 8,
acquireTimeoutMs: 0,
});
const a = batcher.query('SELECT a', []);
expect(vi.getTimerCount()).toBe(1);
const b = batcher.query('SELECT b', []);
expect(vi.getTimerCount()).toBe(0);
const c = batcher.query('SELECT c', []);
expect(vi.getTimerCount()).toBe(1);
await vi.runAllTimersAsync();
await Promise.all([a, b, c]);
expect(vi.getTimerCount()).toBe(0);
expect(getConnection).toHaveBeenCalledTimes(2);
} finally {
vi.useRealTimers();
}
});
it('leaves no timer armed after a failed flush, a drop or a breaker rejection', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
fakeFlushTimers();
try {
const getConnection = vi.fn(async () => {
throw makeError('ECONNREFUSED');
});
const batcher = new SQLBatcher(
{ promise: () => ({ getConnection }) },
{
readOnly: true,
maxQueueSize: 1,
failureThreshold: 1,
cooldownMs: 60_000,
maxInFlightImmediate: 8,
},
);
const dropped = expect(
batcher.query('SELECT a', []),
).rejects.toMatchObject({ reason: 'queueOverflow' });
const failed = expect(
batcher.query('SELECT b', []),
).rejects.toMatchObject({ reason: 'connAcquire' });
expect(vi.getTimerCount()).toBe(1);
await vi.runAllTimersAsync();
await dropped;
await failed;
expect(vi.getTimerCount()).toBe(0);
await expect(batcher.query('SELECT c', [])).rejects.toMatchObject({
reason: 'breakerOpen',
});
expect(vi.getTimerCount()).toBe(0);
expect(batcher.queue).toHaveLength(0);
} finally {
vi.useRealTimers();
warnSpy.mockRestore();
}
});
});
describe('SQLBatcher', () => {
it('resolves each batched item with its own result', async () => {
const conn = makeConnection(happyBatch);
@@ -436,47 +86,8 @@ describe('SQLBatcher', () => {
expect(conn.release).toHaveBeenCalledTimes(1);
});
it('skips the transaction wrapper on a readOnly batcher', async () => {
const conn = makeConnection(happyBatch);
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxTimeInQueue: 5,
readOnly: true,
});
const [a, b] = await Promise.all([
batcher.query('SELECT a', []),
batcher.query('SELECT b', []),
]);
expect(a[0]).toEqual([{ n: 0 }]);
expect(b[0]).toEqual([{ n: 1 }]);
// BEGIN and COMMIT are round trips; a SELECT-only batch has nothing to
// roll back, so it must not pay for them.
expect(conn.beginTransaction).not.toHaveBeenCalled();
expect(conn.commit).not.toHaveBeenCalled();
expect(conn.release).toHaveBeenCalledTimes(1);
});
it('does not roll back a failed readOnly batch', async () => {
const conn = makeConnection((sql: string) => {
if (isBatchQuery(sql)) throw makeError('ER_LOCK_DEADLOCK');
return [[{ n: 0 }], undefined];
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxTimeInQueue: 5,
readOnly: true,
});
await Promise.all([
batcher.query('SELECT a', []),
batcher.query('SELECT b', []),
]);
expect(conn.rollback).not.toHaveBeenCalled();
});
it('drops the oldest item with reason queueOverflow at the high-water mark', async () => {
const conn = makeConnection(() => [[{ n: 0 }], undefined]);
const conn = makeConnection(happyBatch);
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxTimeInQueue: 5,
@@ -627,65 +238,6 @@ describe('SQLBatcher', () => {
);
});
it('sends a lone write as-is, outside a transaction', async () => {
const conn = makeConnection((sql) => {
if (isBatchQuery(sql)) throw new Error('unexpected batch query');
return [{ affectedRows: 1 }, undefined];
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, { maxBatchSize: 1 });
const result = await batcher.query('UPDATE t SET x = ?', [1]);
expect(result[0]).toEqual({ affectedRows: 1 });
expect(conn.query).toHaveBeenCalledTimes(1);
expect(conn.query).toHaveBeenCalledWith('UPDATE t SET x = ?', [1]);
expect(conn.beginTransaction).not.toHaveBeenCalled();
expect(conn.commit).not.toHaveBeenCalled();
expect(conn.release).toHaveBeenCalledTimes(1);
});
it('runs a lone write the server rejects once, without tripping the breaker', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
const conn = makeConnection((sql) => {
if (sql === 'INSERT dup') throw makeError('ER_DUP_ENTRY');
return [{ affectedRows: 1 }, undefined];
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxBatchSize: 1,
failureThreshold: 1,
cooldownMs: 60_000,
});
await expect(batcher.query('INSERT dup', [])).rejects.toMatchObject({
code: 'ER_DUP_ENTRY',
});
expect(conn.query).toHaveBeenCalledTimes(1);
expect(conn.rollback).not.toHaveBeenCalled();
expect(warnSpy).not.toHaveBeenCalled();
// The server answered, so it is up: the next write goes through.
await expect(batcher.query('INSERT fine', [])).resolves.toBeTruthy();
warnSpy.mockRestore();
});
it('retries a lone write on a connection the server closed for inactivity', async () => {
let attempts = 0;
const conn = makeConnection(() => {
attempts++;
if (attempts === 1) {
throw makeError('ER_CLIENT_INTERACTION_TIMEOUT');
}
return [{ affectedRows: 1 }, undefined];
});
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, { maxBatchSize: 1 });
const result = await batcher.query('INSERT x', []);
expect(result[0]).toEqual({ affectedRows: 1 });
expect(attempts).toBe(2);
expect(getConnection).toHaveBeenCalledTimes(2);
});
it('never retries deterministic row-level errors and does not escalate the breaker', async () => {
let fallbackAttempts = 0;
const conn = makeConnection((sql) => {
@@ -18,7 +18,7 @@
*/
import Database from 'better-sqlite3';
import { existsSync, mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { existsSync, mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
@@ -27,15 +27,7 @@ import { DatabaseClientFactory } from './index.js';
import { SqliteDatabaseClient } from './SqliteDatabaseClient.js';
/** Highest schema version the migration table can reach. */
const CURRENT_SCHEMA_VERSION = 101;
/**
* These suites migrate real files on disk. Idle they finish in well under a
* second, but the whole migration chain runs against the filesystem — enough
* that a loaded runner blows the 5s default and reports a timeout where there
* is no fault.
*/
const DISK_MIGRATION_TIMEOUT_MS = 30_000;
const CURRENT_SCHEMA_VERSION = 61;
const SYSTEM_USER_UUID = '5d4adce0-a381-4982-9c02-6e2540026238';
const sqliteConfig = (
@@ -60,7 +52,7 @@ const userVersionOf = async (client: SqliteDatabaseClient): Promise<number> => {
return row.user_version as number;
};
describe('SqliteDatabaseClient — boot and migrations', { timeout: DISK_MIGRATION_TIMEOUT_MS }, () => {
describe('SqliteDatabaseClient — boot and migrations', () => {
let client: SqliteDatabaseClient;
beforeEach(async () => {
@@ -81,312 +73,6 @@ describe('SqliteDatabaseClient — boot and migrations', { timeout: DISK_MIGRATI
expect(await userVersionOf(client)).toBe(CURRENT_SCHEMA_VERSION);
});
it('stamps a version whose migration did not run (known off-by-one)', async () => {
// Existing behaviour, not an endorsement: the stamp overshoots the
// applied work by one entry, so 70 is reported without 0074 running.
const partial = await bootClient({ targetVersion: 70 });
try {
expect(await userVersionOf(partial)).toBe(70);
await expect(
partial.read(
"SELECT 1 FROM pragma_table_info('group') WHERE name = 'handle'",
),
).resolves.toEqual([]);
} finally {
partial.onServerShutdown();
}
});
it('applies the team columns and indexes from 0076', async () => {
const columnsOf = async (table: string) =>
(
(await client.read(
`SELECT name FROM pragma_table_info('${table}')`,
)) as { name: string }[]
).map((r) => r.name);
expect(await columnsOf('group')).toEqual(
expect.arrayContaining([
'kind',
'name',
'handle',
'deleted_at',
]),
);
expect(await columnsOf('jct_user_group')).toContain('org_owned');
expect(await columnsOf('user')).toContain('requires_password_change');
const indexes = (await client.read(
"SELECT name FROM sqlite_master WHERE type = 'index' AND name IN (?, ?, ?)",
['idx_group_handle', 'idx_group_owner', 'idx_jct_user_group_group'],
)) as { name: string }[];
expect(indexes.map((r) => r.name).sort()).toEqual([
'idx_group_handle',
'idx_group_owner',
'idx_jct_user_group_group',
]);
});
it('applies the event_subscriptions indexes from 0081', async () => {
const indexes = (await client.read(
"SELECT name FROM sqlite_master WHERE type = 'index' AND name IN (?, ?, ?)",
[
'idx_event_subscriptions_app_handler',
'idx_event_subscriptions_expires',
'idx_event_subscriptions_suspended',
],
)) as { name: string }[];
expect(indexes.map((r) => r.name).sort()).toEqual([
'idx_event_subscriptions_app_handler',
'idx_event_subscriptions_expires',
'idx_event_subscriptions_suspended',
]);
});
it('keeps `handle` unique but lets the seeded groups share a NULL one', async () => {
const seeded = (await client.read(
'SELECT COUNT(*) AS n FROM `group` WHERE `handle` IS NULL',
)) as { n: number }[];
expect(seeded[0].n).toBeGreaterThan(1);
await client.write(
'UPDATE `group` SET `handle` = ? WHERE `id` = (SELECT MIN(`id`) FROM `group`)',
['taken'],
);
await expect(
client.write(
'UPDATE `group` SET `handle` = ? WHERE `id` = (SELECT MAX(`id`) FROM `group`)',
['taken'],
),
).rejects.toThrow(/UNIQUE/iu);
});
it('treats `handle` case-insensitively, as usernames are treated', async () => {
await client.write(
'UPDATE `group` SET `handle` = ? WHERE `id` = (SELECT MIN(`id`) FROM `group`)',
['design-team'],
);
// Without NOCASE this differs per engine: mysql rejects, the others accept.
await expect(
client.write(
'UPDATE `group` SET `handle` = ? WHERE `id` = (SELECT MAX(`id`) FROM `group`)',
['Design-Team'],
),
).rejects.toThrow(/UNIQUE/iu);
});
it('matches a handle case-insensitively on lookup, not just on insert', async () => {
// Index-only NOCASE would leave `WHERE handle = ?` case-sensitive here while
// mysql's utf8mb4_unicode_ci column matched -- one query, two behaviours.
await client.write(
'UPDATE `group` SET `handle` = ? WHERE `id` = (SELECT MIN(`id`) FROM `group`)',
['design-team'],
);
await expect(
client.read('SELECT `handle` FROM `group` WHERE `handle` = ?', [
'Design-Team',
]),
).resolves.toEqual([{ handle: 'design-team' }]);
});
it('rejects a duplicate membership pair after 0078', async () => {
const [{ user_id, group_id }] = (await client.read(
'SELECT (SELECT MIN(`id`) FROM `user`) AS user_id, ' +
'(SELECT MIN(`id`) FROM `group`) AS group_id',
)) as { user_id: number; group_id: number }[];
const insert = () =>
client.write(
'INSERT INTO `jct_user_group` (`user_id`, `group_id`) VALUES (?, ?)',
[user_id, group_id],
);
await insert();
// `GroupStore.addUsers` has no conflict clause, so before 0075 this
// second call silently doubled every group permission the user reads.
await expect(insert()).rejects.toThrow(/UNIQUE/iu);
});
it('applies the audit table and group-share columns from 0079', async () => {
const columns = (await client.read(
"SELECT name FROM pragma_table_info('audit_team_membership')",
)) as { name: string }[];
expect(columns.map((r) => r.name)).toEqual([
'id',
'group_id',
'group_id_keep',
'user_id',
'user_id_keep',
'actor_user_id',
'action',
'reason',
'created_at',
]);
const shareColumns = (await client.read(
"SELECT name FROM pragma_table_info('share')",
)) as { name: string }[];
expect(shareColumns.map((r) => r.name)).toContain('holder_group_id');
const indexes = (await client.read(
"SELECT name FROM sqlite_master WHERE type = 'index' " +
'AND (name LIKE ? OR name LIKE ?)',
['idx_audit_team_membership%', 'idx_share_holder_group%'],
)) as { name: string }[];
expect(indexes.map((r) => r.name).sort()).toEqual([
// The `_fk` three stop a user or group delete scanning the audit table.
'idx_audit_team_membership_actor_fk',
'idx_audit_team_membership_group',
'idx_audit_team_membership_group_fk',
'idx_audit_team_membership_user',
'idx_audit_team_membership_user_fk',
'idx_share_holder_group',
'idx_share_holder_group_entry_issuer',
]);
});
it('keeps an audit row after its group is deleted, blanking only the FK', async () => {
// 0043 leaves `PRAGMA foreign_keys = ON`, so this is real behaviour.
const [group] = (await client.read(
'SELECT MIN(`id`) AS id FROM `group`',
)) as { id: number }[];
const [user] = (await client.read(
'SELECT MIN(`id`) AS id FROM `user`',
)) as { id: number }[];
await client.write(
'INSERT INTO `audit_team_membership` ' +
'(`group_id`, `group_id_keep`, `user_id`, `user_id_keep`, ' +
'`actor_user_id`, `action`) VALUES (?, ?, ?, ?, ?, ?)',
[group.id, group.id, user.id, user.id, user.id, 'reset_member_password'],
);
await client.write('DELETE FROM `group` WHERE `id` = ?', [group.id]);
await expect(
client.read(
'SELECT `group_id`, `group_id_keep`, `action` FROM `audit_team_membership`',
),
).resolves.toEqual([
{
group_id: null,
group_id_keep: group.id,
action: 'reset_member_password',
},
]);
});
it('deletes the subdomain rows an app owns when the app is deleted', async () => {
const [user] = (await client.read(
'SELECT MIN(`id`) AS id FROM `user`',
)) as { id: number }[];
await client.write(
'INSERT INTO `apps` (`uid`, `owner_user_id`, `name`, `title`, `index_url`) ' +
'VALUES (?, ?, ?, ?, ?)',
[
'app-cascade-test',
user.id,
'cascade-test',
'cascade-test',
'https://cascade.test/',
],
);
const [app] = (await client.read(
'SELECT `id` FROM `apps` WHERE `uid` = ?',
['app-cascade-test'],
)) as { id: number }[];
const insertSubdomain = (
uuid: string,
name: string,
appOwner: number | null,
) =>
client.write(
'INSERT INTO `subdomains` (`uuid`, `subdomain`, `user_id`, `app_owner`) ' +
'VALUES (?, ?, ?, ?)',
[uuid, name, user.id, appOwner],
);
await insertSubdomain('sd-cascade-owned', 'cascade-owned', app.id);
await insertSubdomain('sd-cascade-unowned', 'cascade-unowned', null);
await client.write('DELETE FROM `apps` WHERE `id` = ?', [app.id]);
await expect(
client.read(
'SELECT `subdomain` FROM `subdomains` WHERE `subdomain` LIKE ? ORDER BY `id`',
['cascade-%'],
),
).resolves.toEqual([{ subdomain: 'cascade-unowned' }]);
});
it('keeps every subdomains column through the 0086 rebuild', async () => {
const columns = (await client.read(
"SELECT `name` FROM pragma_table_info('subdomains') ORDER BY `cid`",
)) as { name: string }[];
expect(columns.map((c) => c.name)).toEqual([
'id',
'uuid',
'subdomain',
'user_id',
'root_dir_id',
'associated_app_id',
'ts',
'app_owner',
'protected',
'domain',
'database_id',
'preamble_version',
]);
});
it('constrains team shares that the user-holder index cannot', async () => {
const [user] = (await client.read(
'SELECT MIN(`id`) AS id FROM `user`',
)) as { id: number }[];
const groups = (await client.read(
'SELECT `id` FROM `group` ORDER BY `id` LIMIT 2',
)) as { id: number }[];
await client.write(
'INSERT INTO `fsentries` (`uuid`, `name`, `user_id`, `modified`) ' +
'VALUES (?, ?, ?, ?)',
['11111111-1111-4111-8111-111111111111', 'shared.txt', user.id, 0],
);
const [entry] = (await client.read(
'SELECT `id` FROM `fsentries` WHERE `uuid` = ?',
['11111111-1111-4111-8111-111111111111'],
)) as { id: number }[];
const insertShare = (uid: string, groupId: number) =>
client.write(
'INSERT INTO `share` ' +
'(`uid`, `issuer_user_id`, `recipient_email`, `holder_user_id`, ' +
'`holder_group_id`, `fsentry_id`) VALUES (?, ?, ?, NULL, ?, ?)',
[uid, user.id, '[email protected]', groupId, entry.id],
);
await insertShare('share-team-a', groups[0].id);
// Different group, same file and issuer -- allowed.
await insertShare('share-team-b', groups[1].id);
// Both have `holder_user_id` NULL, so the user-holder index permitted them.
await expect(insertShare('share-team-c', groups[0].id)).rejects.toThrow(
/UNIQUE/iu,
);
});
it('matches a handle case-insensitively on lookup, not just on insert', async () => {
// Index-only NOCASE would leave `WHERE handle = ?` case-sensitive here while
// mysql's utf8mb4_unicode_ci column matched -- one query, two behaviours.
await client.write(
'UPDATE `group` SET `handle` = ? WHERE `id` = (SELECT MIN(`id`) FROM `group`)',
['design-team'],
);
await expect(
client.read('SELECT `handle` FROM `group` WHERE `handle` = ?', [
'Design-Team',
]),
).resolves.toEqual([{ handle: 'design-team' }]);
});
it('runs the javascript migrations, not just the .sql ones', async () => {
// The `system` user only exists because 0025 (a .dbmig.js file) ran
// inside the migration VM.
@@ -397,471 +83,6 @@ describe('SqliteDatabaseClient — boot and migrations', { timeout: DISK_MIGRATI
expect(rows).toEqual([{ username: 'system' }]);
});
// 0070 sweeps three by-hand groups no code reads. Both tables it can reach
// cascade on delete, so the guard is the point: a group that still carries
// permissions or members has to survive, or the sweep silently revokes
// them. Re-running the statement is safe — it is a plain conditional DELETE.
describe('0070 drop-orphaned-default-groups', () => {
const MIGRATION = readFileSync(
new URL(
'./migrations/sqlite/0070_drop-orphaned-default-groups.sql',
import.meta.url,
),
'utf8',
);
const seedGroup = async (name: string): Promise<number> => {
const [system] = await client.read(
'SELECT `id` FROM `user` WHERE `uuid` = ?',
[SYSTEM_USER_UUID],
);
const uid = `grp-${name}-${Math.random().toString(36).slice(2, 10)}`;
await client.write(
'INSERT INTO `group` (`uid`, `owner_user_id`, `extra`, `metadata`) ' +
'VALUES (?, ?, ?, ?)',
[uid, system.id, JSON.stringify({ name }), '{}'],
);
const [row] = await client.read(
'SELECT `id` FROM `group` WHERE `uid` = ?',
[uid],
);
return Number(row.id);
};
const stillThere = async (id: number): Promise<boolean> => {
const rows = await client.read(
'SELECT `id` FROM `group` WHERE `id` = ?',
[id],
);
return rows.length > 0;
};
it('drops an orphaned freeai/experimental/dangerous group', async () => {
const ids = await Promise.all(
['freeai', 'experimental', 'dangerous'].map(seedGroup),
);
await client.write(MIGRATION);
for (const id of ids) expect(await stillThere(id)).toBe(false);
});
it('spares one that still carries a permission row', async () => {
const id = await seedGroup('freeai');
const [system] = await client.read(
'SELECT `id` FROM `user` WHERE `uuid` = ?',
[SYSTEM_USER_UUID],
);
await client.write(
'INSERT INTO `user_to_group_permissions` ' +
'(`user_id`, `group_id`, `permission`, `extra`) VALUES (?, ?, ?, ?)',
[system.id, id, 'service:some-paid-thing', '{}'],
);
await client.write(MIGRATION);
expect(await stillThere(id)).toBe(true);
// And the grant it carries is still intact, not cascaded away.
const perms = await client.read(
'SELECT `permission` FROM `user_to_group_permissions` WHERE `group_id` = ?',
[id],
);
expect(perms).toEqual([{ permission: 'service:some-paid-thing' }]);
});
it('spares one that still has members', async () => {
const id = await seedGroup('experimental');
const [system] = await client.read(
'SELECT `id` FROM `user` WHERE `uuid` = ?',
[SYSTEM_USER_UUID],
);
await client.write(
'INSERT INTO `jct_user_group` (`user_id`, `group_id`) VALUES (?, ?)',
[system.id, id],
);
await client.write(MIGRATION);
expect(await stillThere(id)).toBe(true);
});
it('leaves the groups the platform actually depends on alone', async () => {
// system, admin, user and temp are named by config or code.
await client.write(MIGRATION);
const rows = await client.read(
"SELECT json_extract(`extra`, '$.name') AS name FROM `group`",
);
const names = rows.map((r) => String(r.name));
for (const name of ['system', 'admin', 'user']) {
expect(names).toContain(name);
}
});
});
// 0072 rebuilds `notification` for the scope tuple, and picks up the
// user_id index and the delete cascade mysql and postgres already had.
describe('0072 notification-scope', () => {
// Only the backfill re-runs: the rebuild half is once-only by
// construction, but mysql and postgres replay their whole file on
// every boot, so these statements have to be safe twice.
const BACKFILL = readFileSync(
new URL(
'./migrations/sqlite/0072_notification-scope.sql',
import.meta.url,
),
'utf8',
)
.split(/;\s*\n/)
.map((s) => s.trim())
.filter((s) => /(^|\n)UPDATE /.test(s));
const seedUser = async (): Promise<number> => {
const name = `notif-${Math.random().toString(36).slice(2, 10)}`;
await client.write(
'INSERT INTO `user` (`username`, `uuid`) VALUES (?, ?)',
[name, `${name}-uuid`],
);
const [row] = await client.read(
'SELECT `id` FROM `user` WHERE `username` = ?',
[name],
);
return Number(row.id);
};
const seedLegacy = async (
userId: number,
value: Record<string, unknown> | string,
): Promise<string> => {
const uid = `n-${Math.random().toString(36).slice(2, 12)}`;
await client.write(
'INSERT INTO `notification` (`uid`, `user_id`, `value`) VALUES (?, ?, ?)',
[
uid,
userId,
typeof value === 'string' ? value : JSON.stringify(value),
],
);
return uid;
};
const scopeOf = async (uid: string) => {
const [row] = await client.read(
'SELECT `type`, `audience`, `app_uid` FROM `notification` WHERE `uid` = ?',
[uid],
);
return row;
};
const runBackfill = async () => {
for (const stmt of BACKFILL) await client.write(stmt);
};
it('carries every backfill statement', () => {
expect(BACKFILL).toHaveLength(3);
});
it('classifies the markers legacy rows actually carried', async () => {
const userId = await seedUser();
const received = await seedLegacy(userId, {
source: 'sharing',
template: 'file-shared-with-you',
});
const claimed = await seedLegacy(userId, {
source: 'sharing',
template: 'file-shared-before-you-joined',
});
const deployed = await seedLegacy(userId, {
source: 'worker',
title: 'Successfully deployed https://x.puter.work',
template: 'user-requesting-share',
});
const failed = await seedLegacy(userId, {
source: 'worker',
title: 'Failed to deploy x! boom',
template: 'user-requesting-share',
});
await runBackfill();
expect(await scopeOf(received)).toEqual({
type: 'share.received',
audience: 'account',
app_uid: null,
});
expect(await scopeOf(claimed)).toEqual({
type: 'share.claimed',
audience: 'account',
app_uid: null,
});
// The app is unrecoverable — the payload only ever named a worker.
expect(await scopeOf(deployed)).toEqual({
type: 'app.worker.deployed',
audience: 'developer',
app_uid: null,
});
expect(await scopeOf(failed)).toEqual({
type: 'app.worker.deployFailed',
audience: 'developer',
app_uid: null,
});
});
it('leaves anything it does not recognise reading as legacy', async () => {
const userId = await seedUser();
const unknown = await seedLegacy(userId, {
source: 'sharing',
template: 'something-else',
});
const bare = await seedLegacy(userId, { title: 'hi' });
const notJson = await seedLegacy(userId, 'plain text');
await runBackfill();
for (const uid of [unknown, bare, notJson]) {
expect(await scopeOf(uid)).toEqual({
type: '',
audience: 'account',
app_uid: null,
});
}
});
it('does not reclassify on replay', async () => {
const userId = await seedUser();
const uid = await seedLegacy(userId, {
source: 'sharing',
template: 'file-shared-with-you',
});
await runBackfill();
// A row already classified as something else must survive a
// second pass untouched.
await client.write(
'UPDATE `notification` SET `audience` = ?, `app_uid` = ? WHERE `uid` = ?',
['app-user', 'app-1234', uid],
);
await runBackfill();
expect(await scopeOf(uid)).toEqual({
type: 'share.received',
audience: 'app-user',
app_uid: 'app-1234',
});
});
it('indexes user_id and the scope tuple', async () => {
const rows = await client.read('PRAGMA index_list(`notification`)');
const names = rows.map((r) => String(r.name));
expect(names).toContain('idx_notification_user_id');
expect(names).toContain('idx_notification_scope');
});
it('retires a deleted user rows', async () => {
const [{ foreign_keys: enforcing }] = await client.read(
'PRAGMA foreign_keys',
);
expect(enforcing).toBe(1);
const userId = await seedUser();
const uid = await seedLegacy(userId, { title: 'orphan-to-be' });
await client.write('DELETE FROM `user` WHERE `id` = ?', [userId]);
expect(
await client.read(
'SELECT `uid` FROM `notification` WHERE `uid` = ?',
[uid],
),
).toEqual([]);
});
});
it('adds kv_share_handles.notify from 0099, off for every existing share', async () => {
const dir = mkdtempSync(join(tmpdir(), 'puter-sqlite-notify-'));
const path = join(dir, 'puter.sqlite');
try {
const before = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path, targetVersion: 94 }),
);
await before.onServerStart();
await before.write(
'INSERT INTO `kv_share_handles` (`handle`, `owner_user_id`, ' +
'`grantee_user_id`, `app_uid`, `key_prefix`, `permission`, ' +
'`created_at`) VALUES (?, 1, 1, ?, ?, ?, 0)',
['kvh-before-notify', 'os-global', 'a:', 'kv-share:x'],
);
before.onServerShutdown();
const after = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path }),
);
await after.onServerStart();
await expect(
after.read(
'SELECT `notify` FROM `kv_share_handles` WHERE `handle` = ?',
['kvh-before-notify'],
),
).resolves.toEqual([{ notify: 0 }]);
after.onServerShutdown();
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it('fills a share row missing its entry owner from 0103, invites left alone', async () => {
const dir = mkdtempSync(join(tmpdir(), 'puter-sqlite-share-owner-'));
const path = join(dir, 'puter.sqlite');
try {
const before = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path }),
);
await before.onServerStart();
const [user] = (await before.read(
'SELECT MIN(`id`) AS id FROM `user`',
)) as { id: number }[];
await before.write(
'INSERT INTO `fsentries` (`uuid`, `name`, `user_id`, `modified`) ' +
'VALUES (?, ?, ?, ?)',
['22222222-2222-4222-8222-222222222222', 'owned.txt', user.id, 0],
);
const [entry] = (await before.read(
'SELECT `id` FROM `fsentries` WHERE `uuid` = ?',
['22222222-2222-4222-8222-222222222222'],
)) as { id: number }[];
await before.write(
'INSERT INTO `share` (`uid`, `issuer_user_id`, ' +
'`recipient_email`, `fsentry_id`) VALUES (?, ?, ?, ?)',
['share-no-owner', user.id, '[email protected]', entry.id],
);
await before.write(
'INSERT INTO `share` (`uid`, `issuer_user_id`, ' +
'`recipient_email`) VALUES (?, ?, ?)',
['share-invite', user.id, '[email protected]'],
);
// The full schema as it stood before 0103, so 0103 reruns; 0104
// is undone so it can apply again after it.
await before.write('DROP INDEX idx_user_signup_tls_fingerprint');
await before.write(
'ALTER TABLE `user` DROP COLUMN `signup_tls_fingerprint`',
);
await before.write('PRAGMA user_version = 98');
before.onServerShutdown();
const after = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path }),
);
await after.onServerStart();
await expect(
after.read(
'SELECT `uid`, `entry_owner_user_id` FROM `share` ' +
'WHERE `uid` IN (?, ?) ORDER BY `uid`',
['share-invite', 'share-no-owner'],
),
).resolves.toEqual([
{ uid: 'share-invite', entry_owner_user_id: null },
{ uid: 'share-no-owner', entry_owner_user_id: user.id },
]);
after.onServerShutdown();
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it('indexes notification.created_at for the retention sweep', async () => {
const rows = await client.read('PRAGMA index_list(`notification`)');
expect(rows.map((r) => String(r.name))).toContain(
'idx_notification_created_at',
);
});
it('indexes notification.replaced_at per mailbox slice for catch-up', async () => {
const columns = await client.read(
'PRAGMA index_info(`idx_notification_replaced`)',
);
expect(columns.map((c) => String(c.name))).toEqual([
'user_id',
'audience',
'app_uid',
'replaced_at',
]);
});
it('keeps notification.send_key unique', async () => {
const rows = await client.read('PRAGMA index_list(`notification`)');
expect(
rows.find((r) => r.name === 'idx_notification_send_key'),
).toMatchObject({ unique: 1 });
const columns = await client.read(
'PRAGMA index_info(`idx_notification_send_key`)',
);
expect(columns.map((c) => String(c.name))).toEqual(['send_key']);
});
it('creates event_user_handlers with its unique and holder indexes', async () => {
const columns = await client.read(
'PRAGMA table_info(`event_user_handlers`)',
);
expect(columns.map((c) => String(c.name))).toEqual([
'id',
'app_uid',
'holder_user_id',
'name',
'source',
'source_hash',
'created_at',
'updated_at',
'scope_since',
]);
const indexes = await client.read(
'PRAGMA index_list(`event_user_handlers`)',
);
expect(
indexes.find(
(i) => i.name === 'idx_event_user_handlers_app_holder_name',
),
).toMatchObject({ unique: 1 });
expect(
indexes.find((i) => i.name === 'idx_event_user_handlers_holder_app'),
).toMatchObject({ unique: 0 });
const columnsOf = async (index: string) =>
(await client.read(`PRAGMA index_info(\`${index}\`)`)).map((c) =>
String(c.name),
);
expect(await columnsOf('idx_event_user_handlers_app_holder_name')).toEqual(
['app_uid', 'holder_user_id', 'name'],
);
expect(await columnsOf('idx_event_user_handlers_holder_app')).toEqual([
'holder_user_id',
'app_uid',
]);
});
it('leaves event_handlers as it was', async () => {
const columns = await client.read('PRAGMA table_info(`event_handlers`)');
expect(columns.map((c) => String(c.name))).toEqual([
'id',
'app_uid',
'name',
'source',
'source_hash',
'created_at',
'updated_at',
]);
});
it('adds event_subscriptions.handler_scope, app scope for existing rows', async () => {
const columns = await client.read(
'PRAGMA table_info(`event_subscriptions`)',
);
expect(columns.find((c) => c.name === 'handler_scope')).toMatchObject({
notnull: 1,
dflt_value: '0',
});
const indexes = await client.read(
'PRAGMA index_list(`event_subscriptions`)',
);
expect(indexes.map((i) => String(i.name))).toContain(
'idx_event_subscriptions_holder',
);
});
it('leaves an already-migrated database untouched on a second boot', async () => {
const dir = mkdtempSync(join(tmpdir(), 'puter-sqlite-'));
const path = join(dir, 'nested', 'puter.sqlite');
@@ -895,66 +116,6 @@ describe('SqliteDatabaseClient — boot and migrations', { timeout: DISK_MIGRATI
}
});
it('deduplicates pre-existing membership rows when 0078 applies', async () => {
const dir = mkdtempSync(join(tmpdir(), 'puter-sqlite-dedup-'));
const path = join(dir, 'puter.sqlite');
try {
// Stop at 67 so the duplicates exist the way a live database's do:
// written before the unique index, not after it.
const before = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path, targetVersion: 67 }),
);
await before.onServerStart();
const [{ user_id, group_id }] = (await before.read(
'SELECT (SELECT MIN(`id`) FROM `user`) AS user_id, ' +
'(SELECT MIN(`id`) FROM `group`) AS group_id',
)) as { user_id: number; group_id: number }[];
for (const pair of [
[user_id, group_id],
[user_id, group_id],
[user_id, group_id],
[user_id, group_id + 1],
]) {
await before.write(
'INSERT INTO `jct_user_group` (`user_id`, `group_id`) VALUES (?, ?)',
pair,
);
}
const seeded = (await before.read(
'SELECT `id`, `group_id` FROM `jct_user_group` ORDER BY `id`',
)) as { id: number; group_id: number }[];
expect(seeded).toHaveLength(4);
before.onServerShutdown();
const after = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path }),
);
await after.onServerStart();
expect(await userVersionOf(after)).toBe(CURRENT_SCHEMA_VERSION);
// Three copies collapse to the lowest id; the distinct pair is left
// alone. Losing the higher ids costs nothing -- `addUsers` writes
// only the two id columns, so `extra` and `metadata` are NULL.
await expect(
after.read(
'SELECT `id`, `group_id` FROM `jct_user_group` ORDER BY `id`',
),
).resolves.toEqual([seeded[0], seeded[3]]);
await expect(
after.write(
'INSERT INTO `jct_user_group` (`user_id`, `group_id`) VALUES (?, ?)',
[user_id, group_id],
),
).rejects.toThrow(/UNIQUE/iu);
after.onServerShutdown();
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it('stops early at a configured target version', async () => {
const partial = await bootClient({ targetVersion: 5 });
try {
@@ -976,10 +137,7 @@ describe('SqliteDatabaseClient — boot and migrations', { timeout: DISK_MIGRATI
});
});
describe(
'SqliteDatabaseClient — legacy version inference',
{ timeout: DISK_MIGRATION_TIMEOUT_MS },
() => {
describe('SqliteDatabaseClient — legacy version inference', () => {
let dir: string;
beforeEach(() => {
@@ -1112,7 +270,6 @@ describe('SqliteDatabaseClient — query interface', () => {
affectedRows: 1,
anyRowsAffected: true,
});
expect(inserted).not.toHaveProperty('warningStatus');
const missed = await client.write(
'UPDATE `widget` SET `name` = ? WHERE `id` = ?',
@@ -1221,7 +378,8 @@ describe('SqliteDatabaseClient — query interface', () => {
});
// Single-node engines have no replica to race, so `tryHardRead` must not
// issue the query twice on the one connection.
// issue the query twice — the base-class default fires `pread()` and
// `read()` in parallel, which on sqlite is the same connection.
describe('SqliteDatabaseClient — tryHardRead', () => {
let client: SqliteDatabaseClient;
@@ -1260,50 +418,20 @@ describe('SqliteDatabaseClient — tryHardRead', () => {
expect(readSpy).toHaveBeenCalledTimes(1);
expect(preadSpy).not.toHaveBeenCalled();
});
});
describe('SqliteDatabaseClient — prepared statements', () => {
let client: SqliteDatabaseClient;
let prepare: ReturnType<typeof vi.spyOn>;
beforeEach(async () => {
client = await bootClient();
await client.write(
'CREATE TABLE `widget` (`id` INTEGER PRIMARY KEY, `name` TEXT)',
);
// better-sqlite3 is the driver boundary; count compilations there.
const db = (client as unknown as { db: Database.Database }).db;
prepare = vi.spyOn(db, 'prepare');
it('throws from requireRead when nothing matches', async () => {
await expect(
client.requireRead(
'SELECT `name` FROM `widget` WHERE `id` = ?',
[42],
),
).rejects.toThrow('required read returned no rows');
});
afterEach(() => {
client.onServerShutdown();
});
it('compiles a repeated query once', async () => {
const insert = 'INSERT INTO `widget` (`name`) VALUES (?)';
await client.write(insert, ['a']);
await client.write(insert, ['b']);
await client.batchWrite([{ statement: insert, values: ['c'] }]);
const select = 'SELECT `name` FROM `widget` ORDER BY `id`';
await client.read(select);
await expect(client.read(select)).resolves.toEqual([
{ name: 'a' },
{ name: 'b' },
{ name: 'c' },
]);
expect(prepare).toHaveBeenCalledTimes(2);
});
it('keeps serving a cached query after the schema changes', async () => {
const select = 'SELECT * FROM `widget`';
await client.write('INSERT INTO `widget` (`name`) VALUES (?)', ['a']);
await client.read(select);
await client.write('ALTER TABLE `widget` ADD COLUMN `size` INTEGER');
await expect(client.read(select)).resolves.toEqual([
{ id: 1, name: 'a', size: null },
]);
it('returns the rows from requireRead when something matches', async () => {
await client.write('INSERT INTO `widget` (`name`) VALUES (?)', ['ok']);
await expect(
client.requireRead('SELECT `name` FROM `widget`'),
).resolves.toEqual([{ name: 'ok' }]);
});
});
@@ -21,15 +21,10 @@ import { existsSync, mkdirSync, readFileSync } from 'fs';
import { basename, dirname, extname, join, resolve } from 'path';
import { createContext, runInContext } from 'vm';
import type { IConfig } from '../../types';
import { BoundedTtlMap } from '../../util/boundedTtlMap.js';
import { Span } from '../../util/span.js';
import { AbstractDatabaseClient, type WriteResult } from './DatabaseClient';
const MIGRATIONS_DIR = resolve(__dirname, './migrations/sqlite');
const STATEMENT_CACHE_SIZE = 1000;
type SqliteDatabase = InstanceType<typeof import('better-sqlite3')>;
type SqliteStatement = import('better-sqlite3').Statement<unknown[]>;
/**
* Ordered list of [threshold_version, files[]] pairs. A database whose
@@ -100,57 +95,13 @@ const AVAILABLE_MIGRATIONS: [number, string[]][] = [
[58, ['0063_add_suspended_reason.sql']],
[59, ['0064_abuse-moderation-events.sql']],
[60, ['0065_app-feedback.sql']],
[61, ['0066_owned-email-unique.sql']],
[62, ['0067_share_entries.sql']],
[63, ['0068_referral-code-unique.sql']],
[64, ['0069_user-block.sql']],
[65, ['0070_drop-orphaned-default-groups.sql']],
[66, ['0071_share_issuer_index.sql']],
[67, ['0072_notification-scope.sql']],
[68, ['0073_notification-created-at.sql']],
[69, ['0074_add_user_home.sql']],
[70, ['0075_event-subscriptions.sql']],
[71, ['0076_event-handlers.sql']],
[72, ['0077_teams.sql']],
[73, ['0078_jct-user-group-pair-unique.sql']],
[74, ['0079_team-audit-and-group-shares.sql']],
[75, ['0080_kv-share-handles.sql']],
[76, ['0081_event-subscriptions-indexes.sql']],
[77, ['0082_temp-password-expiry.sql']],
[78, ['0083_team-directory.sql']],
[79, ['0084_share-anyone-with-link.sql']],
[80, ['0085_event-subscriptions-include-value.sql']],
[81, ['0086_subdomains-app-owner-cascade.sql']],
[82, ['0087_feedback-attachments.sql']],
[83, ['0088_apps-index-url.sql']],
[84, ['0089_team-require-2fa.sql']],
[85, ['0090_user-app-permission-index.sql']],
[86, ['0091_icloud-dot-significance.sql']],
[87, ['0092_share-entry-owner.sql']],
[88, ['0093_fsentries-parent-uid-index.sql']],
[89, ['0094_deleted-apps.sql']],
[90, ['0095_kv-share-handle-levels.sql']],
[91, ['0096_kv-share-handle-quota-indexes.sql']],
[92, ['0097_user-to-user-holder-permission-index.sql']],
[93, ['0098_notification-groups.sql']],
[94, ['0099_kv-share-handle-notify.sql']],
[95, ['0100_notification-send-key.sql']],
[96, ['0101_user-scoped-handlers.sql']],
[97, ['0102_notification-scope-id-index.sql']],
[98, ['0103_share-entry-owner-backfill.sql']],
[99, ['0104_signup-tls-fingerprint.sql']],
[100, ['0105_sessions_kind_debug_app.sql']],
];
export class SqliteDatabaseClient extends AbstractDatabaseClient {
override readonly engineName = 'sqlite';
// better-sqlite3 instance — set during onServerStart
private db!: SqliteDatabase;
/** Prepared statements by query text; SQLite re-prepares on schema change. */
private readonly statements = new BoundedTtlMap<string, SqliteStatement>({
maxEntries: STATEMENT_CACHE_SIZE,
});
private db!: InstanceType<typeof import('better-sqlite3')>;
constructor(config: IConfig) {
super(config);
@@ -172,14 +123,12 @@ export class SqliteDatabaseClient extends AbstractDatabaseClient {
mkdirSync(dirname(dbPath), { recursive: true });
}
this.statements.clear();
this.db = new Database(dbPath);
await this.runMigrations(isNew);
}
override onServerShutdown(): void {
this.statements.clear();
if (this.db) {
this.db.close();
}
@@ -194,9 +143,12 @@ export class SqliteDatabaseClient extends AbstractDatabaseClient {
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
return this.prepare(query).all(
...this.transformParams(params),
) as Record<string, unknown>[];
query = this.transformQuery(query);
params = this.transformParams(params);
return this.db.prepare(query).all(...params) as Record<
string,
unknown
>[];
}
override async pread(
@@ -207,20 +159,30 @@ export class SqliteDatabaseClient extends AbstractDatabaseClient {
return this.read(query, params);
}
override async tryHardRead(
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
// No replica to race, so the base class's parallel
// primary-plus-replica read would run the same statement twice on
// the one connection.
return this.read(query, params);
}
@Span('db.write', (query: string) => ({ 'db.statement': query }))
override async write(
query: string,
params: unknown[] = [],
): Promise<WriteResult> {
const info = this.prepare(query).run(...this.transformParams(params));
query = this.transformQuery(query);
params = this.transformParams(params);
const info = this.db.prepare(query).run(...params);
return {
insertId: info.lastInsertRowid,
affectedRows: info.changes,
anyRowsAffected: info.changes > 0,
// No warningStatus: SQLite raises none, but a new row can reuse a
// deleted row's rowid that other tables still name, so a clean
// insert doesn't prove the row is all the caller sent.
};
}
@@ -231,8 +193,10 @@ export class SqliteDatabaseClient extends AbstractDatabaseClient {
entries: { statement: string; values: unknown[] }[],
): Promise<void> {
this.db.transaction(() => {
for (const { statement, values } of entries) {
this.prepare(statement).run(...this.transformParams(values));
for (let { statement, values } of entries) {
statement = this.transformQuery(statement);
values = this.transformParams(values);
this.db.prepare(statement).run(...values);
}
})();
}
@@ -241,15 +205,6 @@ export class SqliteDatabaseClient extends AbstractDatabaseClient {
// SQLite-specific transforms
// ------------------------------------------------------------------
private prepare(query: string): SqliteStatement {
let statement = this.statements.get(query);
if (!statement) {
statement = this.db.prepare(this.transformQuery(query));
this.statements.set(query, statement);
}
return statement;
}
private transformQuery(query: string): string {
return query.replace(/now\(\)/gi, "datetime('now')");
}
@@ -1060,7 +1060,6 @@ CREATE TABLE IF NOT EXISTS `user` (
`signup_server` varchar(255) DEFAULT NULL,
`metadata` json DEFAULT (json_object()),
`reputation` smallint DEFAULT '100',
`home` varchar(64) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `uid` (`uuid`),
UNIQUE KEY `username` (`username`),
@@ -1152,7 +1151,6 @@ CALL _puter_add_col('user', 'signup_user_agent', '`signup_user_agent` varchar(51
CALL _puter_add_col('user', 'signup_origin', '`signup_origin` varchar(255) DEFAULT NULL');
CALL _puter_add_col('user', 'signup_server', '`signup_server` varchar(255) DEFAULT NULL');
CALL _puter_add_col('user', 'metadata', '`metadata` json DEFAULT (json_object())');
CALL _puter_add_col('user', 'home', '`home` varchar(64) DEFAULT NULL');
CALL _puter_add_col('user', 'reputation', '`reputation` smallint DEFAULT ''100''');
--
@@ -1,128 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Enforce "at most one account owns an email address". Mirrors SQLite
-- migration 0066.
--
-- `user.email` is deliberately not UNIQUE: several rows may legitimately hold
-- the same address while unconfirmed (admin-provisioned placeholders, signups
-- that were never confirmed, temp accounts on their way to becoming real). What
-- must never happen is two rows both *owning* an address — owning meaning the
-- row is confirmed, or holds a password and so can drive password recovery for
-- that inbox.
--
-- Signup, save-account, change-email, OIDC and admin provisioning each check for
-- an owner before writing, but a check and an insert are not one operation: two
-- requests can both read "free" and both write. This index is what actually
-- holds the invariant; the application checks just produce a nicer error most of
-- the time.
--
-- SQLite expresses that with a partial index. MySQL has none, so the predicate
-- lives in a generated column that evaluates to NULL for every row that does not
-- own its address — and NULLs do not collide in an InnoDB unique index, which is
-- exactly the "unlimited unconfirmed placeholders" behaviour we need.
--
-- The column is VIRTUAL, not STORED, on purpose: adding a stored generated
-- column rebuilds the table, while a virtual one is a metadata-only change and
-- the index that follows builds INPLACE. On a `user` table of any size with
-- read replicas attached, that is the difference between a routine change and an
-- outage. The ALGORITHM/LOCK clauses are spelled out so a server that cannot
-- honour them refuses the statement instead of quietly copying the table.
--
-- Matching is on the canonical address so provider aliases
-- (`[email protected]` vs `[email protected]`) collide. `clean_email` is
-- written on every modern write path; the COALESCE covers rows old enough to
-- predate the column. Run the `clean_email` backfill before this migration or
-- alias collisions among those rows go unnoticed.
--
-- Idempotent: both steps are guarded on INFORMATION_SCHEMA so the directory
-- replays safely.
--
-- If the index creation fails with ER_DUP_ENTRY, the DB already contains
-- duplicate owners. Collapse them first (admin → One-off Jobs → Collapse
-- Duplicate Emails) — there is no safe automatic merge of two accounts.
DROP PROCEDURE IF EXISTS _puter_add_owned_email;
DELIMITER //
CREATE PROCEDURE _puter_add_owned_email()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user'
AND COLUMN_NAME = 'owned_email'
) THEN
ALTER TABLE `user`
ADD COLUMN `owned_email` VARCHAR(256)
CHARACTER SET latin1 COLLATE latin1_swedish_ci
GENERATED ALWAYS AS (
CASE
WHEN `email` IS NOT NULL
AND (`email_confirmed` = 1 OR `password` IS NOT NULL)
THEN COALESCE(`clean_email`, LOWER(`email`))
ELSE NULL
END
) VIRTUAL,
ALGORITHM=INSTANT;
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user'
AND INDEX_NAME = 'idx_user_owned_email'
) THEN
ALTER TABLE `user`
ADD UNIQUE KEY `idx_user_owned_email` (`owned_email`),
ALGORITHM=INPLACE, LOCK=NONE;
END IF;
END//
DELIMITER ;
CALL _puter_add_owned_email();
DROP PROCEDURE IF EXISTS _puter_add_owned_email;
-- One Puter account per external identity. OIDCStore.link already assumes this
-- constraint exists — it catches the unique violation to tell "re-linking the
-- same account" apart from "this sub belongs to someone else" — but the table
-- never actually had it, so two concurrent first-time logins could each create
-- an account and each link the same sub. Subsequent logins then resolved to
-- whichever row came back first.
--
-- Dedupe `user_oidc_providers` before applying this: keep the lowest `id` per
-- (provider, provider_sub) and point it at the account the collapse job kept.
DROP PROCEDURE IF EXISTS _puter_add_oidc_sub_unique;
DELIMITER //
CREATE PROCEDURE _puter_add_oidc_sub_unique()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user_oidc_providers'
AND INDEX_NAME = 'idx_user_oidc_provider_sub'
) THEN
ALTER TABLE `user_oidc_providers`
ADD UNIQUE KEY `idx_user_oidc_provider_sub` (`provider`, `provider_sub`);
END IF;
END//
DELIMITER ;
CALL _puter_add_oidc_sub_unique();
DROP PROCEDURE IF EXISTS _puter_add_oidc_sub_unique;
@@ -1,106 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Grow `share` from a pending-email-invite table into the index of active
-- shares. See sqlite/0067_share_entries.sql for the column rationale.
--
-- Idempotent: columns go through _puter_add_col (from mig_1), indexes and
-- foreign keys through the guarded procedure below. There is no per-file
-- applied-state tracking, so every statement must tolerate a re-run.
CALL _puter_add_col('share', 'holder_user_id', '`holder_user_id` int unsigned DEFAULT NULL');
CALL _puter_add_col('share', 'fsentry_id', '`fsentry_id` int unsigned DEFAULT NULL');
CALL _puter_add_col('share', 'mode', '`mode` varchar(20) DEFAULT NULL');
CALL _puter_add_col('share', 'applied_at', '`applied_at` timestamp NULL DEFAULT NULL');
DROP PROCEDURE IF EXISTS _puter_add_share_index_constraints;
DELIMITER //
CREATE PROCEDURE _puter_add_share_index_constraints()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_holder'
) THEN
ALTER TABLE `share` ADD INDEX `idx_share_holder` (`holder_user_id`, `id`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_fsentry'
) THEN
ALTER TABLE `share` ADD INDEX `idx_share_fsentry` (`fsentry_id`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_holder_entry_issuer'
) THEN
ALTER TABLE `share` ADD UNIQUE INDEX `idx_share_holder_entry_issuer`
(`holder_user_id`, `fsentry_id`, `issuer_user_id`);
END IF;
-- Retiring a deleted node's grants looks them up by permission text
-- (`fs:<uuid>` and `fs:<uuid>:%`); the subject lives in that text rather
-- than a column, so no foreign key can cascade it. The primary key is
-- (issuer, holder, permission), which never puts `permission` first, so
-- without this both the equality and the left-anchored LIKE degrade to a
-- full scan on the delete path. The column is ascii varchar(255), well
-- inside InnoDB's key limit, so it is indexed whole.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user_to_user_permissions'
AND INDEX_NAME = 'idx_user_to_user_permissions_permission'
) THEN
ALTER TABLE `user_to_user_permissions`
ADD INDEX `idx_user_to_user_permissions_permission` (`permission`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND CONSTRAINT_NAME = 'share_holder_user_fk'
) THEN
ALTER TABLE `share` ADD CONSTRAINT `share_holder_user_fk`
FOREIGN KEY (`holder_user_id`) REFERENCES `user` (`id`)
ON DELETE CASCADE ON UPDATE CASCADE;
END IF;
-- The cascade that retires a share with its file.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND CONSTRAINT_NAME = 'share_fsentry_fk'
) THEN
ALTER TABLE `share` ADD CONSTRAINT `share_fsentry_fk`
FOREIGN KEY (`fsentry_id`) REFERENCES `fsentries` (`id`)
ON DELETE CASCADE ON UPDATE CASCADE;
END IF;
END//
DELIMITER ;
CALL _puter_add_share_index_constraints();
DROP PROCEDURE IF EXISTS _puter_add_share_index_constraints;
@@ -1,35 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- One user refusing contact from another. See sqlite/0068_user-block.sql for
-- the rationale. `created_at` is unix seconds.
--
-- Idempotent: `CREATE TABLE IF NOT EXISTS` with the indexes declared inline,
-- so the directory can replay safely.
CREATE TABLE IF NOT EXISTS `user_block` (
`id` INT NOT NULL AUTO_INCREMENT,
`blocker_user_id` INT UNSIGNED NOT NULL,
`blocked_user_id` INT UNSIGNED NOT NULL,
`created_at` BIGINT NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_user_block_pair` (`blocker_user_id`, `blocked_user_id`),
CONSTRAINT `fk_user_block_blocker` FOREIGN KEY (`blocker_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_user_block_blocked` FOREIGN KEY (`blocked_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
@@ -1,61 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Widen `notification.shown` and `notification.acknowledged` to hold the unix
-- second the store has been writing to them since the backend rework.
--
-- Both arrived from the v1 schema as `tinyint(1)`, where they were flags. The
-- rework changed the writes to timestamps and sqlite (`INTEGER`) and postgres
-- (`bigint`) took them; only mysql was left too narrow, so every `markShown`
-- and `markAcknowledged` there fails with ER_WARN_DATA_OUT_OF_RANGE and the
-- column stays NULL. Dismissing a notification therefore never sticks, and one
-- already delivered is re-sent on every reconnect.
--
-- No backfill: every reader tests `IS NULL` / `IS NOT NULL` only, so a legacy
-- `1` keeps meaning "yes" once widened.
--
-- Guarded on the current type rather than run unconditionally. Changing a
-- column type copies the table, and this one grows without bound — a migration
-- directory that replays on every boot must not rebuild it every time.
DROP PROCEDURE IF EXISTS _puter_widen_notification_stamps;
DELIMITER //
CREATE PROCEDURE _puter_widen_notification_stamps()
BEGIN
IF EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'notification'
AND COLUMN_NAME = 'shown'
AND DATA_TYPE = 'tinyint'
) THEN
ALTER TABLE `notification` MODIFY `shown` BIGINT DEFAULT NULL;
END IF;
IF EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'notification'
AND COLUMN_NAME = 'acknowledged'
AND DATA_TYPE = 'tinyint'
) THEN
ALTER TABLE `notification` MODIFY `acknowledged` BIGINT DEFAULT NULL;
END IF;
END //
DELIMITER ;
CALL _puter_widen_notification_stamps();
DROP PROCEDURE IF EXISTS _puter_widen_notification_stamps;
@@ -1,22 +0,0 @@
-- Drops three groups that no code has ever read: `freeai`, `experimental` and
-- `dangerous`. No migration creates them — they were added by hand back when
-- hardcoded permissions were keyed by group name. That map is now a flat
-- per-user floor (`default_user_permissions` in data/hardcoded-permissions.js),
-- so a group nothing looks up grants nothing, and the rows are dead weight.
--
-- Guarded, and deliberately so: `user_to_group_permissions.group_id` and
-- `jct_user_group.group_id` both cascade on delete, so removing a group that
-- still carries permissions or members would silently revoke them from every
-- member. Only a group with neither is dropped. A group that survives this
-- migration has dependents and needs a deliberate decision, not a sweep --
-- query `user_to_group_permissions` by `group_id` to see what it holds.
--
-- The joins rather than `NOT IN` subqueries: MySQL will not read from the table
-- it is deleting from inside a subquery on the same statement.
DELETE g FROM `group` g
LEFT JOIN `user_to_group_permissions` p ON p.`group_id` = g.`id`
LEFT JOIN `jct_user_group` j ON j.`group_id` = g.`id`
WHERE JSON_UNQUOTE(JSON_EXTRACT(g.`extra`, '$.name'))
IN ('freeai', 'experimental', 'dangerous')
AND p.`group_id` IS NULL
AND j.`group_id` IS NULL;
@@ -1,80 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Give `notification` a scope tuple: what the row is (`type`), the app it is
-- about (`app_uid`, NULL for platform), and who the recipient is in relation
-- to it (`audience`). Attribution lived inside the `value` JSON as two
-- free-form markers, `source` and `template`, that could disagree; these
-- columns replace them and are what a scoped list can be indexed on.
--
-- `app_uid` is char(40) to match `apps`.`uid`, which is `app-` plus a uuid.
--
-- Idempotent: columns go through `_puter_add_col` (mig_1), the index through
-- the guarded procedure. There is no per-file applied-state tracking, so every
-- statement here tolerates a re-run.
CALL _puter_add_col('notification', 'app_uid', '`app_uid` char(40) COLLATE utf8mb4_unicode_ci DEFAULT NULL');
CALL _puter_add_col('notification', 'audience', '`audience` varchar(16) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT ''account''');
CALL _puter_add_col('notification', 'type', '`type` varchar(64) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT ''''');
DROP PROCEDURE IF EXISTS _puter_add_notification_scope_index;
DELIMITER //
CREATE PROCEDURE _puter_add_notification_scope_index()
BEGIN
-- Serves both the scoped list and the unacknowledged count.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'notification'
AND INDEX_NAME = 'idx_notification_scope'
) THEN
ALTER TABLE `notification` ADD INDEX `idx_notification_scope`
(`user_id`, `audience`, `app_uid`, `acknowledged`);
END IF;
END //
DELIMITER ;
CALL _puter_add_notification_scope_index();
DROP PROCEDURE IF EXISTS _puter_add_notification_scope_index;
-- Backfill from the markers the payload actually carried. `type = ''` is the
-- not-yet-classified state, so re-running matches nothing the first pass
-- claimed, and a row matching no marker keeps the defaults (`''` / 'account')
-- and reads as legacy.
UPDATE `notification`
SET `type` = 'share.received', `audience` = 'account', `app_uid` = NULL
WHERE `type` = ''
AND JSON_UNQUOTE(JSON_EXTRACT(`value`, '$.template')) = 'file-shared-with-you';
UPDATE `notification`
SET `type` = 'share.claimed', `audience` = 'account', `app_uid` = NULL
WHERE `type` = ''
AND JSON_UNQUOTE(JSON_EXTRACT(`value`, '$.template')) = 'file-shared-before-you-joined';
-- Worker rows carried only `source: 'worker'` and a title, so the app they
-- belong to is unrecoverable and they stay unattributed. The title prefix is
-- the sole surviving signal of which way the deploy went.
UPDATE `notification`
SET `audience` = 'developer',
`app_uid` = NULL,
`type` = CASE
WHEN JSON_UNQUOTE(JSON_EXTRACT(`value`, '$.title')) LIKE 'Successfully deployed %'
THEN 'app.worker.deployed'
ELSE 'app.worker.deployFailed'
END
WHERE `type` = ''
AND JSON_UNQUOTE(JSON_EXTRACT(`value`, '$.source')) = 'worker';
@@ -1,42 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Index `notification`.`created_at`. The retention sweep selects by age alone
-- and by nothing else, so no existing index on the table narrows it — every
-- pass would otherwise scan the whole table to find the oldest few hundred
-- rows.
--
-- Idempotent: the guarded procedure, as mig_26. There is no per-file
-- applied-state tracking, so a replay has to be a no-op.
DROP PROCEDURE IF EXISTS _puter_add_notification_created_at_index;
DELIMITER //
CREATE PROCEDURE _puter_add_notification_created_at_index()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'notification'
AND INDEX_NAME = 'idx_notification_created_at'
) THEN
ALTER TABLE `notification` ADD INDEX `idx_notification_created_at`
(`created_at`);
END IF;
END //
DELIMITER ;
CALL _puter_add_notification_created_at_index();
DROP PROCEDURE IF EXISTS _puter_add_notification_created_at_index;
@@ -1,25 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Home region column. Mirrors SQLite migration 0074. Home region this
-- account's data belongs in — a node id, not a storage region. NULL for
-- accounts that predate the column, which fall back to `signup_server`.
--
-- Idempotent: the column add uses _puter_add_col (defined in mig_1, which
-- leaves it resident for later migrations).
CALL _puter_add_col('user', 'home', '`home` VARCHAR(64) DEFAULT NULL');
@@ -1,59 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Subscriptions that outlive the connection that made them. See
-- sqlite/0075_event-subscriptions.sql for the column rationale.
--
-- Idempotent: `CREATE TABLE IF NOT EXISTS` with the indexes declared inline,
-- as mig_23. There is no per-file applied-state tracking, so a replay has to
-- be a no-op.
CREATE TABLE IF NOT EXISTS `event_subscriptions` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`sub_id` varchar(80) NOT NULL,
`token` varchar(255) NOT NULL,
`owner_user_id` int unsigned NOT NULL,
`holder_user_id` int unsigned NOT NULL,
-- Matches `apps`.`uid` exactly, charset included, so an equality against
-- one never falls back to a conversion. No foreign key: a row outlives the
-- app that made it, which is what keeps it revocable afterwards.
`app_uid` char(40) CHARACTER SET latin1 COLLATE latin1_swedish_ci
DEFAULT NULL,
`subject` varchar(4096) NOT NULL,
`anchor_uid` char(36) NOT NULL,
`anchor_path` varchar(4096) NOT NULL,
`match` varchar(1024) DEFAULT NULL,
`delivery` varchar(16) NOT NULL,
`ops` varchar(64) DEFAULT NULL,
`handler_name` varchar(128) DEFAULT NULL,
`targets` json NOT NULL,
`context` text,
`permission` varchar(32) NOT NULL,
`suspended_at` bigint DEFAULT NULL,
`suspended_reason` varchar(64) DEFAULT NULL,
`expires_at` bigint DEFAULT NULL,
`created_at` bigint NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_event_subscriptions_sub_id` (`sub_id`),
KEY `idx_event_subscriptions_token` (`token`),
KEY `idx_event_subscriptions_holder` (`holder_user_id`, `app_uid`),
KEY `idx_event_subscriptions_owner` (`owner_user_id`),
CONSTRAINT `fk_event_subscriptions_owner` FOREIGN KEY (`owner_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_event_subscriptions_holder` FOREIGN KEY (`holder_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
@@ -1,43 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- `event_subscriptions`.`anchor_uid` was sized for a filesystem node uuid.
-- A key-value subscription anchors on an app instead, and `apps`.`uid` is
-- `char(40)`, so the column has to hold one.
--
-- Idempotent: the guarded procedure, as mig_27. There is no per-file
-- applied-state tracking, so a replay has to be a no-op.
DROP PROCEDURE IF EXISTS _puter_widen_event_subscriptions_anchor_uid;
DELIMITER //
CREATE PROCEDURE _puter_widen_event_subscriptions_anchor_uid()
BEGIN
IF EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'event_subscriptions'
AND COLUMN_NAME = 'anchor_uid'
AND CHARACTER_MAXIMUM_LENGTH < 40
) THEN
ALTER TABLE `event_subscriptions`
MODIFY COLUMN `anchor_uid` varchar(40) NOT NULL;
END IF;
END //
DELIMITER ;
CALL _puter_widen_event_subscriptions_anchor_uid();
DROP PROCEDURE IF EXISTS _puter_widen_event_subscriptions_anchor_uid;
@@ -1,37 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Named handlers an app deploys once. See sqlite/0076_event-handlers.sql for
-- the column rationale.
--
-- Idempotent: `CREATE TABLE IF NOT EXISTS` with the indexes declared inline,
-- as mig_29. There is no per-file applied-state tracking, so a replay has to
-- be a no-op.
CREATE TABLE IF NOT EXISTS `event_handlers` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
-- Matches `apps`.`uid` exactly, charset included, so an equality against
-- one never falls back to a conversion.
`app_uid` char(40) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL,
`name` varchar(128) NOT NULL,
`source` mediumtext NOT NULL,
`source_hash` char(64) NOT NULL,
`created_at` bigint NOT NULL,
`updated_at` bigint NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_event_handlers_app_name` (`app_uid`, `name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
@@ -1,58 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Teams. See sqlite/0077_teams.sql for the column rationale.
-- No per-file applied-state tracking, so every statement must tolerate a re-run:
-- columns go through _puter_add_col, indexes through the guarded procedure below.
CALL _puter_add_col('group', 'kind', '`kind` varchar(20) DEFAULT NULL');
CALL _puter_add_col('group', 'name', '`name` varchar(255) DEFAULT NULL');
CALL _puter_add_col('group', 'handle', '`handle` varchar(64) DEFAULT NULL');
CALL _puter_add_col('group', 'deleted_at', '`deleted_at` timestamp NULL DEFAULT NULL');
CALL _puter_add_col('jct_user_group', 'org_owned', '`org_owned` tinyint(1) DEFAULT NULL');
CALL _puter_add_col('user', 'requires_password_change', '`requires_password_change` tinyint(1) NOT NULL DEFAULT ''0''');
DROP PROCEDURE IF EXISTS _puter_add_team_indexes;
DELIMITER //
CREATE PROCEDURE _puter_add_team_indexes()
BEGIN
-- Already case-insensitive: the table is utf8mb4_unicode_ci. NULLs stay distinct.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'group'
AND INDEX_NAME = 'idx_group_handle'
) THEN
ALTER TABLE `group` ADD UNIQUE INDEX `idx_group_handle` (`handle`);
END IF;
-- Composite, unlike the existing single-column `group_id` key.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'jct_user_group'
AND INDEX_NAME = 'idx_jct_user_group_group'
) THEN
ALTER TABLE `jct_user_group`
ADD INDEX `idx_jct_user_group_group` (`group_id`, `user_id`);
END IF;
END//
DELIMITER ;
CALL _puter_add_team_indexes();
DROP PROCEDURE IF EXISTS _puter_add_team_indexes;
@@ -1,47 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Deduplicate then constrain. See sqlite/0075 for why this is lossless.
-- Both steps sit behind the index check: mysql re-runs this file on every boot, and
-- guarding them together stops a rolling deploy deleting and indexing concurrently.
DROP PROCEDURE IF EXISTS _puter_add_membership_pair_index;
DELIMITER //
CREATE PROCEDURE _puter_add_membership_pair_index()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'jct_user_group'
AND INDEX_NAME = 'idx_jct_user_group_pair'
) THEN
-- Self-join because mysql refuses `NOT IN (SELECT ... FROM jct_user_group)` (1093).
DELETE dup FROM `jct_user_group` dup
JOIN `jct_user_group` keep
ON dup.`user_id` = keep.`user_id`
AND dup.`group_id` = keep.`group_id`
AND dup.`id` > keep.`id`;
ALTER TABLE `jct_user_group`
ADD UNIQUE INDEX `idx_jct_user_group_pair` (`user_id`, `group_id`);
END IF;
END//
DELIMITER ;
CALL _puter_add_membership_pair_index();
DROP PROCEDURE IF EXISTS _puter_add_membership_pair_index;
@@ -1,84 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0076. Types follow mysql_mig_1; the guarded procedure follows mysql_mig_22.
CREATE TABLE IF NOT EXISTS `audit_team_membership` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`group_id` int unsigned DEFAULT NULL,
`group_id_keep` int unsigned NOT NULL,
`user_id` int unsigned DEFAULT NULL,
`user_id_keep` int unsigned NOT NULL,
`actor_user_id` int unsigned DEFAULT NULL,
`action` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`reason` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_audit_team_membership_group` (`group_id_keep`, `id`),
KEY `idx_audit_team_membership_user` (`user_id_keep`, `id`),
KEY `idx_audit_team_membership_group_fk` (`group_id`),
KEY `idx_audit_team_membership_user_fk` (`user_id`),
KEY `idx_audit_team_membership_actor_fk` (`actor_user_id`),
-- SET NULL, never CASCADE: deleting an account must not erase what was done to it.
CONSTRAINT `fk_audit_team_membership_group` FOREIGN KEY (`group_id`)
REFERENCES `group` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
CONSTRAINT `fk_audit_team_membership_user` FOREIGN KEY (`user_id`)
REFERENCES `user` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
CONSTRAINT `fk_audit_team_membership_actor` FOREIGN KEY (`actor_user_id`)
REFERENCES `user` (`id`) ON DELETE SET NULL ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
CALL _puter_add_col('share', 'holder_group_id', '`holder_group_id` int unsigned DEFAULT NULL');
DROP PROCEDURE IF EXISTS _puter_add_group_share_constraints;
DELIMITER //
CREATE PROCEDURE _puter_add_group_share_constraints()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_holder_group'
) THEN
ALTER TABLE `share` ADD INDEX `idx_share_holder_group` (`holder_group_id`, `id`);
END IF;
-- Team shares leave `holder_user_id` NULL, so the existing unique index binds none.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_holder_group_entry_issuer'
) THEN
ALTER TABLE `share` ADD UNIQUE INDEX `idx_share_holder_group_entry_issuer`
(`holder_group_id`, `fsentry_id`, `issuer_user_id`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND CONSTRAINT_NAME = 'share_holder_group_fk'
) THEN
ALTER TABLE `share` ADD CONSTRAINT `share_holder_group_fk`
FOREIGN KEY (`holder_group_id`) REFERENCES `group` (`id`)
ON DELETE CASCADE ON UPDATE CASCADE;
END IF;
END//
DELIMITER ;
CALL _puter_add_group_share_constraints();
DROP PROCEDURE IF EXISTS _puter_add_group_share_constraints;
@@ -1,68 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Opaque names for shared regions of a user's key-value namespace. See
-- sqlite/0080_kv-share-handles.sql for the column rationale.
--
-- Idempotent: `CREATE TABLE IF NOT EXISTS` with the indexes declared inline,
-- as mig_29. There is no per-file applied-state tracking, so a replay has to
-- be a no-op.
CREATE TABLE IF NOT EXISTS `kv_share_handles` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`handle` varchar(64) NOT NULL,
`owner_user_id` int unsigned NOT NULL,
`grantee_user_id` int unsigned NOT NULL,
-- Matches `apps`.`uid` exactly, charset included, so an equality against
-- one never falls back to a conversion. No foreign key: a handle outlives
-- the app it was minted against, which is what keeps it revocable.
`app_uid` char(40) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL,
`key_prefix` varchar(1024) NOT NULL,
`permission` varchar(1024) NOT NULL,
`created_at` bigint NOT NULL,
`revoked_at` bigint DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_kv_share_handles_handle` (`handle`),
KEY `idx_kv_share_handles_owner` (`owner_user_id`, `id`),
CONSTRAINT `fk_kv_share_handles_owner` FOREIGN KEY (`owner_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_kv_share_handles_grantee` FOREIGN KEY (`grantee_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- A subscription made through a handle stores the grant string it was
-- authorized under rather than an access mode, and a grant string carries a
-- user uuid, an app uid and a key prefix. Guarded on the current length, as
-- mig_24: replaying a column change on a growing table rebuilds it every boot.
DROP PROCEDURE IF EXISTS _puter_widen_subscription_permission;
DELIMITER //
CREATE PROCEDURE _puter_widen_subscription_permission()
BEGIN
IF EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'event_subscriptions'
AND COLUMN_NAME = 'permission'
AND CHARACTER_MAXIMUM_LENGTH < 1024
) THEN
ALTER TABLE `event_subscriptions` MODIFY `permission` varchar(1024) NOT NULL;
END IF;
END //
DELIMITER ;
CALL _puter_widen_subscription_permission();
DROP PROCEDURE IF EXISTS _puter_widen_subscription_permission;
@@ -1,60 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- `event_subscriptions` grew three hot queries with no index behind them:
-- a handler's own rows, the expiry reaper and the suspended reaper. See
-- sqlite/0081_event-subscriptions-indexes.sql for the query shapes.
--
-- Idempotent: the guarded procedure, as mig_30/mig_34/mig_35. There is no
-- per-file applied-state tracking, so a replay has to be a no-op.
DROP PROCEDURE IF EXISTS _puter_add_event_subscriptions_indexes;
DELIMITER //
CREATE PROCEDURE _puter_add_event_subscriptions_indexes()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'event_subscriptions'
AND INDEX_NAME = 'idx_event_subscriptions_app_handler'
) THEN
ALTER TABLE `event_subscriptions`
ADD INDEX `idx_event_subscriptions_app_handler` (`app_uid`, `handler_name`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'event_subscriptions'
AND INDEX_NAME = 'idx_event_subscriptions_expires'
) THEN
ALTER TABLE `event_subscriptions`
ADD INDEX `idx_event_subscriptions_expires` (`expires_at`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'event_subscriptions'
AND INDEX_NAME = 'idx_event_subscriptions_suspended'
) THEN
ALTER TABLE `event_subscriptions`
ADD INDEX `idx_event_subscriptions_suspended` (`suspended_at`, `id`);
END IF;
END//
DELIMITER ;
CALL _puter_add_event_subscriptions_indexes();
DROP PROCEDURE IF EXISTS _puter_add_event_subscriptions_indexes;
@@ -1,21 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0082_temp-password-expiry.sql for the column rationale.
-- No per-file applied-state tracking, so the column goes through _puter_add_col.
CALL _puter_add_col('user', 'temp_password_expires_at', '`temp_password_expires_at` bigint DEFAULT NULL');
@@ -1,21 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0083_team-directory.sql for the column rationale.
-- No per-file applied-state tracking, so the column goes through _puter_add_col.
CALL _puter_add_col('group', 'directory_enabled', '`directory_enabled` tinyint(1) NOT NULL DEFAULT 0');
@@ -1,42 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0084_share-anyone-with-link.sql for the column rationale.
-- No per-file applied-state tracking, so the column goes through _puter_add_col
-- and the index through a guarded procedure, as mysql_mig_34 does.
CALL _puter_add_col('share', 'anyone', '`anyone` tinyint(1) DEFAULT NULL');
DROP PROCEDURE IF EXISTS _puter_add_share_anyone_index;
DELIMITER //
CREATE PROCEDURE _puter_add_share_anyone_index()
BEGIN
-- Every other row leaves `anyone` NULL, so this binds only link shares.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_anyone'
) THEN
ALTER TABLE `share` ADD UNIQUE INDEX `idx_share_anyone` (`fsentry_id`, `anyone`);
END IF;
END//
DELIMITER ;
CALL _puter_add_share_anyone_index();
DROP PROCEDURE IF EXISTS _puter_add_share_anyone_index;
@@ -1,22 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0085_event-subscriptions-include-value.sql for the column
-- rationale. No per-file applied-state tracking, so it goes through
-- _puter_add_col as mysql_mig_39 does.
CALL _puter_add_col('event_subscriptions', 'include_value', '`include_value` tinyint(1) NOT NULL DEFAULT 0');
@@ -1,75 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0086: `subdomains.app_owner` cascades on app
-- delete instead of nulling. The existing constraint is found by column
-- rather than by name, since an older database need not carry the name
-- mysql_mig_1 declares. Guarded procedure as mysql_mig_34: a replay sees
-- DELETE_RULE already CASCADE and does nothing.
DROP PROCEDURE IF EXISTS _puter_subdomains_app_owner_cascade;
DELIMITER //
CREATE PROCEDURE _puter_subdomains_app_owner_cascade()
BEGIN
DECLARE fk_name VARCHAR(64) DEFAULT NULL;
-- A subquery, not SELECT ... INTO: no matching row yields NULL quietly.
SET fk_name = (
SELECT rc.CONSTRAINT_NAME
FROM INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS rc
JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE kcu
ON kcu.CONSTRAINT_SCHEMA = rc.CONSTRAINT_SCHEMA
AND kcu.CONSTRAINT_NAME = rc.CONSTRAINT_NAME
AND kcu.TABLE_NAME = rc.TABLE_NAME
WHERE rc.CONSTRAINT_SCHEMA = DATABASE()
AND rc.TABLE_NAME = 'subdomains'
AND rc.REFERENCED_TABLE_NAME = 'apps'
AND kcu.COLUMN_NAME = 'app_owner'
AND rc.DELETE_RULE <> 'CASCADE'
LIMIT 1
);
IF fk_name IS NOT NULL THEN
SET @s := CONCAT('ALTER TABLE `subdomains` DROP FOREIGN KEY `', fk_name, '`');
PREPARE stmt FROM @s;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END IF;
IF NOT EXISTS (
SELECT 1
FROM INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS rc
JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE kcu
ON kcu.CONSTRAINT_SCHEMA = rc.CONSTRAINT_SCHEMA
AND kcu.CONSTRAINT_NAME = rc.CONSTRAINT_NAME
AND kcu.TABLE_NAME = rc.TABLE_NAME
WHERE rc.CONSTRAINT_SCHEMA = DATABASE()
AND rc.TABLE_NAME = 'subdomains'
AND rc.REFERENCED_TABLE_NAME = 'apps'
AND kcu.COLUMN_NAME = 'app_owner'
AND rc.DELETE_RULE = 'CASCADE'
) THEN
ALTER TABLE `subdomains` ADD CONSTRAINT `fk_subdomains_app_owner`
FOREIGN KEY (`app_owner`) REFERENCES `apps` (`id`)
ON DELETE CASCADE ON UPDATE CASCADE;
END IF;
END//
DELIMITER ;
CALL _puter_subdomains_app_owner_cascade();
DROP PROCEDURE IF EXISTS _puter_subdomains_app_owner_cascade;
@@ -1,39 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0087: Contact Us attachment metadata, a JSON array
-- of `{name, type, size}` (NULL when none). Guarded by a throwaway procedure,
-- as in mysql_mig_21.
DROP PROCEDURE IF EXISTS _puter_add_feedback_attachments;
DELIMITER //
CREATE PROCEDURE _puter_add_feedback_attachments()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'feedback'
AND COLUMN_NAME = 'attachments'
) THEN
ALTER TABLE `feedback` ADD COLUMN `attachments` text DEFAULT NULL;
END IF;
END//
DELIMITER ;
CALL _puter_add_feedback_attachments();
DROP PROCEDURE IF EXISTS _puter_add_feedback_attachments;
@@ -1,44 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0088: index `apps.index_url` for origin lookups.
-- TEXT needs a prefix; 191 characters keeps the key under 767 bytes in
-- utf8mb4 whatever the table's row format, and URLs diverge well before it.
--
-- Idempotent: the guarded procedure, as mig_36. The 1061 handler covers two
-- nodes booting at once, where the loser finds the index already built.
DROP PROCEDURE IF EXISTS _puter_add_apps_index_url_index;
DELIMITER //
CREATE PROCEDURE _puter_add_apps_index_url_index()
BEGIN
DECLARE CONTINUE HANDLER FOR 1061 BEGIN END;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'apps'
AND INDEX_NAME = 'idx_apps_index_url'
) THEN
ALTER TABLE `apps`
ADD INDEX `idx_apps_index_url` (`index_url`(191)),
ALGORITHM=INPLACE, LOCK=NONE;
END IF;
END//
DELIMITER ;
CALL _puter_add_apps_index_url_index();
DROP PROCEDURE IF EXISTS _puter_add_apps_index_url_index;
@@ -1,21 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0089: a team may require 2FA of the accounts it
-- provisioned. Added through `_puter_add_col` as mysql_mig_32 does, so a
-- replay finds the column and does nothing.
CALL _puter_add_col('group', 'require_2fa', '`require_2fa` tinyint(1) NOT NULL DEFAULT 0');
@@ -1,39 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0090. Guarded like mysql_mig_22, because there is
-- no per-file applied-state tracking and a replay must do nothing.
DROP PROCEDURE IF EXISTS _puter_add_user_app_permission_index;
DELIMITER //
CREATE PROCEDURE _puter_add_user_app_permission_index()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user_to_app_permissions'
AND INDEX_NAME = 'idx_user_to_app_permissions_permission'
) THEN
ALTER TABLE `user_to_app_permissions`
ADD INDEX `idx_user_to_app_permissions_permission` (`permission`);
END IF;
END//
DELIMITER ;
CALL _puter_add_user_app_permission_index();
DROP PROCEDURE IF EXISTS _puter_add_user_app_permission_index;
@@ -1,49 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Recompute `clean_email` for Apple's domains: dots are significant there, so
-- rows written while they were stripped still collapse two mailboxes into one.
-- Only rows still holding exactly what the old rule produced, so a NULL or a
-- legacy un-lowercased value is left alone -- rewriting those would fold `+tag`
-- or case into a key another row already owns. Restores dots and nothing else.
UPDATE `user`
SET `clean_email` = CONCAT(SUBSTRING_INDEX(SUBSTRING_INDEX(LOWER(`email`), '@', 1), '+', 1), '@', SUBSTRING_INDEX(LOWER(`email`), '@', -1))
WHERE `email` IS NOT NULL
AND LOCATE('@', `email`) > 1
AND (
LOWER(`email`) LIKE '%@icloud.com'
OR LOWER(`email`) LIKE '%@me.com'
OR LOWER(`email`) LIKE '%@mac.com'
)
AND `clean_email` = CONCAT(REPLACE(SUBSTRING_INDEX(SUBSTRING_INDEX(LOWER(`email`), '@', 1), '+', 1), '.', ''), '@', SUBSTRING_INDEX(LOWER(`email`), '@', -1))
AND `clean_email` <> CONCAT(SUBSTRING_INDEX(SUBSTRING_INDEX(LOWER(`email`), '@', 1), '+', 1), '@', SUBSTRING_INDEX(LOWER(`email`), '@', -1));
-- The same collapse decided who a pending invite belongs to, so the holder
-- of the undotted address claims invites addressed to the dotted one. The
-- typed address rides along as `invitedAddress` whenever it differed.
UPDATE `share`
SET `recipient_email` = CONCAT(SUBSTRING_INDEX(SUBSTRING_INDEX(LOWER(JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))), '@', 1), '+', 1), '@', SUBSTRING_INDEX(LOWER(JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))), '@', -1))
WHERE JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress')) IS NOT NULL
AND LOCATE('@', JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))) > 1
AND (
LOWER(`recipient_email`) LIKE '%@icloud.com'
OR LOWER(`recipient_email`) LIKE '%@me.com'
OR LOWER(`recipient_email`) LIKE '%@mac.com'
)
AND `recipient_email` = CONCAT(REPLACE(SUBSTRING_INDEX(SUBSTRING_INDEX(LOWER(JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))), '@', 1), '+', 1), '.', ''), '@', SUBSTRING_INDEX(LOWER(JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))), '@', -1))
AND `recipient_email` <> CONCAT(SUBSTRING_INDEX(SUBSTRING_INDEX(LOWER(JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))), '@', 1), '+', 1), '@', SUBSTRING_INDEX(LOWER(JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))), '@', -1));
@@ -1,52 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- The owner of the shared entry, recorded on the share row. Mirrors SQLite
-- migration 0092. Guarded like mysql_mig_45, because there is no per-file
-- applied-state tracking and a replay must do nothing.
DROP PROCEDURE IF EXISTS _puter_add_share_entry_owner;
DELIMITER //
CREATE PROCEDURE _puter_add_share_entry_owner()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND COLUMN_NAME = 'entry_owner_user_id'
) THEN
ALTER TABLE `share` ADD COLUMN `entry_owner_user_id` BIGINT DEFAULT NULL;
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_entry_owner'
) THEN
ALTER TABLE `share` ADD INDEX `idx_share_entry_owner` (`entry_owner_user_id`, `id`);
END IF;
END//
DELIMITER ;
CALL _puter_add_share_entry_owner();
DROP PROCEDURE IF EXISTS _puter_add_share_entry_owner;
UPDATE `share` JOIN `fsentries` ON `fsentries`.`id` = `share`.`fsentry_id`
SET `share`.`entry_owner_user_id` = `fsentries`.`user_id`
WHERE `share`.`fsentry_id` IS NOT NULL
AND `share`.`entry_owner_user_id` IS NULL;
@@ -1,26 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Uid and owner of each deleted app. Mirrors SQLite migration 0094.
--
-- Idempotent: `CREATE TABLE IF NOT EXISTS` lets the directory replay safely.
CREATE TABLE IF NOT EXISTS `deleted_apps` (
`uid` VARCHAR(40) NOT NULL,
`owner_user_id` INT UNSIGNED DEFAULT NULL,
PRIMARY KEY (`uid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
@@ -1,50 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Share levels on `kv_share_handles`. See
-- sqlite/0095_kv-share-handle-levels.sql for the rationale. No per-file
-- applied-state tracking, so every step is guarded and a replay does nothing.
CALL _puter_add_col('kv_share_handles', 'revoked_by', '`revoked_by` varchar(8) DEFAULT NULL');
CALL _puter_add_col('kv_share_handles', 'label', '`label` varchar(512) DEFAULT NULL');
DROP PROCEDURE IF EXISTS _puter_add_kv_share_handle_indexes;
DELIMITER //
CREATE PROCEDURE _puter_add_kv_share_handle_indexes()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'kv_share_handles'
AND INDEX_NAME = 'idx_kv_share_handles_grantee'
) THEN
ALTER TABLE `kv_share_handles` ADD INDEX `idx_kv_share_handles_grantee` (`grantee_user_id`, `id`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'kv_share_handles'
AND INDEX_NAME = 'idx_kv_share_handles_app'
) THEN
ALTER TABLE `kv_share_handles` ADD INDEX `idx_kv_share_handles_app` (`app_uid`);
END IF;
END//
DELIMITER ;
CALL _puter_add_kv_share_handle_indexes();
DROP PROCEDURE IF EXISTS _puter_add_kv_share_handle_indexes;
@@ -1,72 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Share-handle quota, listing and purge indexes. See
-- sqlite/0096_kv-share-handle-quota-indexes.sql for what each serves. No
-- per-file applied-state tracking, so every index is guarded and a replay does
-- nothing.
DROP PROCEDURE IF EXISTS _puter_add_kv_share_handle_quota_indexes;
DELIMITER //
CREATE PROCEDURE _puter_add_kv_share_handle_quota_indexes()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'kv_share_handles'
AND INDEX_NAME = 'idx_kv_share_handles_owner_live'
) THEN
ALTER TABLE `kv_share_handles` ADD INDEX `idx_kv_share_handles_owner_live` (`owner_user_id`, `revoked_at`, `app_uid`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'kv_share_handles'
AND INDEX_NAME = 'idx_kv_share_handles_owner_app'
) THEN
ALTER TABLE `kv_share_handles` ADD INDEX `idx_kv_share_handles_owner_app` (`owner_user_id`, `app_uid`, `id`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'kv_share_handles'
AND INDEX_NAME = 'idx_kv_share_handles_grantee_app'
) THEN
ALTER TABLE `kv_share_handles` ADD INDEX `idx_kv_share_handles_grantee_app` (`grantee_user_id`, `app_uid`, `id`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'kv_share_handles'
AND INDEX_NAME = 'idx_kv_share_handles_pair'
) THEN
ALTER TABLE `kv_share_handles` ADD INDEX `idx_kv_share_handles_pair` (`owner_user_id`, `grantee_user_id`, `revoked_at`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'kv_share_handles'
AND INDEX_NAME = 'idx_kv_share_handles_revoked'
) THEN
ALTER TABLE `kv_share_handles` ADD INDEX `idx_kv_share_handles_revoked` (`revoked_at`);
END IF;
END//
DELIMITER ;
CALL _puter_add_kv_share_handle_quota_indexes();
DROP PROCEDURE IF EXISTS _puter_add_kv_share_handle_quota_indexes;
@@ -1,38 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Lets a holder's grant check read only the permissions it names. Guarded, so
-- a replay does nothing.
DROP PROCEDURE IF EXISTS _puter_add_u2u_holder_permission_index;
DELIMITER //
CREATE PROCEDURE _puter_add_u2u_holder_permission_index()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user_to_user_permissions'
AND INDEX_NAME = 'idx_user_to_user_permissions_holder_permission'
) THEN
ALTER TABLE `user_to_user_permissions` ADD INDEX `idx_user_to_user_permissions_holder_permission` (`holder_user_id`, `permission`);
END IF;
END//
DELIMITER ;
CALL _puter_add_u2u_holder_permission_index();
DROP PROCEDURE IF EXISTS _puter_add_u2u_holder_permission_index;
@@ -1,53 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- `notification`.`group_key` and `replaced_at`. See
-- sqlite/0098_notification-groups.sql for the rationale. The key is a hex
-- digest, so ascii and compared byte for byte. No per-file applied-state
-- tracking, so every step is guarded and a replay does nothing.
CALL _puter_add_col('notification', 'group_key', '`group_key` char(64) CHARACTER SET ascii COLLATE ascii_bin DEFAULT NULL');
CALL _puter_add_col('notification', 'replaced_at', '`replaced_at` bigint DEFAULT NULL');
DROP PROCEDURE IF EXISTS _puter_add_notification_group_indexes;
DELIMITER //
CREATE PROCEDURE _puter_add_notification_group_indexes()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'notification'
AND INDEX_NAME = 'idx_notification_group'
) THEN
ALTER TABLE `notification` ADD INDEX `idx_notification_group`
(`user_id`, `app_uid`, `group_key`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'notification'
AND INDEX_NAME = 'idx_notification_replaced'
) THEN
ALTER TABLE `notification` ADD INDEX `idx_notification_replaced`
(`user_id`, `audience`, `app_uid`, `replaced_at`);
END IF;
END//
DELIMITER ;
CALL _puter_add_notification_group_indexes();
DROP PROCEDURE IF EXISTS _puter_add_notification_group_indexes;
@@ -1,23 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Notifications through a share handle. See
-- sqlite/0099_kv-share-handle-notify.sql for the column. No per-file
-- applied-state tracking, so it goes through _puter_add_col and a replay does
-- nothing.
CALL _puter_add_col('kv_share_handles', 'notify', '`notify` tinyint(1) NOT NULL DEFAULT 0');
@@ -1,43 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- `notification`.`send_key`. See sqlite/0100_notification-send-key.sql for the
-- rationale. The key is a uuid, so ascii and compared byte for byte. No
-- per-file applied-state tracking, so every step is guarded and a replay does
-- nothing.
CALL _puter_add_col('notification', 'send_key', '`send_key` char(36) CHARACTER SET ascii COLLATE ascii_bin DEFAULT NULL');
DROP PROCEDURE IF EXISTS _puter_add_notification_send_key_index;
DELIMITER //
CREATE PROCEDURE _puter_add_notification_send_key_index()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'notification'
AND INDEX_NAME = 'idx_notification_send_key'
) THEN
ALTER TABLE `notification` ADD UNIQUE KEY `idx_notification_send_key`
(`send_key`), ALGORITHM=INPLACE, LOCK=NONE;
END IF;
END//
DELIMITER ;
CALL _puter_add_notification_send_key_index();
DROP PROCEDURE IF EXISTS _puter_add_notification_send_key_index;
@@ -1,66 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Handlers an app's user published into their own scope, kept apart from
-- `event_handlers` so code that predates them never reads them. See
-- sqlite/0101_user-scoped-handlers.sql for the column rationale.
--
-- Idempotent: `CREATE TABLE IF NOT EXISTS` with the indexes declared inline,
-- and a guarded column add. There is no per-file applied-state tracking, so a
-- replay has to be a no-op.
CREATE TABLE IF NOT EXISTS `event_user_handlers` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
-- Matches `apps`.`uid` exactly, charset included, so an equality against
-- one never falls back to a conversion.
`app_uid` char(40) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL,
`holder_user_id` int unsigned NOT NULL,
`name` varchar(128) NOT NULL,
`source` mediumtext NOT NULL,
`source_hash` char(64) NOT NULL,
`created_at` bigint NOT NULL,
`updated_at` bigint NOT NULL,
-- When the scope first held a handler; kept across handler churn, so the
-- rent sweep sees the scope's age, not its oldest remaining handler's.
`scope_since` bigint NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_event_user_handlers_app_holder_name` (`app_uid`, `holder_user_id`, `name`),
KEY `idx_event_user_handlers_holder_app` (`holder_user_id`, `app_uid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Guarded here rather than through `_puter_add_col`, so the file also applies
-- on its own to a database that never ran the earlier migrations.
DROP PROCEDURE IF EXISTS _puter_add_event_subscriptions_handler_scope;
DELIMITER //
CREATE PROCEDURE _puter_add_event_subscriptions_handler_scope()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'event_subscriptions'
AND COLUMN_NAME = 'handler_scope'
) THEN
ALTER TABLE `event_subscriptions`
ADD COLUMN `handler_scope` tinyint unsigned NOT NULL DEFAULT 0,
ALGORITHM=INSTANT;
END IF;
END//
DELIMITER ;
CALL _puter_add_event_subscriptions_handler_scope();
DROP PROCEDURE IF EXISTS _puter_add_event_subscriptions_handler_scope;
Loaded 100 of 2396 files, more files were not shown because too many files have changed in this diff. Show more