Compare commits

..
Author SHA1 Message Date
Nariman Jelveh 734715f461 Merge branch 'feat/app-user-feedback' of https://github.com/HeyPuter/puter into feat/app-user-feedback 2026-08-12 13:32:34 -07:00
jelveh b290c1d922 Merge remote-tracking branch 'origin/main' into feat/app-user-feedback 2026-08-12 09:22:00 -07:00
jelveh dd068e31d0 Require verified users for app feedback API
Add `requireVerified: true` to both app feedback routes (`GET /target` and `POST /`) in `AppFeedbackController`. This tightens access control so only verified user accounts can fetch feedback targets or submit app feedback.
2026-08-12 09:13:11 -07:00
jelveh 90cffbfcde refactor: prep the feedback cap error once instead of via a factory
Both throw sites are in one call and only one can ever run, so a plain
const reads the same and drops a function that existed only to defer a
constructor.
2026-08-11 22:11:24 -07:00
jelveh 25acf14c8b test: cover the feedback service and store layers directly
The service owns every feedback business rule — target resolution,
eligibility, message normalization, the durable caps, and the owner-email
preconditions — but was only reachable through the controller's tests. Give
it and the store their own suites so a regression names the layer it broke.

Service coverage adds the branches the route tests could not reach: a
blocked origin resolving to null rather than surfacing a 403, owners who are
suspended or unsubscribed, length measured after normalization, the 24h cap
window boundary, subject-header injection via the app title, and the email
links being rooted at config.origin.

The two describes already labelled `AppFeedbackService ...` move out of the
controller test, which keeps only the caller-facing promise that a failed
send still returns success.
2026-08-11 22:10:04 -07:00
Nariman Jelveh be781c5f1b fix: restore the feedback Send button's gradient on hover
1c2d33f14 added :not(:disabled) to the base .app-feedback-btn hover
rule, which raised its specificity above the primary button's hover rule
— so hovering Send swapped the blue gradient for the light hover
background, leaving white text on a pale pill. Give the primary hover
rule the same :not(:disabled) so it wins by source order again.
2026-08-11 19:57:56 -07:00
Nariman Jelveh 4cb0730875 feat: point the feedback email footer at the Dev Center
The footer told developers to turn feedback off via a puter.apps.update
one-liner, but the toggle lives in the Dev Center app settings — and
Dev Center is where apps get feedback enabled by default in the first
place. Reword it to 'manage it in the Dev Center' with a link built
from config.origin (like app_link) so it holds on self-hosted
deployments.
2026-08-11 19:19:38 -07:00
Nariman Jelveh 00b2fcdea3 docs: document the COOP false-resolve limitation of showFeedbackDialog
Under COOP the popup's opener link is severed, so the SDK deliberately
resolves false while the popup stays open and the user can still submit
(a feedback submission has no server read-back the way a permission
grant does). The documented contract ('resolves to true if the user
submitted feedback') was silently wrong on cross-origin-isolated pages —
state the limitation in the doc and the SDK jsdoc: false means 'not
confirmed', not 'not sent'.
2026-08-11 18:31:39 -07:00
Nariman Jelveh f70347b579 docs: state exactly what showFeedbackDialog shares with the developer
The doc claimed the dialog 'tells the user their username will be
shared', while the dialog's note talked only about the email address and
the implementation shares the username always plus the email (as
Reply-To) only when verified. Describe the actual disclosure: username
always, email when verified.
2026-08-11 18:31:02 -07:00
Nariman Jelveh 3d9691b11a fix: stop HTML-escaping email subject lines
Subjects were compiled with default Handlebars escaping, so the
app-user-feedback subject rendered a title like "Bob's App & Games" as
"Bob's App & Games" — literal entities in the recipient's mail
client. Subjects are plain-text headers, not HTML; compile them with
noEscape. Header safety is unaffected: the transport encodes newlines
and free-form values collapse whitespace upstream.
2026-08-11 18:30:29 -07:00
Nariman Jelveh c2fe813929 fix: make the feedback dialog's privacy note match what is shared
The note unconditionally said 'Your email address will be shared with
the developer so they can respond', but AppFeedbackService shares the
username always and the email only when it exists and is verified — an
unverified or temp-user sender was promised a reply path that never
materializes, and nobody was told about the username.

Show 'username and email' when the signed-in user's email is verified,
and 'username' otherwise.
2026-08-11 18:29:22 -07:00
Nariman Jelveh ef7fc05dcc fix: reject feedback origins longer than the source_origin column
readTargetParam accepted values up to 3000 chars but the raw origin is
stored verbatim into source_origin VARCHAR(2048) (MySQL/Postgres), so a
2049-3000 char origin passed every validation and then blew up the
INSERT with an HTTP 500 on Postgres/strict MySQL — or was silently
truncated on non-strict MySQL, corrupting the abuse-forensics value the
column exists for. Cap the param at the column size.
2026-08-11 18:27:42 -07:00
Nariman Jelveh 1c2d33f149 fix: gate feedback dialog Cancel/X on an in-flight submit
Escape and backdrop clicks were already ignored while the POST was
pending, but the X, Close, and Cancel buttons weren't — clicking one
mid-send settled the promise false and tore down the overlay while the
submission still landed server-side: the developer got the email, the
app was told sent=false, and a user who resubmitted 'the failed one'
sent a duplicate and burned a daily-cap slot.

Apply the same !sending gate to the buttons and disable them visually
while the send is in flight.
2026-08-11 18:26:47 -07:00
Nariman Jelveh b1fdc8d665 fix: carry feedback_enabled in suggested/recommended app summaries
launch_app uses options.app_obj verbatim when provided, and the
suggested-apps launch paths (open_item.js, UIWindowSearch.js,
UIDesktop.js) pass summaries from toAppSummary — which omitted
feedback_enabled. So an opted-in editor launched by opening a .txt file
showed no Send Feedback control in the dashboard drawer, while the same
app launched from the Apps tab (full puter.apps.get object) did.
2026-08-11 18:23:43 -07:00
Nariman Jelveh 1ecc23cf83 fix: guard app-triggered feedback dialog against desktop-lockout loops
The showFeedbackDialog IPC handler had no re-entry or abuse guard, and
the dialog it opens is a full-viewport overlay above the taskbar and
every window — so 'while (true) await puter.ui.showFeedbackDialog()'
kept the desktop permanently covered (for signed-out users, the same
loop spams the full-page signup window instead). Every dismissal just
settled the promise and let the app immediately reopen it.

Allow one dialog at a time, and back off reopens per app after each
dismissal that sent nothing: 10s, then 60s, then blocked until page
reload. A successful send resets the backoff, and user-initiated paths
(dashboard drawer) are unaffected since they don't go through IPC.
2026-08-11 18:21:43 -07:00
Nariman Jelveh 7230580b18 fix: settle showFeedbackDialog instead of hanging on older host GUIs
In the app environment showFeedbackDialog awaited an IPC reply with no
capability check. A host GUI that predates this feature (self-hosted
Puter running the live js.puter.com SDK) has no handler for the message
and never replies, so the promise documented as 'never rejects' also
never resolved.

The GUI now advertises the IPC dialogs it can answer via a
puter.gui_features param on the app iframe URL, and the SDK resolves
false when 'feedback-dialog' isn't listed. A reply timeout could not
substitute: legitimate replies only arrive when the user closes the
dialog, so any timeout would false-negative while the user is typing.
Older SDKs ignore the extra param.
2026-08-11 18:19:46 -07:00
Nariman Jelveh 1460751b41 fix: refuse feedback when the deployment cannot deliver it
With no email transport configured (the common self-hosted default),
submissions were stored in app_feedback — a table with no read path
beyond the abuse-cap COUNTs — the owner email was silently skipped, and
the sender was still shown 'Feedback sent. Thank you!'. The developer
never learns the feedback exists while the user believes it was
delivered.

Gate acceptsFeedback on clients.email.isConfigured so the pre-flight
reports enabled:false (the dialog shows its 'not accepting feedback'
pane) and submit returns 403 instead of swallowing messages. Owner-level
store-without-email cases (unconfirmed owner email, per-app email cap
overflow) keep their existing deliberate semantics.
2026-08-11 18:16:14 -07:00
Nariman Jelveh 4f96d371f4 fix: don't mint a user-app token as a side effect of the feedback popup
Every embedded_in_popup boot ran the user-app token exchange, and the
exchange is a write: /auth/get-user-app-token bootstraps an app row for
the opener origin, grants flag:app-is-authenticated (what makes the
site count as connected to the account), and creates its AppData dir.
So merely opening — or immediately cancelling — a send-feedback popup
recorded a user<->site relationship the read-only feedback flow never
needs: the server resolves the feedback target from the attested origin
without any of it.

Gate the exchange behind runsUserAppTokenExchange(action) in all three
popup paths that mint (main postAuthActions exchange, temp-user signup
success, manual signup fallback). request-permission keeps the exchange
since grants are written against the app row it bootstraps.
2026-08-11 18:13:13 -07:00
Nariman Jelveh c5cf1dc390 docs: disclose the Dev Center's feedback-on-by-default in SDK docs/types
The Dev Center deliberately creates apps with feedbackEnabled (see
32c950de5), but the SDK docs, apps.d.ts, and AppFeedbackService's class
doc all described feedback as strictly opt-in / default-false with no
qualification — so a Dev Center developer reading them would wrongly
conclude feedback is off for their app. State the Dev Center behavior
alongside the API default, and correct the update-path docs: an omitted
feedbackEnabled leaves the current value unchanged rather than
defaulting to false.
2026-08-11 18:07:47 -07:00
Nariman Jelveh 503a450b40 fix: make feedback daily caps fail closed under concurrent submissions
The per-user and per-user-per-app caps were check-then-insert and the
per-app email cap was count-then-send, so parallel requests (or multiple
nodes, or the route limiter failing open) could all read a stale
under-cap count and push past every limit — the exact scenario the
DB-backed caps exist to stop.

Now the user caps recount after the insert (own row included) and roll
the row back with 429 if a burst breached them, and the email cap claims
its slot (email_sent=1) before sending, recounts, and releases the slot
if over cap or if the send fails.
2026-08-11 18:06:05 -07:00
Nariman Jelveh f4dfbd735b fix: resolve 'app-'-prefixed app names in feedback target lookup
APP_NAME_REGEX allows names beginning with "app-" (e.g. the seeded
app-center), but resolveTargetApp's startsWith('app-') heuristic sent
those to a uid-only lookup with no name fallback, so feedback for such
apps 403'd even when enabled. Use AppStore.resolveApp (uid, then name)
like the rest of the codebase.
2026-08-11 18:03:15 -07:00
Nariman Jelveh 81402918d2 fix: feedback modal polish + share sender email
Address four issues with the app feedback UI:

- Dashboard app-drawer: the extra "feedback" control pushed the close
  button past the drawer's derived width and clipped it. A `has-feedback`
  modifier widens the surface by one button + gap so all three controls
  fit. The control's glyph is now a message bubble with text lines, which
  reads more clearly at 14px than the previous bare speech bubble.

- The feedback dialog is no longer a UIWindow. It's a from-scratch
  overlay modal in the spirit of the dashboard modals (uninstall,
  add-app): a fixed scrim + centered card with self-contained,
  theme-aware color tokens (light default + dark override), a bottom-sheet
  layout on narrow screens, backdrop/Escape close, and an entrance
  transition. This renders consistently across the three contexts it's
  opened from (desktop app-IPC, dashboard drawer, standalone popup), so
  the callers no longer pass UIWindow-specific window_options.

- Feedback now shares the sender's email (not just their username) with
  the developer so they can respond: the owner email sets Reply-To to the
  sender and shows the address in the body — but only when the sender's
  email is verified (an unverified address could be anyone's, so it's
  never used as a reply target). EmailClient.send gains an optional
  replyTo. The dialog note now says the email will be shared.

Tests: e2e updated for the new modal (7 pass); backend feedback suite
covers the verified/unverified sender-email split (sqlite + postgres);
EmailClient + GUI unit suites pass; type-check clean.
2026-08-11 17:11:36 -07:00
Nariman Jelveh 32c950de5b feat: enable app feedback by default in Dev Center
New apps created in Dev Center now have feedbackEnabled set on creation,
so users can send the developer feedback without any extra setup. A "User
Feedback" toggle in the app's settings lets developers turn it off (and
back on); it's wired into the save payload, the dirty-state tracking, and
the reset-to-original path like the neighboring toggles.

The Save update omits feedbackEnabled unless the toggle is present, and
the backend leaves an omitted field untouched, so the default survives
the create-then-save flow Dev Center runs. Add an SDK apps-suite guard
covering that round-trip (create-on -> unrelated update keeps it -> can
be turned off).
2026-08-11 16:47:12 -07:00
Nariman Jelveh 792361507e feat: add feedback control to the dashboard app-drawer
Surface the feedback dialog directly from the app window's chrome in
dashboard mode: apps that opt in (apps.feedbackEnabled) get a "Send
Feedback" button in the dashboard app-drawer, next to minimize/close.
It opens the same UIWindowAppFeedback dialog, targeting this app by uid.

The control is only rendered when the app opted in — feedback_enabled is
threaded from the launched app's metadata into the window options — and
the dialog still re-checks opt-in server-side, so a stale flag can't send
anywhere. Reuses the existing .dashboard-app-drawer-btn styling and the
app_feedback_title i18n string, so no new CSS or strings.

Adds e2e coverage: the control appears and opens the dialog for an
opted-in app, and is absent for an app that hasn't opted in.
2026-08-11 16:39:05 -07:00
Nariman Jelveh 4f267fb83e feat: app user feedback system
Add puter.ui.showFeedbackDialog(), letting users send feedback to an
app's developer. In the app environment the Puter desktop renders the
dialog; on a third-party website a puter.com popup hosts it. The message
is stored in a new app_feedback table and emailed to the app owner's
confirmed email — it never passes through the app's own code.

Feedback is strictly opt-in per app via a new apps.feedback_enabled
column (a real column, not an app-metadata key, so Dev Center's
whole-blob metadata saves can't silently erase it), settable through the
existing puter.apps.update path (feedbackEnabled).

Backend follows the layered stack: AppFeedbackStore (durable count
queries) -> AppFeedbackService (opt-in check, message normalization,
abuse caps, best-effort owner email) -> AppFeedbackController
(POST /app-feedback, GET /app-feedback/target). New app-user-feedback
email template uses the escaping-safe nl2br triple-stash.

Defensive by design:
- requireUserActor blocks app tokens, so feedback can't be submitted
  programmatically; guiOriginOnly keeps cross-origin pages out.
- App identity comes only from the validated IPC sender (desktop) or the
  browser-attested opener origin (popup), never from message contents.
- The send-feedback popup action is in NON_AUTH_POPUP_ACTIONS, so it
  never delivers a token to the opener.
- Layered limits: route rate limits, plus DB-count caps that fail closed
  when the limiter backend is down, plus a per-app daily owner-email cap.
- Owner email is fully best-effort: an unconfigured transport,
  unconfirmed/unsubscribed/suspended owner, or send failure never fails
  the request or blocks storage.
- The dialog and SDK method are resolve-only and always settle, so a
  caller is never left hanging.

Migrations for sqlite/mysql/postgres, puter.js types, docs, backend
tests (sqlite + postgres), and a Playwright e2e spec are included.
2026-08-11 16:31:27 -07:00
1399 changed files with 26245 additions and 246096 deletions

No files matched your search

-5
View File
@@ -16,8 +16,3 @@ MARIADB_PASSWORD=replace-with-strong-password
S3_ACCESS_KEY=puter
S3_SECRET_KEY=replace-with-strong-secret
S3_BUCKET=puter-local
# ── Inbound SMTP (compose profile `smtp`) ------------------------------
# Host port mapped to the receiver's 2525. 25 is what an MX record means;
# many hosting providers block it.
SMTP_PORT=25
-72
View File
@@ -1,72 +0,0 @@
# Default owner, including backend and SDK areas without a specialist below.
# The last matching rule takes precedence.
- @Salazareo
# GUI
/src/gui/ @jelveh @Salazareo
# Documentation, developer site, and CLI
/doc/ @reynaldichernando @Salazareo
/src/docs/ @reynaldichernando @Salazareo
/src/dev-center/ @reynaldichernando @Salazareo
/src/cli/ @reynaldichernando @Salazareo
# Backend AI
/src/backend/controllers/puterai/ @404oops @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/ai-\*/ @404oops @ProgrammerIn-wonderland @Salazareo
# Backend: AI, hosting, workers, networking, and WebDAV
/src/backend/controllers/hosting/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/webdav/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/wisp/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/util/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/workers/ @ProgrammerIn-wonderland @Salazareo
/src/backend/services/localworker/ @ProgrammerIn-wonderland @Salazareo
/src/backend/stores/subdomain/ @ProgrammerIn-wonderland @Salazareo
/extensions/workerSandbox\* @ProgrammerIn-wonderland @Salazareo
/src/worker/ @ProgrammerIn-wonderland @Salazareo
/src/worker-types/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/peer/ @ProgrammerIn-wonderland @Salazareo
# Backend: sharing, permissions, and teams
/src/backend/controllers/share/ @jfcastro92 @Salazareo
/src/backend/controllers/team/ @jfcastro92 @Salazareo
/src/backend/services/share/ @jfcastro92 @Salazareo
/src/backend/services/permission/ @jfcastro92 @Salazareo
/src/backend/services/acl/ @jfcastro92 @Salazareo
/src/backend/services/team/ @jfcastro92 @Salazareo
/src/backend/services/apps/AppPermissionService\* @jfcastro92 @Salazareo
/src/backend/stores/share/ @jfcastro92 @Salazareo
/src/backend/stores/permission/ @jfcastro92 @Salazareo
/src/backend/stores/group/ @jfcastro92 @Salazareo
/src/backend/stores/team/ @jfcastro92 @Salazareo
# SDK: AI, hosting, workers, and networking
/src/puter-js/src/modules/ai/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/hosting/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Workers* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/networking/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/lib/networkUtils* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/ai.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/hosting.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/workers.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/net.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Peer\* @ProgrammerIn-wonderland @Salazareo
# SDK: sharing and permissions
/src/puter-js/src/modules/perms/ @jfcastro92
/src/puter-js/src/modules/FileSystem/operations/share* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/unshare* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/getShares* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/listShared* @jfcastro92 @Salazareo
/src/puter-js/tests/api/suites/perms.suite.ts @jfcastro92 @Salazareo
/src/puter-js/tests/api/suites/sharing.suite.ts @jfcastro92 @Salazareo
-20
View File
@@ -1,20 +0,0 @@
version: 2
updates:
- package-ecosystem: "npm"
directory: "/"
schedule:
interval: "weekly"
open-pull-requests-limit: 10
ignore:
- dependency-name: "*"
update-types: ["version-update:semver-major", "version-update:semver-minor"]
- package-ecosystem: "docker"
directory: "/"
schedule:
interval: "weekly"
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
@@ -17,10 +17,10 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v7
uses: actions/setup-node@v4
with:
node-version: '24'
cache: 'npm'
+25 -54
View File
@@ -1,39 +1,28 @@
name: Backend Tests
# Always triggers so it can be a required PR check; the `changes` job below
# decides whether backend/extension paths actually changed (test-only changes
# match the same globs, since tests are co-located with the code), and every
# other job is skipped — not absent — when they didn't, which still satisfies
# a required check. puter.js SDK changes are covered by puterjs-tests.yaml.
# Runs on backend and extension changes. Tests are co-located with the
# code (src/backend/**/*.test.ts, extensions/**/*.test.ts), so test-only
# changes match the same globs. puter.js SDK changes are covered by
# puterjs-tests.yaml instead.
on:
pull_request:
types: [opened, synchronize, reopened]
paths:
- 'src/backend/**'
- 'extensions/**'
- 'tools/**'
- 'package.json'
- 'package-lock.json'
# The type-check job below reads these.
- 'tsconfig.json'
- 'tsconfig.build.json'
- '.github/workflows/backend-tests.yaml'
permissions:
contents: read
pull-requests: write
jobs:
changes:
runs-on: ubuntu-latest
outputs:
backend: ${{ steps.filter.outputs.backend }}
steps:
- uses: actions/checkout@v7
- uses: dorny/paths-filter@v4
id: filter
with:
filters: |
backend:
- 'src/backend/**'
- 'extensions/**'
- 'tools/**'
- 'package.json'
- 'package-lock.json'
- 'tsconfig.json'
- 'tsconfig.build.json'
- '.github/workflows/backend-tests.yaml'
# `tsconfig.build.json` builds with `noCheck: true`, so nothing else in CI
# runs the type checker and a missing export compiles to `undefined`,
# surfacing only when the call is finally reached at runtime. This diffs
@@ -46,15 +35,13 @@ jobs:
# Its own job rather than a step in `test`: that one runs a base/PR matrix for
# coverage comparison, and the check only needs the PR ref, once.
typecheck:
needs: changes
if: needs.changes.outputs.backend == 'true'
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v7
uses: actions/setup-node@v4
with:
node-version: '24'
cache: 'npm'
@@ -65,19 +52,7 @@ jobs:
- name: Type check (new errors only)
run: npm run typecheck
# `needs.changes` gates the steps below, not this job, on purpose: a
# job-level `if` on a matrix job skips the whole job as one unexpanded
# check ("test (${{ matrix.artifact }})", literally, since there's no
# matrix context to fill it in) instead of the two real per-artifact
# checks a required rule pins to ("test (base)" / "test (pr)") — so the
# required check never gets reported and the PR blocks forever. Letting
# the matrix always expand keeps the check names stable either way.
test:
needs: changes
# Overrides the default matrix naming, which would otherwise bake the
# base/head branch name (matrix.ref) into the check name — making it a
# different string on every PR and impossible to pin as a required check.
name: test (${{ matrix.artifact }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
@@ -90,53 +65,49 @@ jobs:
steps:
- name: Checkout ${{ matrix.ref }}
if: needs.changes.outputs.backend == 'true'
uses: actions/checkout@v7
uses: actions/checkout@v4
with:
ref: ${{ matrix.ref }}
repository: ${{ github.event.pull_request.head.repo.full_name }}
- name: Set up Node.js
if: needs.changes.outputs.backend == 'true'
uses: actions/setup-node@v7
uses: actions/setup-node@v4
with:
node-version: '24'
cache: 'npm'
- name: Install dependencies
if: needs.changes.outputs.backend == 'true'
run: npm ci
- name: Run backend tests with coverage
if: needs.changes.outputs.backend == 'true'
run: npm run test:backend -- --coverage
env:
CI: 'true'
- name: Upload coverage artifact
if: needs.changes.outputs.backend == 'true' && !cancelled()
uses: actions/upload-artifact@v7
if: always()
uses: actions/upload-artifact@v4
with:
name: backend-coverage-${{ matrix.artifact }}
path: src/backend/coverage/
retention-days: 14
report-coverage:
needs: [changes, test]
if: always() && needs.changes.outputs.backend == 'true'
needs: test
if: always()
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@v4
- name: Download PR coverage
uses: actions/download-artifact@v8
uses: actions/download-artifact@v4
with:
name: backend-coverage-pr
path: src/backend/coverage
- name: Download base coverage
uses: actions/download-artifact@v8
uses: actions/download-artifact@v4
with:
name: backend-coverage-base
path: coverage-base
@@ -1,25 +0,0 @@
name: Dependabot auto-merge
on: pull_request
permissions:
contents: write
pull-requests: write
jobs:
automerge:
runs-on: ubuntu-latest
if: github.actor == 'dependabot[bot]'
steps:
- name: Dependabot metadata
id: metadata
uses: dependabot/fetch-metadata@v3
with:
github-token: "${{ secrets.GITHUB_TOKEN }}"
- name: Enable auto-merge for patch updates
if: steps.metadata.outputs.update-type == 'version-update:semver-patch'
run: gh pr merge --auto --squash "$PR_URL"
env:
PR_URL: ${{ github.event.pull_request.html_url }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+8 -8
View File
@@ -45,30 +45,30 @@ jobs:
fi
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@v4
- name: Log in to GitHub Package Container registry
uses: docker/login-action@v4
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Log in to Docker Hub
uses: docker/login-action@v4
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Set up QEMU
uses: docker/setup-qemu-action@v4
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
uses: docker/setup-buildx-action@v3
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v6
uses: docker/metadata-action@v5
with:
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
@@ -82,7 +82,7 @@ jobs:
type=raw,value=main,enable=${{ github.event_name == 'push' || inputs.push_main }}
- name: Build and push Docker image
uses: docker/build-push-action@v7
uses: docker/build-push-action@v5
with:
platforms: linux/amd64,linux/arm64
context: .
@@ -95,7 +95,7 @@ jobs:
cache-to: type=gha,mode=max
- name: Sync README to Docker Hub
uses: peter-evans/dockerhub-description@v5
uses: peter-evans/dockerhub-description@v4
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
+2 -2
View File
@@ -11,10 +11,10 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v7
uses: actions/setup-node@v4
with:
node-version: '24'
cache: 'npm'
+3 -3
View File
@@ -9,7 +9,7 @@ jobs:
update-release-pr:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v3
- name: Setup Git
run: |
git config user.name github-actions
@@ -20,7 +20,7 @@ jobs:
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
branch: release
- uses: actions/checkout@v7
- uses: actions/checkout@v4
if: steps.find-pr.outputs.number == ''
with:
ref: release
@@ -31,7 +31,7 @@ jobs:
git reset --hard main
- name: Create/Update PR
if: steps.find-pr.outputs.number == ''
uses: peter-evans/create-pull-request@v8
uses: peter-evans/create-pull-request@v6
with:
token: ${{ secrets.GITHUB_TOKEN }}
commit-message: Update release branch
+27 -61
View File
@@ -1,42 +1,28 @@
name: Puter.js API Tests
# Always triggers so it can be a required PR check; the `changes` job below
# decides whether puter.js-relevant paths actually changed (the suites live
# in src/puter-js/tests, so test-only changes are covered by the same glob,
# plus the pieces the runners depend on: the worker preamble, the in-memory
# test env in testUtil, and the backend vitest config the API-tests config
# extends), and every other job is skipped — not absent — when they didn't,
# which still satisfies a required check. Backend and extension changes run
# backend-tests.yaml instead.
# Runs on puter.js SDK changes — the suites live in src/puter-js/tests,
# so test changes are covered by the same glob — plus the pieces the
# runners depend on: the worker preamble, the in-memory test env in
# testUtil, and the backend vitest config the API-tests config extends.
# Backend and extension changes run backend-tests.yaml instead.
on:
pull_request:
types: [opened, synchronize, reopened]
paths:
- 'src/puter-js/**'
- 'src/worker/**'
- 'src/backend/testUtil.ts'
- 'src/backend/vitest.config.ts'
- 'tools/**'
- 'package.json'
- 'package-lock.json'
- '.github/workflows/puterjs-tests.yaml'
permissions:
contents: read
pull-requests: write
jobs:
changes:
runs-on: ubuntu-latest
outputs:
puterjs: ${{ steps.filter.outputs.puterjs }}
steps:
- uses: actions/checkout@v7
- uses: dorny/paths-filter@v4
id: filter
with:
filters: |
puterjs:
- 'src/puter-js/**'
- 'src/worker/**'
- 'src/backend/testUtil.ts'
- 'src/backend/vitest.config.ts'
- 'tools/**'
- 'package.json'
- 'package-lock.json'
- '.github/workflows/puterjs-tests.yaml'
# The JSDoc in src/puter-js/src is the source of truth for the SDK's public
# types. The declarations shipped to npm are generated from it at build time
# and never committed, so what needs guarding is the JSDoc itself: this
@@ -46,16 +32,14 @@ jobs:
# Its own job rather than a step in `test`: that one builds bundles and
# installs a browser, and this needs neither.
types:
needs: changes
if: needs.changes.outputs.puterjs == 'true'
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v7
uses: actions/setup-node@v4
with:
node-version: '24'
cache: 'npm'
@@ -67,16 +51,14 @@ jobs:
run: npm run check:puterjs:types
test:
needs: changes
if: needs.changes.outputs.puterjs == 'true'
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v7
uses: actions/setup-node@v4
with:
node-version: '24'
cache: 'npm'
@@ -101,18 +83,7 @@ jobs:
# SDK coverage on both refs, reported as a PR comment — same shape as
# backend-tests.yaml. This run uses the istanbul-instrumented bundle
# (the `test` job above keeps exercising the production build).
# `needs.changes` gates the steps below, not this job, on purpose: a
# job-level `if` on a matrix job skips the whole job as one unexpanded
# check ("coverage (${{ matrix.artifact }})", literally, since there's no
# matrix context to fill it in) instead of two real per-artifact checks —
# confusing to read, and unusable if this ever gets pinned as a required
# check. Letting the matrix always expand keeps the check names stable.
coverage:
needs: changes
# Overrides the default matrix naming, which would otherwise bake the
# base/head branch name (matrix.ref) into the check name — making it a
# different string on every PR and impossible to pin as a required check.
name: coverage (${{ matrix.artifact }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
@@ -125,39 +96,34 @@ jobs:
steps:
- name: Checkout ${{ matrix.ref }}
if: needs.changes.outputs.puterjs == 'true'
uses: actions/checkout@v7
uses: actions/checkout@v4
with:
ref: ${{ matrix.ref }}
repository: ${{ github.event.pull_request.head.repo.full_name }}
- name: Set up Node.js
if: needs.changes.outputs.puterjs == 'true'
uses: actions/setup-node@v7
uses: actions/setup-node@v4
with:
node-version: '24'
cache: 'npm'
- name: Install dependencies
if: needs.changes.outputs.puterjs == 'true'
run: npm ci
- name: Install Playwright chromium
if: needs.changes.outputs.puterjs == 'true'
run: npx playwright install --with-deps chromium
# The base leg is best-effort — it only enriches the PR comment with
# a comparison, and the base ref may predate the coverage script.
- name: Run puter.js API tests with coverage
if: needs.changes.outputs.puterjs == 'true'
continue-on-error: ${{ matrix.artifact == 'base' }}
run: npm run test:puterjs:coverage
env:
CI: 'true'
- name: Upload coverage artifact
if: needs.changes.outputs.puterjs == 'true' && !cancelled()
uses: actions/upload-artifact@v7
if: always()
uses: actions/upload-artifact@v4
with:
name: puterjs-coverage-${{ matrix.artifact }}
path: src/puter-js/coverage/coverage-*.json
@@ -165,15 +131,15 @@ jobs:
if-no-files-found: ignore
report-coverage:
needs: [changes, coverage]
if: always() && needs.changes.outputs.puterjs == 'true'
needs: coverage
if: always()
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@v4
- name: Download PR coverage
uses: actions/download-artifact@v8
uses: actions/download-artifact@v4
with:
name: puterjs-coverage-pr
path: src/puter-js/coverage
@@ -183,7 +149,7 @@ jobs:
- name: Download base coverage
id: base-coverage
continue-on-error: true
uses: actions/download-artifact@v8
uses: actions/download-artifact@v4
with:
name: puterjs-coverage-base
path: coverage-base
+2 -2
View File
@@ -45,14 +45,14 @@ jobs:
fi
- name: Log in to GitHub Package Container registry
uses: docker/login-action@v4
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
uses: docker/setup-buildx-action@v3
- name: Create alias tag
env:
-3
View File
@@ -72,6 +72,3 @@ servers.json
config.*.json
volatile/
# Local build override
docker-compose.override.yml
+5 -39
View File
@@ -35,8 +35,6 @@ These apply everywhere — backend, puter.js, and GUI.
Keep comments light; prefer self-documenting code. Comment only when the _why_ is non-obvious or a usage detail would trip the next reader. Use `//` for single lines and `/** ... */` JSDoc when it genuinely needs more — if a comment runs long, it's probably too long. Don't restate the code and don't reference the current task, PR, or version — those rot. **No ticket references** (`PUT-1234`, `// fix for FOO-99`) in code, comments, or test names; describe the why in domain terms, not project-management terms. Use plain ASCII `-` in comment section dividers (`// -- Section --`), never box-drawing characters.
**Be concise — a comment is a sentence or two, not a paragraph block.** One or two lines for `//`, a few for JSDoc. Don't narrate the history ("used to swallow the failure, which meant…"), don't argue the design at length, and don't explain each parameter when the signature already does. State the constraint and stop. If the reasoning genuinely needs paragraphs, it belongs in a design doc, not above the function.
### Security & privacy
Before opening a PR, scan the diff for:
@@ -47,8 +45,6 @@ Before opening a PR, scan the diff for:
When in doubt, return less. Auth-, permission-, or data-export-related changes deserve an explicit callout in the PR description.
Commit messages and PR descriptions never credit anyone or mention a bug report. For security fixes, state what the change does mechanically ("scoped tokens do xyz now"), not the vulnerability or how it could be abused.
### Working rules of thumb
- **Run it, don't just compile it.** "It type-checks" is not "it works." Exercise the code path end-to-end at least once.
@@ -56,37 +52,6 @@ Commit messages and PR descriptions never credit anyone or mention a bug report.
- **Test new behavior.** Every new function, endpoint, driver method, or logic branch gets a test; every bug fix gets a regression test that fails before the fix. If something is genuinely hard to test, skip it but say so in the PR.
- **Boy Scout Rule, proportional to the change.** Fix the obvious typo or dead import in files you're already touching; don't ride a refactor along with a bug fix.
- **Understand what you commit.** AI assistance is fine; shipping code you couldn't defend in review is not.
- **A limit change is not done until the docs change with it.** Every rate limit, concurrency cap, quota, and allowance in the code is published in [src/docs/src/rate-limits-and-quotas.md](src/docs/src/rate-limits-and-quotas.md) — an undisclosed limit is one developers discover as a service failure. If a PR moves any of these numbers, the same PR updates that page. The numbers live in `src/backend/controllers/fs/limits.ts` (filesystem), `src/backend/drivers/util/aiLimits.ts` (all AI drivers), per-driver `rateLimit`/`concurrent` configs, and `src/backend/data/subPolicies/` (free-tier allowances).
---
## Local development configuration
To test a local Docker build, create `docker-compose.override.yml` in the repository
root (or merge into the existing file):
```yaml
services:
puter:
pull_policy: never
build:
context: .
```
Compose merges it automatically. Run `docker compose up -d --build` to build and
start the stack, and repeat after source changes. Keep local build settings in
this gitignored override instead of editing `docker-compose.yml`, so they stay
out of pull requests and do not conflict when pulling updates. See
[Self-Hosting Puter](doc/self-hosting.md#building-from-source-instead-of-pulling)
for the required runtime configuration.
For local API testing, agents may decide whether to set `"unlimitedMetering": true`
when budget checks block unrelated tests; no additional approval is needed.
Merge it into the ignored `config.json` for `npm start` or
`puter/config/config.json` for Docker, then restart Puter. It applies to all
accounts on that local instance, including guests, and usage is still recorded.
Keep normal settings when testing budget or subscription enforcement, and leave
shared defaults unchanged.
---
@@ -126,7 +91,6 @@ Follow the same layered structure inside an extension — unless it only needs a
- Vitest; test files sit next to the code they test (`*.test.ts` / `*.test.js`). Run with `npm run test:backend`.
- **Mock data, not methods.** Stub inputs (fixtures, fake rows, payloads), not the function under test or the layer beneath it — over-mocking produces tests that pass while production breaks. If you must mock, mock at a real boundary (a client/external service).
- **Prefer the test server over mocking deps.** `setupPuterTestEnv()` in [src/backend/testUtil.ts](src/backend/testUtil.ts) boots a fully in-memory backend; hit a real database/client shape where reasonable — integration shapes catch what mocked unit tests miss.
- **Test code, not docs.** Never write a test that reads a file under `src/docs/` and asserts on its wording or numbers. Docs are kept in step by the PR (see the limits rule above) and checked in review; a test that greps a markdown page fails on every rewording and verifies nothing about behavior.
---
@@ -147,15 +111,17 @@ Declare a shape where it belongs, and reference it with an `import(...)` type fr
Use `@typedef {Object}` + `@property` for any shape whose fields need documenting — it is the only JSDoc form that carries a doc comment per field into the generated declaration. Keep the inline object-literal form for small internal shapes with nothing to say about each field, and prefer `unknown` over `*`:
```js
/** @typedef {{ key: string; value: unknown }} KVEntry */
/** @typedef {{ key: string, value: unknown }} KVEntry */
```
Public (exposed) methods must carry JSDoc types — parameters and return value — with one `@overload` block per accepted call form, since those overloads _are_ the published signature. Unexposed/private helpers are typed at the contributor's discretion: annotate where it helps the next reader, and either way keep them clean. Members tagged `@internal` are stripped from the generated declarations, so use that tag rather than `@private` to keep something off the public surface.
Public (exposed) methods must carry JSDoc types — parameters and return value — with one `@overload` block per accepted call form, since those overloads *are* the published signature. Unexposed/private helpers are typed at the contributor's discretion: annotate where it helps the next reader, and either way keep them clean. Members tagged `@internal` are stripped from the generated declarations, so use that tag rather than `@private` to keep something off the public surface.
Typing in JS files is encouraged: annotate with JSDoc `@type`/`@param`/`@returns` using the TypeScript type system, and define shared shapes with `@typedef`. API types must not be `unknown` or untyped `...args` — spell out the real parameter and return shapes; the only exception is values passed through transparently to an upstream layer that owns their type. For example:
```js
/** @typedef {{ key: string; value: unknown }} KVEntry */
/**
* @typedef {{key:string, value: unknown}} KVEntry
*/
/** @type {KVEntry[]} */
let entries = [];
+35 -46
View File
@@ -6,40 +6,33 @@ We at **Puter** are committed to maintaining a secure experience for our users a
The following are in scope for this program:
- **The Puter open-source project** (available at [github.com/HeyPuter/puter](https://github.com/HeyPuter/puter))
- **`puter.com`**
- **`api.puter.com`**
* **The Puter open-source project** (available at [github.com/HeyPuter/puter](https://github.com/HeyPuter/puter))
* **`puter.com`**
* **`api.puter.com`**
Out-of-scope:
- Third-party services, applications, or libraries not maintained by Puter.
- Social engineering attacks (e.g., phishing against staff).
- Denial of Service (DoS), spam, or volumetric attacks.
- Physical security issues.
* Third-party services, applications, or libraries not maintained by Puter.
* Social engineering attacks (e.g., phishing against staff).
* Denial of Service (DoS), spam, or volumetric attacks.
* Physical security issues.
## Known Non-Issues (Please Check Before Submitting)
The following have already been reviewed and determined **not to be vulnerabilities**. Reports that only re-describe one of these are **not eligible for a reward** and will be closed as non-issues — even if they include new code references. Please review this list before submitting:
- **XSS / CORS / token issues scoped to `api.puter.com`.** The primary user session cookie lives on `puter.com`, not on the API origin, so reflected/stored XSS, CORS, or token handling on `api.puter.com` is out of scope. This explicitly includes **missing `Content-Security-Policy: sandbox` on any endpoint that serves a file inline** — `/token-read`, `/file`, `/fs/read`, WebDAV, and any MIME type the `applyInlineContentSecurity` allow-list does not cover (`text/xsl`, `image/svg+xml`, `application/xhtml+xml`, …). Script execution on the API origin has nothing to reach: there is no session cookie, no `localStorage`, and no ambient credential there. WebDAV additionally has to return an accurate MIME type for real WebDAV clients. The one exception we _do_ evaluate on its merits is anything that abuses the app-scoped `puter_token_v2` companion cookie set on the API host — if your report does not name a specific credential that is actually reachable from the API origin, it is this non-issue.
- **SSRF via `secureFetch`.** Production routes outbound requests through an isolated proxy that has no access to internal/SSRF-sensitive resources.
- **Attacks that depend on guessing an `appInstanceID` or app UID.** These are random 128-bit secret values and are not considered guessable.
- **Apps invoking drivers, creating workers, or using KV.** Applications are intended to do this; worker permissions are scoped to the owning app. This is by design.
- **App metadata or app user-count "leaks".** This information is currently public by design. This covers **private apps** too: private apps are an unreleased, admin-only beta for paid listings in the Puter App Center. Their `index_url` and other metadata are not secret, the flag is not user-settable, and no production app relies on it. Reports that a private app's metadata is reachable through a sibling route (taskbar items, `/get-launch-apps`, `index_url` drift on the hosting gate, etc.) are normalization work, not vulnerabilities.
- **Worker hot-reload driven by a share recipient.** Sharing a worker's source file with write access is how you let a collaborator change that worker's code and have it deploy live. The redeploy running as the file owner, with the owner's worker credentials, is the intended behavior of that grant — the owner chose to hand out write access to executable code. If you do not want that, do not share the source file with write access.
- **Share-mode downgrades that do not cascade.** Demoting a delegate does not retroactively revoke the grants that delegate already issued downstream. Shares are independent grants once issued, not a live-evaluated chain. Revoking the downstream grant is a separate, supported action.
- **Concurrency / TOCTOU overspend of credits or storage quota.** Firing concurrent requests to exceed an AI credit hold or a storage quota by some margin is known and accepted; the per-account concurrency and rate-limit gates are the control we rely on, and the residual overage is bounded and billed. Report these only if you can demonstrate _unbounded_ overspend.
- **Session and credential lifecycle around 2FA enrollment.** Enrolling in 2FA is not a credential-revocation event: existing authenticated sessions stay valid, and outstanding password-reset links stay valid. Likewise, account-management actions behind the `/user-protected/*` gate are password-gated by design, not TOTP-gated. A recovery code that stops being accepted fails _closed_ — it denies access rather than granting it, so it is a bug report, not a vulnerability.
- **Group permissions** (`/auth/revoke-user-group` and siblings). The group-permission surface is unfinished and not exposed anywhere in the product. Findings there are not reachable by any user.
- **Correctness bugs with no security boundary crossed.** For example, an API that reports success while silently leaving data behind. We want these reported — please open a GitHub issue — but they are not bounty-eligible.
- **Missing Subresource Integrity (SRI) on external scripts/styles.** Accepted as hardening, not as a vulnerability, absent a demonstrated compromise of the CDN in question.
- **General "token in a URL" / token-lifetime / token-storage designs** — signed directory URLs exposing children, a write signature implying read, app tokens outliving a web session, or the session token being held in `localStorage` on `puter.com`. These are current intended behaviors. Shorter-lived tokens with a refresh flow are on the roadmap; their absence is not a vulnerability on its own. Show a reproducible attack that does not already presuppose script execution on `puter.com`.
- **Missing PKCE, unverified `id_token` signatures, or other OIDC hardening.** Provider tokens are obtained through a server-to-server authorization-code exchange with the provider's token endpoint over TLS, so the resulting `id_token` / userinfo claims are trusted from that channel rather than from local JWKS signature verification — the callback never accepts a caller-supplied `id_token`. Adding local signature / `aud` / `iss` / `exp` checks is welcome defense-in-depth (please open a GitHub issue/PR), but their absence is not an account-takeover vector on its own.
- **JWT "algorithm confusion" / unpinned `algorithms` in `jwt.verify`.** Puter's session tokens are HMAC-signed (HS256) with a server-side secret, and there is no asymmetric public key anywhere in the verification path, so `alg`-substitution attacks do not apply. Explicitly pinning `algorithms` is a fine hardening PR, but it is not a vulnerability.
- **Static-source "SQL injection" in internal pagination.** Findings such as `LIMIT ${limit}` in list/notification queries: the limit is numerically coerced and clamped before it reaches the query, so it is not reachable with attacker-controlled string input. Hardening PRs to the internal stores are welcome, but these are not exploitable as reported.
- **Deprecated `saveTo*` GUI app messages.** The legacy `saveToDesktop` / `saveToDocuments` / etc. app-IPC handlers can create — never overwrite — new files in standard user folders. The behavior is non-destructive, path-traversal-safe, and deprecated (slated for removal); it is not treated as a sandbox escape.
- **Best-practice suggestions** such as login/registration username enumeration (kept intentionally for UX) or unauthenticated unsubscribe links (industry norm).
- **Rate-limiting suggestions for TURN credential issuance** (intentional; not billed per tunnel).
* **XSS / CORS / token issues scoped to `api.puter.com`.** The primary user session cookie lives on `puter.com`, not on the API origin, so reflected/stored XSS, CORS, or token handling on `api.puter.com` is generally out of scope. Two exceptions we *do* evaluate on their merits — please report these: (a) attacker-controlled content served **inline** (e.g. an HTML `Content-Type`) from API file/response endpoints, and (b) anything that abuses the app-scoped `puter_token_v2` companion cookie set on the API host.
* **SSRF via `secureFetch`.** Production routes outbound requests through an isolated proxy that has no access to internal/SSRF-sensitive resources.
* **Attacks that depend on guessing an `appInstanceID` or app UID.** These are random 128-bit secret values and are not considered guessable.
* **Apps invoking drivers, creating workers, or using KV.** Applications are intended to do this; worker permissions are scoped to the owning app. This is by design.
* **App metadata or app user-count "leaks".** This information is currently public by design.
* **General "token in a URL" / token-lifetime designs** — signed directory URLs exposing children, a write signature implying read, or app tokens outliving a web session. These are current intended behaviors.
* **Missing PKCE, unverified `id_token` signatures, or other OIDC hardening.** Provider tokens are obtained through a server-to-server authorization-code exchange with the provider's token endpoint over TLS, so the resulting `id_token` / userinfo claims are trusted from that channel rather than from local JWKS signature verification — the callback never accepts a caller-supplied `id_token`. Adding local signature / `aud` / `iss` / `exp` checks is welcome defense-in-depth (please open a GitHub issue/PR), but their absence is not an account-takeover vector on its own.
* **JWT "algorithm confusion" / unpinned `algorithms` in `jwt.verify`.** Puter's session tokens are HMAC-signed (HS256) with a server-side secret, and there is no asymmetric public key anywhere in the verification path, so `alg`-substitution attacks do not apply. Explicitly pinning `algorithms` is a fine hardening PR, but it is not a vulnerability.
* **Static-source "SQL injection" in internal pagination.** Findings such as `LIMIT ${limit}` in list/notification queries: the limit is numerically coerced and clamped before it reaches the query, so it is not reachable with attacker-controlled string input. Hardening PRs to the internal stores are welcome, but these are not exploitable as reported.
* **Deprecated `saveTo*` GUI app messages.** The legacy `saveToDesktop` / `saveToDocuments` / etc. app-IPC handlers can create — never overwrite — new files in standard user folders. The behavior is non-destructive, path-traversal-safe, and deprecated (slated for removal); it is not treated as a sandbox escape.
* **Best-practice suggestions** such as login/registration username enumeration (kept intentionally for UX) or unauthenticated unsubscribe links (industry norm).
* **Rate-limiting suggestions for TURN credential issuance** (intentional; not billed per tunnel).
If you believe you have a **genuinely new** exploit chain that defeats one of these rationales (for example, demonstrating a sensitive credential that really is reachable on `api.puter.com`), say so explicitly and show why the reasoning above does not apply.
@@ -47,7 +40,7 @@ If you believe you have a **genuinely new** exploit chain that defeats one of th
To participate, you must:
1. **Report responsibly**: Provide detailed steps to reproduce the issue, including proof-of-concept code, screenshots, or a screen recording (see _Proof of Reproduction_ below).
1. **Report responsibly**: Provide detailed steps to reproduce the issue, including proof-of-concept code, screenshots, or a screen recording (see *Proof of Reproduction* below).
2. **Do no harm**: Do not exfiltrate, modify, or delete data. Only access your own account or test data.
3. **Respect availability**: Do not perform denial-of-service attacks or automated scans that degrade service.
4. **Follow disclosure policy**: Do not publicly disclose vulnerabilities until we have confirmed and patched the issue.
@@ -58,30 +51,26 @@ Reports that do not meet these guidelines may not be eligible for a reward.
## Proof of Reproduction
Reports **MUST** demonstrate a **working, reproducible exploit with real impact** — not a theoretical or static-source-review finding. Please include:
Reports must demonstrate a **working, reproducible exploit with real impact** — not a theoretical or static-source-review finding. Please include:
- Exact steps to reproduce, the relevant request/response or code path, and the commit or version you tested.
- The **observed** result versus the **expected** result.
- For client-side, UI, or authentication-flow bugs: a short screen recording (≤ 2 minutes) showing the exploit working end-to-end on a real Puter instance.
- For server-side bugs: a runnable proof-of-concept and screenshot or video showing the attack process.
* Exact steps to reproduce, the relevant request/response or code path, and the commit or version you tested.
* The **observed** result versus the **expected** result.
* For client-side, UI, or authentication-flow bugs: a short screen recording (≤ 2 minutes) showing the exploit working end-to-end on a real Puter instance.
* For server-side bugs: a runnable proof-of-concept.
Reports based solely on reading the source ("source review only, not tested") or unverified AI/LLM-generated reports are the **lowest triage priority and not eligible for bounty**.
If you used an AI tool to help find an issue, you must personally verify that it actually reproduces before submitting.
Reports based solely on reading the source ("source review only, not tested") or unverified AI/LLM-generated reports are the **lowest triage priority and are generally not eligible**. If you used an AI tool to help find an issue, you must personally verify that it actually reproduces before submitting.
Please submit **one issue per report**.
Bundled "audit packs" of many speculative findings will not be elligible for bounty; send each confirmed issue separately.
Additionally please **DO NOT** submit too many issues at once; these will be treated as a bundled report, and will not be elligible for bounty.
Please submit **one issue per report**. Bundled "audit packs" of many speculative findings will be declined; send each confirmed issue separately.
## Reporting Process
To report a vulnerability, email us at: **[[email protected]](mailto:[email protected])**.
Include:
- A description of the vulnerability
- Steps to reproduce
- Potential impact
- Suggested remediation (if available)
- Paypal id where we can send the bounty (if eligible)
* A description of the vulnerability
* Steps to reproduce
* Potential impact
* Suggested remediation (if available)
We aim to acknowledge receipt within **72 hours** and provide a resolution timeline.
@@ -89,10 +78,10 @@ We aim to acknowledge receipt within **72 hours** and provide a resolution timel
We offer monetary rewards based on the severity of the vulnerability, as determined by our internal assessment (using CVSS as a guide).
- **Critical: \$1,000 – \$2,000**
- **High: \$500 – \$1,000**
- **Medium: \$200 – \$500**
- **Low: \$50 – \$100**
* **Critical: \$1,000 – \$2,000**
* **High: \$500 – \$1,000**
* **Medium: \$200 – \$500**
* **Low: \$50 – \$100**
Non-security issues, suggestions, and best practices feedback are always welcome, but may not qualify for a reward.
If multiple researchers report the same issue, the bounty will be awarded to the first eligible report we receive.
-2
View File
@@ -32,8 +32,6 @@ Don't commit code you couldn't have written, debugged, or defended yourself. Rea
If you add or change a public API (an endpoint, driver method, or puter-js method), follow [doc/contributing-apis.md](doc/contributing-apis.md) — backward compatibility, [developer docs](src/docs/), types, and tests all move in the same PR.
The same rule applies to limits: rate limits, concurrency caps, quotas, and allowances are published in [src/docs/src/rate-limits-and-quotas.md](src/docs/src/rate-limits-and-quotas.md), and a PR that changes one of those numbers updates that page in the same PR. A limit nobody published is a limit developers only discover as a service failure.
## 6. Boy Scout Rule — leave it 1% better
![Boy Scout Rule](https://imgs.search.brave.com/DMmIWl5-NuZVtrR9kXBb06AKF8kturkgSW9UMb2-6m4/rs:fit:860:0:0:0/g:ce/aHR0cHM6Ly9sYXdz/b2Zzb2Z0d2FyZWVu/Z2luZWVyaW5nLmNv/bS9pbWFnZXMvbGF3/cy9ib3ktc2NvdXQt/cnVsZS5wbmc)
-2
View File
@@ -79,8 +79,6 @@ ENV PUTER_CONFIG_PATH=/etc/puter/config.json
ENV NODE_OPTIONS=--enable-source-maps
EXPOSE 4100
# The inbound SMTP receiver, when run from this image with the `smtp` profile.
EXPOSE 2525
USER node
+7 -6
View File
@@ -10,10 +10,12 @@
·
<a href="https://apps.puter.com/">App Store</a>
·
<a href="https://builder.puter.com/">AI Builder</a>
·
<a href="https://developer.puter.com" target="_blank">Developers</a>
·
<a href="https://discord.com/invite/PQcx7Teh8u">Discord</a>
·
<a href="https://reddit.com/r/puter">Reddit</a>
·
<a href="https://twitter.com/HeyPuter">X</a>
</p>
@@ -48,8 +50,6 @@ npm start
```
**→** This should launch Puter at http://puter.localhost:4100
To run this checkout with Docker, follow [Building from source](./doc/self-hosting.md#building-from-source-instead-of-pulling). Create a local `docker-compose.override.yml` to select the local build; keeping these settings out of `docker-compose.yml` avoids conflicts when pulling updates and keeps local configuration out of pull requests.
<br/>
@@ -82,7 +82,10 @@ Puter is available as a hosted service at [**puter.com**](https://puter.com).
Connect with the maintainers and community through these channels:
- Bug report or feature request? Please [open an issue](https://github.com/HeyPuter/puter/issues/new/choose).
- Discord: [discord.com/invite/PQcx7Teh8u](https://discord.com/invite/PQcx7Teh8u)
- X (Twitter): [x.com/HeyPuter](https://x.com/HeyPuter)
- Reddit: [reddit.com/r/puter/](https://www.reddit.com/r/puter/)
- Mastodon: [mastodon.social/@puter](https://mastodon.social/@puter)
- Security issues or abuse reports? [[email protected]](mailto:[email protected])
- Email maintainers at [[email protected]](mailto:[email protected])
@@ -117,9 +120,7 @@ This repository, including all its contents, sub-projects, modules, and componen
- [Korean / 한국어](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.ko.md)
- [Malay / Bahasa Malaysia](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.my.md)
- [Malayalam / മലയാളം](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.ml.md)
- [Dutch / Nederlands](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.nl.md)
- [Polish / Polski](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.pl.md)
- [Odia / ଓଡ଼ିଆ](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.od.md)
- [Portuguese / Português](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.pt.md)
- [Punjabi / ਪੰਜਾਬੀ](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.pa.md)
- [Romanian / Română](https://github.com/HeyPuter/puter/blob/main/doc/i18n/README.ro.md)
-4
View File
@@ -22,10 +22,6 @@
"default_temp_group": "b7220104-7905-4985-b996-649fdcdb3c8f",
"storage_capacity": 104857600,
"disable_user_signup": false,
"teams_enabled": false,
"max_teams_per_user": 1,
"max_seats_per_team_free": 4,
"max_seats_per_team_paid": 40,
"strict_email_verification_required": false,
"gui_assets_root": "./src/gui",
"puterjs_root": "./src/puter-js/dist",
+3 -126
View File
@@ -32,10 +32,6 @@
"port": 4100,
// Externally-visible port (set this when behind a reverse proxy on 80/443).
"pub_port": 4100,
// Idle keep-alive timeout in ms. Must stay above the idle timeout of any
// proxy in front, or the proxy reuses connections this server has closed
// and its clients see 502s.
"keep_alive_timeout": 620000,
"protocol": "http",
"domain": "puter.localhost",
// Fully-qualified externally-visible URL. Computed from protocol/domain/
@@ -255,99 +251,6 @@
}
},
// ── Inbound mail (separate process) ─────────────────────────────────
// Mailboxes addressed at one of your own domains. `secret` gates the
// ingress endpoint that files a message into the recipient's ~/.mail.
//
// `localServer` additionally runs an SMTP listener that accepts mail from
// other mail servers and hands each message to that endpoint. It runs as
// its own process - `npm run start:smtp`, or the `smtp` compose profile -
// and stores nothing itself. Without it, nothing delivers to the endpoint
// unless you point something else at it.
//
// The listener is unauthenticated by nature: anything on the internet may
// hand it a message for an address in `localDomains`. It performs no spam,
// SPF, DKIM or DMARC checks. Read the self-hosting guide before exposing
// it.
// "userEmail": {
// "secret": "a long random string",
// "localServer": true,
// "localDomains": ["example.com"],
// // Container-internal by default; publish 25 onto it from outside.
// "localPort": 2525,
// // Defaults to <api_base_url>/email/ingress. The endpoint is served
// // on the `api` subdomain, so when this points somewhere internal,
// // `localIngressHost` has to name that virtual host.
// "localIngressUrl": "http://puter:4100/email/ingress",
// "localIngressHost": "api.example.com",
// "localHostname": "mx.example.com",
// "localMaxRecipients": 50,
// // Concurrent connections. Each message is held in memory while it
// // is delivered, so this also bounds memory use.
// "localMaxClients": 20
// },
// ── Sharing ─────────────────────────────────────────────────────────
// Email a recipient who already has an account about a new share. On
// unless set to false; they opt out via the unsubscribe link, or by
// blocking a sender. An invite to an address with no account always goes.
// "share_email_notifications": false,
//
// New shares one account may create per UTC day. Default 200.
// "share_daily_limit": 200,
//
// How often a share may interrupt its recipient — the notification pushed
// to their screen and the email with it. Over budget the share still
// succeeds and their notification is still brought up to date; only the
// interruption is dropped. The recipient bounds are what stop many senders
// burying one person between them. A non-positive value removes a bound.
// "share_notify_limits": {
// "pairWindowSeconds": 900,
// "pairDaily": 20,
// "recipientHourly": 10,
// "recipientDaily": 50,
// "emailBatchSeconds": 90
// },
// ── Teams / teams ──────────────────────────────────────────────
// Off by default, and off is the state every existing install stays in.
// The tables ship either way and sit inert; this gates whether `/teams`
// is registered at all, so with it off the paths do not exist rather than
// existing and refusing. It is also the backout: turning it off removes
// the feature without touching data.
"teams_enabled": false,
//
// Live teams one user may own. Soft-deleted ones do not count, so
// deleting yours frees the slot. Default 1.
"max_teams_per_user": 1,
//
// Seats one team may provision. Each seat is a real Puter account taking a
// name from the global username pool, so this is what bounds a team's blast
// radius. The owner's plan decides which of the two applies; defaults are 4
// and 40. Lowering either below a team's current seat count blocks new
// provisioning and disables nobody. Deployments upgrading from the old flat
// default of 50: without a billing extension every owner reads as free, so
// set `max_seats_per_team` to keep the old behavior.
"max_seats_per_team_free": 4,
"max_seats_per_team_paid": 40,
//
// One flat cap for every team, whatever the owner pays. Setting it
// overrides both of the above, so a deployment that does not sell seats can
// ignore plans entirely. Unset by default — leave it that way to keep the
// free cap meaningful.
// "max_seats_per_team": 50,
//
// Staged rollout: only these email domains may create a team, and the tab
// renders for them even with `gui_params.teams_ui` off. Members of an
// existing team always pass. Unset means everyone, gated by `teams_ui`.
// "teams_allowed_email_domains": ["puter.com"],
// ── Notifications ───────────────────────────────────────────────────
// How long a notification is kept, in days from creation. Acknowledged or
// not, a row past this is swept. Off by default; any value above 0 arms
// the sweep.
// "notificationRetentionDays": 14,
// ── Alarms / alerting ───────────────────────────────────────────────
// Where system alarms go. Severity is the routing decision — each
// transport takes everything at or above its own `minSeverity`:
@@ -421,10 +324,6 @@
"apiURL": ""
},
"gemini": { "apiKey": "" },
// Meta Model API (Muse Spark). `muse-spark-1.2-contributor`
// trades a ~12x discount for Meta training on its prompts and
// completions, so it is only served to callers that name it.
"meta": { "apiKey": "" },
"groq": { "apiKey": "" },
"deepseek": { "apiKey": "" },
"mistral": { "apiKey": "" },
@@ -442,20 +341,6 @@
"apiKey": "",
"apiBaseUrl": "https://llm.onerouter.pro/v1"
},
// BytePlus ModelArk. One key powers chat (Seed/GLM/DeepSeek), image
// generation (Seedream) and video generation (Seedance); `apiBaseUrl`
// selects the region; see
// https://docs.byteplus.com/en/docs/ModelArk/1330310 for options.
"byteplus": {
"apiKey": "",
"apiBaseUrl": "https://ark.ap-southeast.bytepluses.com/api/v3"
},
"zai": { "apiKey": "" },
"alibaba": { "apiKey": "" },
"hoonify": {
"apiKey": "",
"apiBaseUrl": "https://api.hoonify.ai/v1"
},
"together-ai": { "apiKey": "" },
// Local Ollama. `enabled: false` skips the auto-probe at startup
// (otherwise Puter logs ECONNREFUSED on every boot when no Ollama
@@ -477,6 +362,7 @@
"xai-image-generation": { "apiKey": "" },
// ─ Video generation ─
"openai-video-generation": { "apiKey": "" },
"together-video-generation": { "apiKey": "" },
"gemini-video-generation": { "apiKey": "" },
@@ -562,11 +448,8 @@
// operations that spend it — file transfers, KV calls. Recording is
// unaffected either way. `workers` extends the same refusal to
// worker-driven calls, which are exempt by default because a deployed
// worker has nowhere to surface a payment prompt. `subscriptions` covers
// the separate question of surfaces reserved for paid plans (the
// vendor-compatible AI endpoints): turn it off on a deployment with no
// paid plans, or every account is refused them.
// "meteringEnforcement": { "enabled": true, "workers": false, "subscriptions": true },
// worker has nowhere to surface a payment prompt.
// "meteringEnforcement": { "enabled": true, "workers": false },
// Fleet-wide spend rate, in micro-cents per minute, past which metering
// raises the `metering:excessiveGlobalUsageRate` alarm. Omit it (the
@@ -574,10 +457,4 @@
// what this deployment's normal traffic costs, so it has to be measured
// rather than guessed, and a stale number here alarms on healthy growth.
// "maxGlobalUsagePerMinute": 200000000
// How many full months past its own a monthly usage record is kept before
// it expires. Defaults to 3; set to 0 to keep every month's records
// forever. Changing it, including to 0, applies to records first written
// after the change; records already stamped keep their expiry.
// "meteringRetentionMonths": 3
}
+2 -22
View File
@@ -44,15 +44,7 @@ pass one explicitly unless you really mean "page someone".
event; the 429 is the whole signal, and alarming on it only produces noise
proportional to traffic. An *upstream provider* rate-limiting us is the
opposite case and still alarms (`upstream_rate_limited`, `info`) — that one
is not something we chose. The exception is a free model: nothing is billed,
nothing is actionable, and the throttling is the price of the model, so the
AI chat driver marks those `noAlarm` and only paid models still record.
`noAlarm` is the code-level mute: an `HttpError` carrying it skips the
terminal gate entirely, no matter its status or code. It is for failures the
call site *already knows* are expected and traffic-proportional — reach for it
there, not as a way to quiet an alarm you haven't diagnosed
(`severityOverrides` below is the knob for that).
is not something we chose.
An extension whose signals are all one tier can default its own local
`raiseAlarm` helper to that tier instead of repeating it at every call site —
@@ -78,11 +70,6 @@ this.clients.alarm.create(alarmId, message, fields, 'critical', {
The HTTP error handler uses it: its id is route + error signature, so a hot
loop of the same crash is one incident with N occurrences instead of N pages.
Dedup and Slack's repeat throttle mean a responder sees only the latest
occurrence, so the handler attaches the `HttpError`'s `fields` to the alarm as
`details` — put what a responder needs (which upstreams failed, and how)
there. Everything in `fields` is also returned to the client in the response
body, so it has to be safe to show the caller.
Reach for it anywhere else only when the id is that specific — otherwise a
per-request alarm can flood the pager.
@@ -126,14 +113,7 @@ config always has the last word.
### Repeat throttling
Two levels, and they stack.
The client itself backs off before a repeat is reported at all: every
occurrence is reported up to a small burst, then one per interval. A fault
recurring hundreds of times a second still counts every occurrence, but it
does not write a log line or build an alert payload for each one.
Below that, the chat transport won't repost the same alarm id within
The chat transport won't repost the same alarm id within
`repeatThrottleMs` (default 15 minutes). The first occurrence always posts,
and the next one that gets through reports how many piled up in between —
so a hot loop reads as one message with a count, not a wall of them.
-22
View File
@@ -59,16 +59,6 @@ We use [`Context`](../src/backend/core/context.ts) — backed by `AsyncLocalStor
Prefer explicit arguments. Reach for `Context` only when the value is truly request-scoped and would otherwise need to thread through many layers.
## Actors
An [`Actor`](../src/backend/core/actor.ts) is who a request is acting as. It has two app-shaped fields and they are not interchangeable: `app` is the app the actor carries *directly* (an app-under-user token), while `effectiveApp` is the app it ultimately acts as — its own, or the one that issued its access token. An access-token actor has no `app` of its own, so a gate that reads `app` answers "no app" for it and falls open.
`makeActor` resolves `effectiveApp` once at construction, and `assertResolvedActor` at the request edge rejects any actor that skipped it — so reading `effectiveApp` needs no fallback. Reach for `app` only where the direct app genuinely is the question, and say why in a comment.
- `actor.effectiveApp` — which app is acting. This is the default; nearly every scoping rule, namespace key and attribution wants it.
- `isAppActor(actor)` — a direct app actor specifically. Use it where a token must *not* be treated as its issuer, e.g. the permission scanners that read an app's own grant rows.
- `isPlainUserActor(actor)` — the account acting through nothing at all; `isAccountContext(actor)` for the wider "the account's own reach", which also admits a full-access token.
## Extensions
Extensions live alongside core ([packages/puter/extensions/](../extensions/)) and parallel the layered stack. They are meant for **non-crucial parts of the system** — things Puter still works without if removed.
@@ -106,18 +96,6 @@ extension.on('user.signup', (_key, data) => {
});
```
The `app.recommended` event lets extensions customize desktop recommendations.
Its `appNames` array starts as a fresh copy of the default ordered names on each
call. Listeners may mutate or replace it, including setting it to an empty array.
Async listeners are awaited before names are resolved to app summaries; names
that do not exist are skipped.
```ts
extension.on('app.recommended', (_key, data) => {
data.appNames = ['editor', 'camera'];
});
```
## Conventions
- **TypeScript preferred** in new code where feasible. Existing JS is fine; convert opportunistically when you're already touching a file.
+1 -4
View File
@@ -53,10 +53,7 @@ Both are supported ways to define an API. **Prefer a controller when you need fi
#### Middleware and gates
- **Controller routes** (and extension routes — same options) take [`RouteOptions`](../src/backend/core/http/types.ts): auth gates (`requireAuth`, `requireUserActor`, `noUserSession`, `adminOnly`, `allowedAppIds`, and the access-token controls), `subdomain` routing, per-route `rateLimit`, body parsers, and arbitrary extra `middleware`. The auth flavors are subtle and default-deny — read the JSDoc on each field before picking.
- **Driver methods** get their policies from the `@Driver` options: per-method `rateLimit` (limit/window/backend), `concurrent` in-flight caps (optionally `bySubscription`), `requireSubscription`, `requireReputation`, and `noUserSession`. The `/drivers/call` surface enforces them.
- **A surface only paying accounts should reach** declares `requireSubscription` — the route option on a controller endpoint, the per-method `@Driver({ requireSubscription })` block on a driver (`/drivers/call` is one shared route, so a driver's requirement can't live in route options). `true` accepts any plan that isn't free, so a plan registered by an extension counts without core naming it; an array of policy ids (`['business', 'pro']`) accepts only those; `false` is "no requirement", the same as leaving it out. An empty array is a boot error rather than a silent no-op — it reads as subscribers-only while admitting everyone. Off unless asked for, and answered from the metering service's cached per-actor subscription, so it costs a map lookup. Distinct from `requireCredits`: this asks which plan the account is on, not whether it has budget left. Deployments with no paid plans (self-hosted installs) turn the whole thing off with `meteringEnforcement.subscriptions: false`.
- **A surface only a trusted-enough account should reach** declares `requireReputation` — the route option on a controller endpoint, the per-method `@Driver({ requireReputation })` block on a driver. The value names a tier; what that tier takes is `reputationGate.tiers` in config, so the score a surface is worth is a deployment call and can be retuned without editing the surface. A tier the running config doesn't define is inert and everyone passes — an install that doesn't score its accounts must not start turning traffic away on a score it never computed — and `reputationGate.enabled: false` stops every declared gate at once. Opt-in, implies `requireAuth`, and denies with a bare 403 `reputation_required` that names neither the score nor the tier. Nothing in the tree declares one yet: the mechanism ships ahead of the enrollment.
- **An account must have verified a specific factor** — `requireVerified` (email, and only under `strict_email_verification_required`), `requirePhoneVerified`, `requireCardVerified`. These are opt-in and require the factor to have actually been verified, unlike the default-on gate that turns away accounts still carrying a pending verification the abuse harness asked for. `requireAnyVerified: ['phone', 'card']` is the OR of the last two: any listed factor verified at any point passes, as does a paid plan where `card` is listed (a paying account has a card on file already); otherwise only the factors the deployment can verify are asked for (an SMS provider configured, a card gate an extension reports on), and with none verifiable the gate is inert rather than locking the route on a self-hosted install. It denies with the first verifiable factor's code plus `factors`, the verifiable ones in the route's order, so a client can lead with one flow and offer the other. `verifiedFactorGate.enabled: false` in config stops every declared gate at once, for an SMS-provider outage.
- **Driver methods** get their policies from the `@Driver` options: per-method `rateLimit` (limit/window/backend), `concurrent` in-flight caps (optionally `bySubscription`), and `noUserSession`. The `/drivers/call` surface enforces them.
- **An endpoint that spends metered resources** on the caller's behalf — moving file content, making object-store requests, anything else the account is billed for — also declares `requireCredits: true`, which turns an account with nothing left of its budget away with a 402 before the handler runs. Endpoints that only describe or delete things deliberately don't: an account that has run out still has to be able to see what it has, clear it, and reach its billing pages. Drivers have no route options to declare this on, so they call `assertActorHasCredits` themselves ([src/backend/services/metering/enforcement.ts](../src/backend/services/metering/enforcement.ts)) — see `KVStoreDriver`, which does it once for every method.
### 3. puter.js
+1 -164
View File
@@ -35,12 +35,6 @@ Optional services (compose profile `ai`, opt-in):
| `puter-ollama` | `ollama/ollama` | Local LLM provider (CPU; GPU passthrough opt-in) |
| `puter-ollama-init` | `ollama/ollama` | One-shot — pulls the default model (`tinyllama`) on first boot |
Optional services (compose profile `smtp`, opt-in):
| Container | Image | Role |
| ------------ | ------------------------ | -------------------------------------------------------- |
| `puter-smtp` | `ghcr.io/heyputer/puter` | Receives mail over SMTP and files it into user mailboxes |
State lives under `./puter/data/<service>/`.
---
@@ -128,10 +122,6 @@ cat > puter/config/config.json <<EOF
"ollama": { "enabled": false }
},
"meteringEnforcement": {
"subscriptions": false
},
"trust_proxy": 1
}
EOF
@@ -146,20 +136,9 @@ Why these knobs:
- `database.migrationPaths` — Puter applies the bundled MySQL/MariaDB schema on boot. The migration files are idempotent, so it is safe to leave this configured across restarts.
- `dynamo.bootstrapTables: true` — Puter creates its KV table on boot. **Only set against a local emulator**, never real AWS.
- `dynamo.aws` keys are dummies; DynamoDB-local doesn't validate them but the AWS SDK requires _something_. **Note:** DynamoDB uses `access_key` / `secret_key` (snake_case); S3 below uses `accessKeyId` / `secretAccessKey` (camelCase). Not interchangeable.
- `meteringEnforcement.subscriptions: false` — a few surfaces (the OpenAI/Anthropic-compatible AI endpoints) are declared paid-plan-only. A self-hosted install has no paid plans, so every account sits on the free policy and those routes would answer `402 subscription_required` for everyone. Turning the plan gates off opens them; nothing else about metering changes.
- `providers.ollama.enabled: false` — Puter auto-probes a local Ollama at `127.0.0.1:11434` by default; without one running you'd see `ECONNREFUSED` on every boot. To run a bundled Ollama, see [Optional: local LLM (Ollama)](#optional-local-llm-ollama) below.
- `s3.s3Config.forcePathStyle: true` — RustFS / MinIO / fauxqs need path-style URLs (`<endpoint>/<bucket>`). Real AWS S3 wants virtual-hosted (`<bucket>.<endpoint>`) — drop this flag (or set `false`) when you swap to real S3.
- `s3.s3Config.publicEndpoint` — `endpoint` (`http://s3:9000`) only resolves inside the docker network; presigned upload/download URLs handed to the browser need a host-reachable URL. Caddy routes the `s3.<domain>` subdomain to RustFS internally and preserves the Host header end-to-end (required for S3 signature validation), so the browser hits the same port/protocol as the rest of the app — no separate published port, no mixed-content surprises when you turn on TLS. Switch to `https://s3.<your-domain>` once you enable TLS in Step 3. Real AWS S3 doesn't need this — its endpoint is already public; drop the field entirely.
- **Bucket CORS** — the browser uploads file bytes straight to RustFS with a cross-origin `PUT` to the presigned URL (Dev Center deploy, `puter.fs.upload`). That only works if the bucket answers the `OPTIONS` preflight with `Access-Control-Allow-*` headers. The `s3-init` container applies those rules on every boot (`put-bucket-cors`, origins `*` — the signed URL is the credential, and the SDK sends no cookies). If deploys fail with `No 'Access-Control-Allow-Origin' header is present`, re-run it: `docker compose run --rm s3-init`. Verify with:
```bash
curl -sk -X OPTIONS "https://s3.<domain>/<bucket>/test" \
-H "Origin: https://<domain>" \
-H "Access-Control-Request-Method: PUT" \
-H "Access-Control-Request-Headers: content-type" -D - -o /dev/null | grep -i access-control
```
You should see `access-control-allow-origin`, `allow-methods: GET, HEAD, PUT, POST, DELETE`, `allow-headers: *`, and `max-age: 3600`.
- `trust_proxy: 1` — Caddy terminates TLS and forwards `X-Forwarded-For`. Without this, `req.ip` is the docker-network address of the Caddy container instead of the real client IP, which breaks rate limiting and IP-based audit logs. `1` = one trusted hop (Caddy). Bump to `2` if you put Cloudflare in front of Caddy; never set `true` (it trusts every hop and makes XFF forgeable).
> If you ever change `MARIADB_PASSWORD` after first boot, `.env` alone won't update MariaDB — its credentials are baked into `./puter/data/mariadb/` on first init. Either rotate the password inside MariaDB by hand or `docker compose down && rm -rf ./puter/data/mariadb` to start fresh.
@@ -316,18 +295,6 @@ Used for password resets, email confirmation, and notifications. Without it thos
To require email confirmation before login, also set `"strict_email_verification_required": true`.
To read the mail yourself instead of sending it, point the transport at a local SMTP sink — [MailHog](https://github.com/mailhog/MailHog) catches everything and serves it at `http://localhost:8025`. Publish both ports; a container that only exposes them is unreachable from the host:
```sh
docker run -d -p 1025:1025 -p 8025:8025 mailhog/mailhog
```
```json
"email": { "from": "\"Puter\" <[email protected]>", "host": "127.0.0.1", "port": 1025, "secure": false, "ignoreTLS": true }
```
Share email is on once a transport exists: a recipient who already has an account is emailed as well as notified in-app, and they opt out with the unsubscribe link the mail carries. Set `"share_email_notifications": false` to keep those in the app only — an address with no account is still emailed, since there is nothing else to reach them with.
### Sign in with Google (or another OIDC provider)
```json
@@ -424,12 +391,6 @@ Every account then resolves to an unlimited policy. Usage is still recorded, so
the dashboard still shows what is being consumed; nothing is ever refused for
lack of budget.
For API testing, you can also set `"unlimitedMetering": true` in your
ignored runtime config: `puter/config/config.json` for Docker, or the repository's
`config.json` for `npm start`. Merge it into the existing config and restart Puter.
This includes guest accounts and applies to the whole local instance. Keep normal
metering settings when testing budget or subscription enforcement.
To keep the budgets but stop them blocking anything — recording only:
```json
@@ -440,17 +401,6 @@ Calls driven by a deployed worker are exempt from enforcement by default,
because a worker has no prompt to show and nobody watching it fail. Set
`"meteringEnforcement": { "workers": true }` to include them.
Monthly usage records expire on their own once they're old enough to be of no
further use — `meteringRetentionMonths` full months past the month they
belong to, 3 by default. Set it to `0` to keep every month's records forever:
```json
"meteringRetentionMonths": 6
```
Changing it, including to `0`, applies to records first written after the
change; records already stamped keep their expiry.
### Captcha on signup / login
Built-in proof-of-work captcha — no external service needed.
@@ -519,127 +469,14 @@ Without `--profile ai`, the `ollama` containers stay down and Puter (with `enabl
For GPU acceleration (NVIDIA), uncomment the `deploy:` block under the `ollama` service in [docker-compose.yml](../docker-compose.yml). Requires `nvidia-container-toolkit` on the host.
## Optional: receiving mail over SMTP
Puter can give each account a mailbox at a domain you control. Two pieces are
involved, and they are configured together under `userEmail`:
- an **ingress endpoint** (`POST /email/ingress`) that files a message into the
recipient's `~/.mail`. It exists whenever `userEmail.secret` is set.
- the **`puter-smtp` receiver**, which accepts mail from other mail servers and
hands each message to that endpoint. It runs as its own process, behind the
`smtp` compose profile.
> **Read this before you expose it.** An inbound mail server is unauthenticated
> by nature: anyone on the internet may hand it a message for any address in
> `localDomains`. Puter performs **no** spam filtering and **no** SPF, DKIM or
> DMARC checks — whatever is delivered is stored in the recipient's mailbox and
> counts against their storage. If that is not acceptable, put a filtering relay
> in front of it and point `localDomains` at that instead.
1. Configure it in `puter/config/config.json`:
```json
"userEmail": {
"secret": "a long random string",
"localServer": true,
"localDomains": ["example.com"]
}
```
`secret` is shared between the two halves; generate it with
`openssl rand -hex 32`. `localDomains` lists the domains this server accepts
mail for — anything else is refused at `RCPT TO`, which is what stops it
being used as an open relay. It has no default, and the receiver refuses to
start without it.
The receiver posts to `<api_base_url>/email/ingress` unless you set
`localIngressUrl`. The endpoint is served on the `api` subdomain, so if you
point it at an internal address to skip the reverse proxy, name that virtual
host too or the request will 404:
```json
"localIngressUrl": "http://puter:4100/email/ingress",
"localIngressHost": "api.example.com"
```
`localIngressHost` defaults to the host of `api_base_url`, which is usually
what you want.
2. Bring it up with the `smtp` profile:
```bash
docker compose --profile smtp up -d
docker compose logs -f puter-smtp
```
3. Point DNS at it:
```dns
mx.example.com. A 203.0.113.10
example.com. MX 10 mx.example.com.
```
Also set a `PTR` record for the address that resolves back to
`mx.example.com` — many senders refuse mail from a server without one.
Inbound TCP port 25 must reach the host. **Most cloud providers block port
25 by default and residential ISPs block it permanently**; if you cannot get
it unblocked, this will not work no matter how it is configured. The
container itself binds 2525 (it runs unprivileged) and compose publishes
`${SMTP_PORT:-25}` onto it, so you can move the host port if you need to.
A message is held in memory while it is being delivered, so peak memory is
bounded by `localMaxClients` (default 20) times the 25 MiB message ceiling.
Lower `localMaxClients` on a small host. There is no queue and no retry
schedule — if delivery fails, the sending server is told to try again later and
retrying is its job.
Mail is stored as `message/rfc822` objects under `~/.mail/objects/<date>/` and
read back through `puter.email` in the SDK.
Every setting is documented in
[config.template.jsonc](../config.template.jsonc). Two deliberate limits, both
in the name of keeping the self-hosted path simple: it does not offer STARTTLS,
so mail arrives in the clear, and it applies no per-sender rate limiting. Put a
filtering relay in front if either matters to you.
Note this is a *receiving* server only: it never sends, and it must not be used
for authenticated submission. Outbound transactional mail is the separate
`email` block above.
## Building from source instead of pulling
To run a local build against the full stack, use a source checkout and complete
the configuration steps above. Create `docker-compose.override.yml` in the
repository root, next to [docker-compose.yml](../docker-compose.yml):
```yaml
services:
puter:
pull_policy: never
build:
context: .
```
If that file already exists, merge these settings into its `puter` service.
Compose loads and merges the override automatically. `build.context: .` selects
this checkout, and `pull_policy: never` prevents pulling the published Puter image.
Use this override for local build settings instead of editing `docker-compose.yml`.
The override is ignored by Git, keeping local configuration out of pull requests
and avoiding conflicts in the shared Compose file when pulling updates.
From the repository root, build and start the stack:
If you want to test local Dockerfile changes against the full stack, uncomment the `build:` block in [docker-compose.yml](../docker-compose.yml) under the `puter` service, change `pull_policy: always` → `pull_policy: never`, then:
```bash
docker compose up -d --build
```
Run the same command after changing or pulling source code to rebuild the image.
To return to the published image, remove the local build settings (or the override
file if those are its only settings), then run `docker compose up -d`.
---
## Managing running backend
+11 -52
View File
@@ -139,10 +139,8 @@ services:
start_period: 5s
s3-init:
# One-shot container that creates the `puter-local` bucket and
# applies its CORS rules on every boot (put-bucket-cors is
# idempotent). Without CORS, browser uploads to presigned URLs
# fail the preflight check.
# One-shot container that creates the `puter-local` bucket on first
# boot. Exits 0 once the bucket exists; stays exited 0 thereafter.
image: amazon/aws-cli:latest
container_name: puter-s3-init
depends_on:
@@ -165,26 +163,6 @@ services:
echo "creating bucket $$bucket"
aws --endpoint-url "$$endpoint" s3 mb "s3://$$bucket"
fi
# Browser uploads (Dev Center deploy, puter.fs.upload) PUT to
# presigned URLs cross-origin, so the bucket must answer the
# OPTIONS preflight. Origins are open because the presigned URL
# itself is the credential and the SDK sends no cookies.
cat > /tmp/cors.json <<'EOF'
{
"CORSRules": [
{
"AllowedOrigins": ["*"],
"AllowedMethods": ["GET", "HEAD", "PUT", "POST", "DELETE"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["ETag", "x-amz-request-id"],
"MaxAgeSeconds": 3600
}
]
}
EOF
aws --endpoint-url "$$endpoint" s3api put-bucket-cors \
--bucket "$$bucket" --cors-configuration file:///tmp/cors.json
echo "bucket CORS configured"
restart: "no"
# ── Optional: local LLM ───────────────────────────────────────────
@@ -252,8 +230,15 @@ services:
puter:
image: ghcr.io/heyputer/puter:main
pull_policy: always
# For local builds, create a docker-compose.override.yml;
# see doc/self-hosting.md#building-from-source-instead-of-pulling.
# Uncomment to build from this directory instead of pulling the published
# image. Also flip pull_policy to `never` so compose doesn't overwrite
# your local build by re-pulling :latest.
# build:
# context: .
# # buildx-only: cross-compile to both archs in a single push
# # platforms:
# # - linux/amd64
# # - linux/arm64
container_name: puter
restart: unless-stopped
depends_on:
@@ -286,32 +271,6 @@ services:
retries: 3
start_period: 30s
# Inbound mail. Behind the `smtp` profile, so it only starts when asked for:
# docker compose --profile smtp up -d
# Accepts SMTP from other mail servers and hands each message to the ingress
# endpoint, which files it into the recipient's mailbox. Needs `userEmail`
# configured — see doc/self-hosting.md#optional-receiving-mail-over-smtp.
puter-smtp:
profiles: ["smtp"]
image: ghcr.io/heyputer/puter:main
pull_policy: always
container_name: puter-smtp
restart: unless-stopped
command: ["node", "-r", "./dist/src/backend/telemetry.js", "./dist/src/backend/smtp/index.js"]
depends_on:
puter:
condition: service_healthy
ports:
# The container runs unprivileged and binds 2525; 25 is what an MX record
# means, and many providers block it.
- "${SMTP_PORT:-25}:2525"
environment:
PUID: 1000
PGID: 1000
volumes:
# The same config.json the backend reads; only `userEmail` applies here.
- ./puter/config:/etc/puter:z
caddy:
image: caddy:2.11-alpine
container_name: puter-caddy
+1 -40
View File
@@ -9,7 +9,6 @@ import {
vi,
} from 'vitest';
import { runWithContext } from '../src/backend/core/context.ts';
import { extension } from '../src/backend/extensions.ts';
import { PuterServer } from '../src/backend/server.ts';
import { setupTestServer } from '../src/backend/testUtil.ts';
import { handleInstalledApps } from './installedApps.ts';
@@ -52,14 +51,13 @@ const seedUser = async () => {
});
};
const seedApp = async (ownerUserId: number, overrides = {}) => {
const seedApp = async (ownerUserId: number) => {
const slug = Math.random().toString(36).slice(2, 8);
return server.stores.app.create(
{
name: `iaapp_${slug}`,
title: `Installed App ${slug}`,
index_url: `https://example.com/${slug}`,
...overrides,
},
{ ownerUserId },
);
@@ -147,43 +145,6 @@ describe('installedApps extension — handleInstalledApps', () => {
).toBe(false);
});
it('pairs iconUrl with the direct subdomain URL for a generated icon', async () => {
const user = await seedUser();
// The resize pipeline leaves an http(s) icon column behind, which is
// what says the sized PNG is on the icons subdomain.
const app = await seedApp(user.id as number, {
icon: 'https://api.puter.com/app-icon/x',
});
await grantInstalled(app!.id as number, user.id as number);
const { res, captured } = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleInstalledApps(makeReq({}), res),
);
const list = captured.body as Array<Record<string, unknown>>;
const { protocol, static_hosting_domain } = extension.config;
expect(list[0].iconCdnUrl).toBe(
`${protocol}://puter-app-icons.${static_hosting_domain}/${app!.uid}-256.png`,
);
});
it('reports no subdomain URL for an app with no generated icon', async () => {
const user = await seedUser();
const app = await seedApp(user.id as number);
await grantInstalled(app!.id as number, user.id as number);
const { res, captured } = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleInstalledApps(makeReq({}), res),
);
const list = captured.body as Array<Record<string, unknown>>;
expect(list[0].iconCdnUrl).toBeNull();
});
it('flags apps with no owner_user_id as external', async () => {
const user = await seedUser();
const slug = Math.random().toString(36).slice(2, 8);
+1 -6
View File
@@ -2,10 +2,7 @@ import type { Request, Response } from 'express';
import { Context } from '@heyputer/backend/src/core';
import { HttpError } from '@heyputer/backend/src/core/http';
import { extension } from '@heyputer/backend/src/extensions';
import {
getAppIconCdnUrl,
getAppIconUrl,
} from '@heyputer/backend/src/util/appIcon.js';
import { getAppIconUrl } from '@heyputer/backend/src/util/appIcon.js';
const clients = extension.import('client');
@@ -76,8 +73,6 @@ export const handleInstalledApps = async (
return {
...rest,
iconUrl: getAppIconUrl(app, { apiBaseUrl }),
// Direct subdomain URL for the client to try before iconUrl.
iconCdnUrl: getAppIconCdnUrl(app, extension.config),
external,
};
}),
-165
View File
@@ -8,7 +8,6 @@ import {
it,
vi,
} from 'vitest';
import { makeActor, type Actor } from '../src/backend/core/actor.ts';
import { runWithContext } from '../src/backend/core/context.ts';
import { PuterServer } from '../src/backend/server.ts';
import { setupTestServer } from '../src/backend/testUtil.ts';
@@ -89,80 +88,6 @@ describe('metering extension — handleMeteringUsage', () => {
(captured.body as Record<string, unknown>).allowanceInfo,
).toBeDefined();
});
describe('usage split across apps', () => {
const seedUsage = async () => {
const user = await seedUser();
const owner = { uuid: user.uuid, id: user.id as number };
const metering = server.services.metering;
await metering.incrementUsage(
makeActor({ user: owner, app: { uid: 'app-mine' } }),
'kv:read',
1,
10,
);
await metering.incrementUsage(
makeActor({ user: owner, app: { uid: 'app-other' } }),
'kv:write',
1,
20,
);
await metering.incrementUsage(
makeActor({ user: owner }),
'kv:list',
1,
40,
);
await server.stores.meteringBuffer.flushCycle();
return owner;
};
const usageAs = async (actor: Actor) => {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleMeteringUsage(makeReq(), res),
);
return captured.body as {
usage: Record<string, unknown> & { total: number };
appTotals: Record<string, unknown>;
allowanceInfo: Record<string, unknown>;
};
};
it('reports only the calling app’s own usage to an app actor', async () => {
const owner = await seedUsage();
const body = await usageAs(
makeActor({ user: owner, app: { uid: 'app-mine' } }),
);
expect(body.usage.total).toBe(10);
expect(Object.keys(body.usage).sort()).toEqual([
'kv:read',
'total',
]);
expect(body.appTotals).toEqual({
'app-mine': { total: 10, count: 1 },
});
expect(body.allowanceInfo).toMatchObject({
remaining: expect.any(Number),
monthUsageAllowance: expect.any(Number),
});
});
it('still reports the whole account to the user', async () => {
const owner = await seedUsage();
const body = await usageAs(makeActor({ user: owner }));
expect(body.usage.total).toBe(70);
expect(body.usage).toHaveProperty('kv:read');
expect(body.usage).toHaveProperty('kv:write');
expect(body.usage).toHaveProperty('kv:list');
expect(body.appTotals).toMatchObject({
'app-mine': { total: 10, count: 1 },
'app-other': { total: 20, count: 1 },
});
});
});
});
describe('metering extension — handleMeteringUsageForApp', () => {
@@ -207,96 +132,6 @@ describe('metering extension — handleMeteringUsageForApp', () => {
),
).rejects.toMatchObject({ statusCode: 404 });
});
describe('scoped to the calling app', () => {
const appUsageAs = async (actor: Actor, appIdOrName: string) => {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleMeteringUsageForApp(
makeReq({ params: { appIdOrName } }),
res,
),
);
return captured.body as { total: number };
};
const seedApps = async () => {
const user = await seedUser();
const owner = { uuid: user.uuid, id: user.id as number };
const mine = `app-${uuidv4()}`;
const other = `app-${uuidv4()}`;
await server.services.metering.incrementUsage(
makeActor({ user: owner, app: { uid: mine } }),
'kv:read',
1,
10,
);
await server.services.metering.incrementUsage(
makeActor({ user: owner, app: { uid: other } }),
'kv:write',
1,
20,
);
await server.stores.meteringBuffer.flushCycle();
return {
owner,
mine,
other,
asMine: makeActor({ user: owner, app: { uid: mine } }),
};
};
it('lets an app read its own usage', async () => {
const { mine, asMine } = await seedApps();
expect((await appUsageAs(asMine, mine)).total).toBe(10);
});
it('refuses an app another app’s usage', async () => {
const { other, asMine } = await seedApps();
await expect(appUsageAs(asMine, other)).rejects.toMatchObject({
statusCode: 403,
});
});
it('refuses an app another app’s usage by name', async () => {
const { owner, other, asMine } = await seedApps();
const name = `other-${uuidv4()}`;
await server.clients.db.write(
'INSERT INTO `apps` (`uid`, `name`, `title`, `index_url`, `owner_user_id`) VALUES (?, ?, ?, ?, ?)',
[other, name, name, `https://${name}.example/`, owner.id],
);
await expect(appUsageAs(asMine, name)).rejects.toMatchObject({
statusCode: 403,
});
});
it('refuses an app the usage outside any app', async () => {
const { asMine } = await seedApps();
await expect(appUsageAs(asMine, 'os-global')).rejects.toMatchObject(
{ statusCode: 403 },
);
});
it('still lets the user read any of their apps', async () => {
const { owner, mine, other } = await seedApps();
const asUser = makeActor({ user: owner });
expect((await appUsageAs(asUser, mine)).total).toBe(10);
expect((await appUsageAs(asUser, other)).total).toBe(20);
});
it('lets the user read their usage outside any app', async () => {
const { owner } = await seedApps();
const asUser = makeActor({ user: owner });
await server.services.metering.incrementUsage(
asUser,
'kv:read',
1,
5,
);
await server.stores.meteringBuffer.flushCycle();
expect((await appUsageAs(asUser, 'os-global')).total).toBe(5);
});
});
});
describe('metering extension — handleMeteringGlobalUsage', () => {
+5 -111
View File
@@ -6,11 +6,6 @@ import {
servicesContainers,
} from '@heyputer/backend/src/exports';
import { extension } from '@heyputer/backend/src/extensions';
import { GLOBAL_APP_KEY } from '@heyputer/backend/src/services/metering/consts';
import {
creditMultiplierFrom,
toCredits,
} from '@heyputer/backend/src/services/metering/utils';
import type { Request, Response } from 'express';
const services = extension.import('service');
@@ -55,42 +50,6 @@ function collectAllCosts(): Record<string, unknown>[] {
return all;
}
// -- Credits scaling ---------------------------------------------------
//
// When the deployment configures a credit multiplier, every monetary amount
// is scaled to display credits BEFORE it leaves the server — raw metered
// amounts are never reported. Counts, units, and byte figures pass through
// untouched. A response that was scaled says so (`unit: 'credits'`); without
// a configured multiplier amounts pass through raw and clients render them
// as dollars.
/** The per-usage-type records of a UsageByType map, costs scaled. */
const scaleUsageByType = (
usage: Record<string, unknown>,
multiplier: number,
): Record<string, unknown> => {
const out: Record<string, unknown> = {};
for (const [key, value] of Object.entries(usage)) {
// `allowanceUsed` is the month's allowance-charged spend — monetary,
// scalar, scaled like the total.
if (key === 'total' || key === 'allowanceUsed') {
out[key] = toCredits(Number(value) || 0, multiplier);
} else if (
value &&
typeof value === 'object' &&
typeof (value as { cost?: unknown }).cost === 'number'
) {
out[key] = {
...(value as Record<string, unknown>),
cost: toCredits((value as { cost: number }).cost, multiplier),
};
} else {
out[key] = value;
}
}
return out;
};
export const handleMeteringUsage = async (
_req: Request,
res: Response,
@@ -102,44 +61,7 @@ export const handleMeteringUsage = async (
services.metering.getActorCurrentMonthUsageDetails(actor),
services.metering.getAllowedUsage(actor),
]);
const multiplier = creditMultiplierFrom(extension.config);
if (!multiplier) {
res.json({ ...actorUsage, allowanceInfo });
return;
}
res.json({
usage: scaleUsageByType(
actorUsage.usage as unknown as Record<string, unknown>,
multiplier,
),
appTotals: Object.fromEntries(
Object.entries(actorUsage.appTotals).map(([appId, totals]) => [
appId,
{ ...totals, total: toCredits(totals.total, multiplier) },
]),
),
allowanceInfo: {
...allowanceInfo,
remaining: toCredits(allowanceInfo.remaining, multiplier),
monthUsageAllowance: toCredits(
allowanceInfo.monthUsageAllowance,
multiplier,
),
// Monetary addon fields scale; absent ones stay absent rather
// than materializing as NaN/null. purchasedStorage is bytes.
addons: Object.fromEntries(
Object.entries(allowanceInfo.addons ?? {}).map(([k, v]) => [
k,
(k === 'purchasedCredits' ||
k === 'consumedPurchaseCredits') &&
typeof v === 'number'
? toCredits(v, multiplier)
: v,
]),
),
unit: 'credits',
},
});
res.json({ ...actorUsage, allowanceInfo });
};
export const handleMeteringUsageForApp = async (
@@ -152,9 +74,8 @@ export const handleMeteringUsageForApp = async (
let appId = String(req.params.appIdOrName ?? '');
if (!appId) throw new HttpError(400, 'appId parameter is required');
// If not a UUID-shaped app UID or the global sentinel, look up by name.
// Which apps an actor may read is MeteringService's call, not this route's.
if (!appId.startsWith('app-') && appId !== GLOBAL_APP_KEY) {
// If not a UUID-shaped app UID, look up by name
if (!appId.startsWith('app-')) {
const appRows = (await clients.db.read(
'SELECT `uid` FROM `apps` WHERE `name` = ? LIMIT 1',
[appId],
@@ -171,18 +92,7 @@ export const handleMeteringUsageForApp = async (
actor,
appId,
);
const multiplier = creditMultiplierFrom(extension.config);
if (!multiplier) {
res.json(appUsage);
return;
}
res.json({
...scaleUsageByType(
appUsage as unknown as Record<string, unknown>,
multiplier,
),
unit: 'credits',
});
res.json(appUsage);
};
export const handleMeteringGlobalUsage = async (
@@ -199,23 +109,7 @@ export const handleMeteringAllCosts = async (
res: Response,
): Promise<void> => {
if (!cachedAllCosts) {
const raw = collectAllCosts();
const multiplier = creditMultiplierFrom(extension.config);
// Price the catalogue in the same unit the usage endpoints report:
// per-unit credits when a multiplier is configured, raw otherwise.
cachedAllCosts = multiplier
? raw.map(({ ucentsPerUnit, costs_currency: _cc, ...rest }) =>
typeof ucentsPerUnit === 'number'
? {
...rest,
creditsPerUnit: toCredits(
ucentsPerUnit,
multiplier,
),
}
: rest,
)
: raw;
cachedAllCosts = collectAllCosts();
}
res.json({ costs: cachedAllCosts });
};
+173 -496
View File
@@ -1,10 +1,8 @@
import {
GetObjectCommand,
HeadObjectCommand,
PutObjectCommand,
type S3Client,
} from '@aws-sdk/client-s3';
import type { Request, Response } from 'express';
import crypto from 'node:crypto';
import {
afterAll,
@@ -14,17 +12,8 @@ import {
it,
vi,
} from 'vitest';
import type { FSController } from '../src/backend/controllers/fs/FSController.ts';
import type {
ClientSignedWriteResponse,
CompleteWriteRequest,
SignedWriteRequest,
} from '../src/backend/controllers/fs/requestTypes.ts';
import type { Actor } from '../src/backend/core/actor.ts';
import { runWithContext } from '../src/backend/core/context.ts';
import { PuterServer } from '../src/backend/server.ts';
import { setupTestServer } from '../src/backend/testUtil.ts';
import { generateDefaultFsentries } from '../src/backend/util/userProvisioning.ts';
import {
handleFsCopyNodeThumbnail,
handleFsRemoveNodeThumbnail,
@@ -38,33 +27,9 @@ const TINY_PNG_BASE64 =
'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII=';
const BUCKET = 'puter-local';
const BUCKET_ENDPOINT = 'http://127.0.0.1:4566/puter-local/';
// Keys the extension mints: `thumbnails/<entry uuid>/<uuid>`.
const mintedKey = (entryUuid: string = crypto.randomUUID()) =>
`thumbnails/${entryUuid}/${crypto.randomUUID()}`;
// Minted before keys were bound to an entry.
const legacyKey = () => `thumbnails/${crypto.randomUUID()}`;
const putObject = async (s3: S3Client, key: string, body: Buffer) => {
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: key,
Body: body,
ContentType: 'image/png',
}),
);
};
const objectExists = async (s3: S3Client, key: string) => {
try {
await s3.send(new HeadObjectCommand({ Bucket: BUCKET, Key: key }));
return true;
} catch {
return false;
}
};
// Keys the extension will accept back: its own `thumbnails/<uuid>` namespace.
const mintedKey = () => `thumbnails/${crypto.randomUUID()}`;
const streamToBuffer = async (
body: { transformToByteArray: () => Promise<Uint8Array> } | undefined,
@@ -75,42 +40,28 @@ const streamToBuffer = async (
describe('thumbnails extension — handleThumbnailCreated', () => {
let server: PuterServer;
let s3: S3Client;
beforeAll(async () => {
server = await setupTestServer();
s3 = server.clients.s3.get();
});
afterAll(async () => {
await server?.shutdown();
});
const created = async (url: unknown, uuid?: unknown) => {
const event: Record<string, unknown> = { url, uuid };
await handleThumbnailCreated(event, {
s3,
bucketName: BUCKET,
bucketEndpoint: BUCKET_ENDPOINT,
});
return event.url;
};
it('uploads a valid data: URL thumbnail to S3 and rewrites event.url to an s3:// pointer', async () => {
const s3 = server.clients.s3.get();
const event: Record<string, unknown> = {
url: `data:image/png;base64,${TINY_PNG_BASE64}`,
};
it('uploads a valid data: URL thumbnail under a key bound to the entry', async () => {
const entryUuid = crypto.randomUUID();
const newUrl = await created(
`data:image/png;base64,${TINY_PNG_BASE64}`,
entryUuid,
);
await handleThumbnailCreated(event, { s3, bucketName: BUCKET });
expect(typeof newUrl).toBe('string');
expect(
(newUrl as string).startsWith(
`s3://${BUCKET}/thumbnails/${entryUuid}/`,
),
).toBe(true);
expect(typeof event.url).toBe('string');
const newUrl = event.url as string;
expect(newUrl.startsWith(`s3://${BUCKET}/`)).toBe(true);
const key = (newUrl as string).slice(`s3://${BUCKET}/`.length);
const key = newUrl.slice(`s3://${BUCKET}/`.length);
const obj = await s3.send(
new GetObjectCommand({ Bucket: BUCKET, Key: key }),
);
@@ -122,77 +73,34 @@ describe('thumbnails extension — handleThumbnailCreated', () => {
});
it('sets event.url to null when the data: URL does not decode to a valid image', async () => {
const url = await created(
`data:image/png;base64,${Buffer.from('not an image').toString('base64')}`,
crypto.randomUUID(),
);
expect(url).toBeNull();
const s3 = server.clients.s3.get();
const event: Record<string, unknown> = {
url: `data:image/png;base64,${Buffer.from('not an image').toString('base64')}`,
};
await handleThumbnailCreated(event, { s3, bucketName: BUCKET });
expect(event.url).toBeNull();
});
it('sets event.url to null for a data: URL with no entry to bind it to', async () => {
const url = await created(`data:image/png;base64,${TINY_PNG_BASE64}`);
expect(url).toBeNull();
});
// A documented SDK input: `thumbnail` / `thumbnailGenerator` may return
// a plain URL. It never names our storage, so it is stored as given.
it('leaves an external image URL untouched', async () => {
it('leaves event.url untouched when the URL is not a data: URL', async () => {
const s3 = server.clients.s3.get();
const original = 'https://example.com/thumb.png';
expect(await created(original, crypto.randomUUID())).toBe(original);
const event: Record<string, unknown> = { url: original };
await handleThumbnailCreated(event, { s3, bucketName: BUCKET });
expect(event.url).toBe(original);
});
it('returns without writing to S3 when event.url is missing', async () => {
const s3 = server.clients.s3.get();
const event: Record<string, unknown> = {};
await handleThumbnailCreated(event, {
s3,
bucketName: BUCKET,
bucketEndpoint: BUCKET_ENDPOINT,
});
await handleThumbnailCreated(event, { s3, bucketName: BUCKET });
expect(event.url).toBeUndefined();
});
it('keeps an uploaded pointer minted for the entry, normalised to our bucket', async () => {
const entryUuid = crypto.randomUUID();
const key = mintedKey(entryUuid);
await putObject(s3, key, Buffer.from(TINY_PNG_BASE64, 'base64'));
expect(
await created(`s3://client-named-bucket/${key}`, entryUuid),
).toBe(`s3://${BUCKET}/${key}`);
});
// A thumbnail key is visible to anyone who can list the entry holding it.
// Planting it on a file of one's own used to hand its deletion to them.
it.each([
['a key minted for another entry', () => mintedKey()],
['a key that predates entry binding', legacyKey],
])('drops a pointer naming %s', async (_label, makeKey) => {
const key = makeKey();
await putObject(s3, key, Buffer.from(TINY_PNG_BASE64, 'base64'));
expect(
await created(`s3://${BUCKET}/${key}`, crypto.randomUUID()),
).toBeNull();
expect(await objectExists(s3, key)).toBe(true);
});
it('drops a pointer whose object was never uploaded', async () => {
const entryUuid = crypto.randomUUID();
expect(
await created(`s3://${BUCKET}/${mintedKey(entryUuid)}`, entryUuid),
).toBeNull();
});
it('drops and removes an uploaded object over the size bound', async () => {
const entryUuid = crypto.randomUUID();
const key = mintedKey(entryUuid);
await putObject(s3, key, Buffer.alloc(2 * 1024 * 1024 + 1));
expect(await created(`s3://${BUCKET}/${key}`, entryUuid)).toBeNull();
expect(await objectExists(s3, key)).toBe(false);
});
});
describe('thumbnails extension — handleThumbnailUploadPrepare', () => {
@@ -240,15 +148,10 @@ describe('thumbnails extension — handleThumbnailUploadPrepare', () => {
expect(item.thumbnailUrl).toBeUndefined();
});
it.each([
['exceeds the max thumbnail bytes', { size: 999_999_999 }],
['is not declared', {}],
['is not a whole number', { size: 10.5 }],
])('skips items whose size %s', async (_label, sizeField) => {
it('skips items whose size exceeds the max thumbnail bytes', async () => {
const item: Record<string, unknown> = {
contentType: 'image/png',
item_uid: crypto.randomUUID(),
...sizeField,
size: 999_999_999,
};
await handleThumbnailUploadPrepare(
{ items: [item] },
@@ -258,26 +161,10 @@ describe('thumbnails extension — handleThumbnailUploadPrepare', () => {
expect(item.thumbnailUrl).toBeUndefined();
});
it('skips items with no entry to bind the key to', async () => {
it('mints a presigned uploadUrl and an s3:// thumbnailUrl for valid items', async () => {
const item: Record<string, unknown> = {
contentType: 'image/png',
size: 1024,
item_uid: 'not-a-uuid',
};
await handleThumbnailUploadPrepare(
{ items: [item] },
{ s3Presign, bucketName: BUCKET },
);
expect(item.uploadUrl).toBeUndefined();
expect(item.thumbnailUrl).toBeUndefined();
});
it('mints a size-bound upload URL for a key bound to the entry', async () => {
const entryUuid = crypto.randomUUID();
const item: Record<string, unknown> = {
contentType: 'image/png',
size: 1024,
item_uid: entryUuid,
};
await handleThumbnailUploadPrepare(
{ items: [item] },
@@ -287,15 +174,8 @@ describe('thumbnails extension — handleThumbnailUploadPrepare', () => {
expect((item.uploadUrl as string).startsWith('http')).toBe(true);
expect(typeof item.thumbnailUrl).toBe('string');
expect(
(item.thumbnailUrl as string).startsWith(
`s3://${BUCKET}/thumbnails/${entryUuid}/`,
),
(item.thumbnailUrl as string).startsWith(`s3://${BUCKET}/`),
).toBe(true);
// A signed `content-length` makes the store refuse any other body size.
const signedHeaders = new URL(
item.uploadUrl as string,
).searchParams.get('X-Amz-SignedHeaders');
expect(signedHeaders).toContain('content-length');
});
});
@@ -316,41 +196,34 @@ describe('thumbnails extension — handleThumbnailRead', () => {
await server?.shutdown();
});
// Thumbnails stored before keys were bound to entries keep displaying.
it.each([
['an entry-bound key', () => mintedKey()],
['a key that predates entry binding', legacyKey],
])(
'rewrites an s3:// thumbnail naming %s into a presigned https URL',
async (_label, makeKey) => {
// Seed an object so the presigned URL points at something real
// (the signer itself doesn't validate existence, but this keeps
// the test honest).
const key = makeKey();
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: key,
Body: Buffer.from(TINY_PNG_BASE64, 'base64'),
ContentType: 'image/png',
}),
);
it('rewrites an s3:// thumbnail into a presigned https URL', async () => {
// Seed an object so the presigned URL points at something real
// (the signer itself doesn't validate existence, but this keeps
// the test honest).
const key = mintedKey();
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: key,
Body: Buffer.from(TINY_PNG_BASE64, 'base64'),
ContentType: 'image/png',
}),
);
const entry: Record<string, unknown> = {
thumbnail: `s3://${BUCKET}/${key}`,
};
await handleThumbnailRead(entry, {
s3,
s3Presign,
bucketName: BUCKET,
bucketEndpoint: 'http://127.0.0.1:4566/puter-local/',
db: stubDb,
});
const entry: Record<string, unknown> = {
thumbnail: `s3://${BUCKET}/${key}`,
};
await handleThumbnailRead(entry, {
s3,
s3Presign,
bucketName: BUCKET,
bucketEndpoint: 'http://127.0.0.1:4566/puter-local/',
db: stubDb,
});
expect(typeof entry.thumbnail).toBe('string');
expect((entry.thumbnail as string).startsWith('http')).toBe(true);
},
);
expect(typeof entry.thumbnail).toBe('string');
expect((entry.thumbnail as string).startsWith('http')).toBe(true);
});
// `fsentries.thumbnail` is writable through the FS API, so a stored
// pointer is attacker input. Signing one the extension didn't mint would
@@ -372,10 +245,6 @@ describe('thumbnails extension — handleThumbnailRead', () => {
`s3://${BUCKET}/thumbnails/../${crypto.randomUUID()}`,
],
['a non-uuid inside the namespace', `s3://${BUCKET}/thumbnails/etc`],
[
'a key nested deeper than the namespace allows',
`s3://${BUCKET}/${mintedKey()}/${crypto.randomUUID()}`,
],
])('refuses to presign a pointer naming %s', async (_label, thumbnail) => {
const entry: Record<string, unknown> = { thumbnail };
await handleThumbnailRead(entry, {
@@ -432,9 +301,8 @@ describe('thumbnails extension — handleThumbnailRead', () => {
});
it('migrates an inline data: URL by uploading to S3 and updating the DB row', async () => {
const entryUuid = crypto.randomUUID();
const entry: Record<string, unknown> = {
uuid: entryUuid,
uuid: 'fs-entry-uuid',
thumbnail: `data:image/png;base64,${TINY_PNG_BASE64}`,
};
@@ -454,27 +322,9 @@ describe('thumbnails extension — handleThumbnailRead', () => {
await Promise.resolve();
expect(stubDb.write).toHaveBeenCalledWith(
'UPDATE `fsentries` SET `thumbnail` = ? WHERE `uuid` = ?',
[
expect.stringMatching(
new RegExp(`^s3://${BUCKET}/thumbnails/${entryUuid}/`),
),
entryUuid,
],
[expect.stringMatching(/^s3:\/\//), 'fs-entry-uuid'],
);
});
it('leaves an inline data: URL as is when there is no entry to bind it to', async () => {
const thumbnail = `data:image/png;base64,${TINY_PNG_BASE64}`;
const entry: Record<string, unknown> = { thumbnail };
await handleThumbnailRead(entry, {
s3,
s3Presign,
bucketName: BUCKET,
bucketEndpoint: BUCKET_ENDPOINT,
db: stubDb,
});
expect(entry.thumbnail).toBe(thumbnail);
});
});
describe('thumbnails extension — handleFsRemoveNodeThumbnail', () => {
@@ -490,63 +340,66 @@ describe('thumbnails extension — handleFsRemoveNodeThumbnail', () => {
await server?.shutdown();
});
const remove = (target: { thumbnail?: string | null; uuid?: string }) =>
handleFsRemoveNodeThumbnail(
{ target },
{ s3, bucketName: BUCKET, bucketEndpoint: BUCKET_ENDPOINT },
it('deletes the S3 object referenced by an s3:// thumbnail URL', async () => {
const key = mintedKey();
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: key,
Body: Buffer.from(TINY_PNG_BASE64, 'base64'),
ContentType: 'image/png',
}),
);
it('deletes the object minted for the removed entry', async () => {
const entryUuid = crypto.randomUUID();
const key = mintedKey(entryUuid);
await putObject(s3, key, Buffer.from(TINY_PNG_BASE64, 'base64'));
await handleFsRemoveNodeThumbnail(
{ target: { thumbnail: `s3://${BUCKET}/${key}` } },
{ s3, bucketName: BUCKET },
);
await remove({ thumbnail: `s3://${BUCKET}/${key}`, uuid: entryUuid });
expect(await objectExists(s3, key)).toBe(false);
// GetObject should now error because the key was deleted.
await expect(
s3.send(new GetObjectCommand({ Bucket: BUCKET, Key: key })),
).rejects.toThrow();
});
// The destructive half of the same confused deputy: the stored pointer
// decides which object is deleted, so only a key minted for the removed
// entry may reach DeleteObject.
it.each([
// Shaped like an fs object key.
[
'an object outside the thumbnails namespace',
() => crypto.randomUUID(),
],
// Visible to anyone who can list the entry it belongs to.
["another entry's thumbnail", () => mintedKey()],
// Can't be told apart from a planted or shared pointer.
['a thumbnail that predates entry binding', legacyKey],
])('does not delete %s', async (_label, makeKey) => {
const victimKey = makeKey();
await putObject(s3, victimKey, Buffer.from(TINY_PNG_BASE64, 'base64'));
// decides which object is deleted, so a key the extension didn't mint
// must never reach DeleteObject.
it('does not delete an object the pointer names but we did not mint', async () => {
const victimKey = crypto.randomUUID(); // shaped like an fs object key
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: victimKey,
Body: Buffer.from(TINY_PNG_BASE64, 'base64'),
ContentType: 'image/png',
}),
);
await remove({
thumbnail: `s3://${BUCKET}/${victimKey}`,
uuid: crypto.randomUUID(),
});
await handleFsRemoveNodeThumbnail(
{ target: { thumbnail: `s3://${BUCKET}/${victimKey}` } },
{ s3, bucketName: BUCKET },
);
expect(await objectExists(s3, victimKey)).toBe(true);
});
it('does not delete through an external URL that names our key', async () => {
const entryUuid = crypto.randomUUID();
const key = mintedKey(entryUuid);
await putObject(s3, key, Buffer.from(TINY_PNG_BASE64, 'base64'));
await remove({
thumbnail: `https://cdn.example.com/${BUCKET}/${key}`,
uuid: entryUuid,
});
expect(await objectExists(s3, key)).toBe(true);
const survivor = await s3.send(
new GetObjectCommand({ Bucket: BUCKET, Key: victimKey }),
);
expect(survivor.ContentType).toBe('image/png');
});
it('is a no-op when the target has no thumbnail', async () => {
// Should not throw or attempt a delete.
await remove({});
await handleFsRemoveNodeThumbnail(
{ target: {} },
{ s3, bucketName: BUCKET },
);
});
it('is a no-op when the thumbnail URL is not an s3:// pointer', async () => {
await handleFsRemoveNodeThumbnail(
{ target: { thumbnail: 'https://cdn.example.com/x.png' } },
{ s3, bucketName: BUCKET },
);
});
});
@@ -563,58 +416,64 @@ describe('thumbnails extension — handleFsCopyNodeThumbnail', () => {
await server?.shutdown();
});
const copyNode = async (thumbnail: string | null, copyUuid: string) => {
it('duplicates the shared thumbnail object and repoints the copied row', async () => {
const sourceKey = mintedKey();
const body = Buffer.from(TINY_PNG_BASE64, 'base64');
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: sourceKey,
Body: body,
ContentType: 'image/png',
}),
);
const copyUuid = crypto.randomUUID();
const db = { write: vi.fn().mockResolvedValue(undefined) };
await handleFsCopyNodeThumbnail(
{ copy: { thumbnail, uuid: copyUuid } },
{ s3, bucketName: BUCKET, bucketEndpoint: BUCKET_ENDPOINT, db },
{
copy: {
thumbnail: `s3://${BUCKET}/${sourceKey}`,
uuid: copyUuid,
},
},
{ s3, bucketName: BUCKET, db },
);
return db;
};
it.each([
['an entry-bound key', () => mintedKey()],
['a key that predates entry binding', legacyKey],
])(
'duplicates a thumbnail naming %s under a key bound to the copy',
async (_label, makeKey) => {
const sourceKey = makeKey();
const body = Buffer.from(TINY_PNG_BASE64, 'base64');
await putObject(s3, sourceKey, body);
// The copied row was repointed at a fresh object...
expect(db.write).toHaveBeenCalledTimes(1);
const [, params] = db.write.mock.calls[0] as [string, [string, string]];
const [newPointer, updatedUuid] = params;
expect(updatedUuid).toBe(copyUuid);
expect(newPointer.startsWith(`s3://${BUCKET}/thumbnails/`)).toBe(true);
expect(newPointer).not.toBe(`s3://${BUCKET}/${sourceKey}`);
const copyUuid = crypto.randomUUID();
const db = await copyNode(`s3://${BUCKET}/${sourceKey}`, copyUuid);
// The copied row was repointed at a fresh object of its own...
expect(db.write).toHaveBeenCalledTimes(1);
const [, params] = db.write.mock.calls[0] as [
string,
[string, string],
];
const [newPointer, updatedUuid] = params;
expect(updatedUuid).toBe(copyUuid);
expect(
newPointer.startsWith(`s3://${BUCKET}/thumbnails/${copyUuid}/`),
).toBe(true);
// ...whose content matches, while the source object survives — so
// deleting either entry can no longer break the other's thumbnail.
const newKey = newPointer.slice(`s3://${BUCKET}/`.length);
const duplicated = await s3.send(
new GetObjectCommand({ Bucket: BUCKET, Key: newKey }),
);
expect(
(await streamToBuffer(duplicated.Body as never)).equals(body),
).toBe(true);
expect(await objectExists(s3, sourceKey)).toBe(true);
},
);
// ...whose content matches, while the source object survives — so
// deleting either entry can no longer break the other's thumbnail.
const newKey = newPointer.slice(`s3://${BUCKET}/`.length);
const duplicated = await s3.send(
new GetObjectCommand({ Bucket: BUCKET, Key: newKey }),
);
expect(
(await streamToBuffer(duplicated.Body as never)).equals(body),
).toBe(true);
const original = await s3.send(
new GetObjectCommand({ Bucket: BUCKET, Key: sourceKey }),
);
expect(original.ContentType).toBe('image/png');
});
it('drops the pointer when the shared object is already gone', async () => {
const copyUuid = crypto.randomUUID();
const db = await copyNode(
`s3://${BUCKET}/${mintedKey()}`, // never uploaded
copyUuid,
const db = { write: vi.fn().mockResolvedValue(undefined) };
await handleFsCopyNodeThumbnail(
{
copy: {
thumbnail: `s3://${BUCKET}/${mintedKey()}`, // never uploaded
uuid: copyUuid,
},
},
{ s3, bucketName: BUCKET, db },
);
expect(db.write).toHaveBeenCalledTimes(1);
@@ -623,209 +482,27 @@ describe('thumbnails extension — handleFsCopyNodeThumbnail', () => {
expect(params).toEqual([copyUuid]);
});
// Only reachable for objects uploaded before upload URLs were size-bound;
// copying one would multiply bytes no allowance counts.
it('drops the pointer instead of duplicating an object over the size bound', async () => {
const sourceKey = legacyKey();
await putObject(s3, sourceKey, Buffer.alloc(2 * 1024 * 1024 + 1));
const copyUuid = crypto.randomUUID();
const db = await copyNode(`s3://${BUCKET}/${sourceKey}`, copyUuid);
expect(db.write).toHaveBeenCalledTimes(1);
const [sql, params] = db.write.mock.calls[0] as [string, [string]];
expect(sql).toContain('NULL');
expect(params).toEqual([copyUuid]);
});
it('does not duplicate an object the pointer names but we did not mint', async () => {
const foreignKey = crypto.randomUUID(); // shaped like an fs object key
const db = await copyNode(
`s3://${BUCKET}/${foreignKey}`,
crypto.randomUUID(),
);
expect(db.write).not.toHaveBeenCalled();
});
it('leaves an external image URL to ride along as is', async () => {
const sourceKey = legacyKey();
await putObject(s3, sourceKey, Buffer.from(TINY_PNG_BASE64, 'base64'));
const db = await copyNode(
`https://cdn.example.com/${BUCKET}/${sourceKey}`,
crypto.randomUUID(),
const db = { write: vi.fn().mockResolvedValue(undefined) };
await handleFsCopyNodeThumbnail(
{
copy: {
thumbnail: `s3://${BUCKET}/${foreignKey}`,
uuid: crypto.randomUUID(),
},
},
{ s3, bucketName: BUCKET, db },
);
expect(db.write).not.toHaveBeenCalled();
});
it('is a no-op when the copy has no thumbnail', async () => {
const db = await copyNode(null, crypto.randomUUID());
const db = { write: vi.fn().mockResolvedValue(undefined) };
await handleFsCopyNodeThumbnail(
{ copy: { thumbnail: null, uuid: crypto.randomUUID() } },
{ s3, bucketName: BUCKET, db },
);
expect(db.write).not.toHaveBeenCalled();
});
});
// The flow `puter.fs.upload()` runs — start, PUT to the signed URLs, complete
// with the returned pointer — through the real controller and the listeners
// this extension registers.
describe('thumbnails extension — signed batch upload through /fs', () => {
let server: PuterServer;
let controller: FSController;
const png = Buffer.from(TINY_PNG_BASE64, 'base64');
beforeAll(async () => {
server = await setupTestServer();
controller = server.controllers.fs as unknown as FSController;
});
afterAll(async () => {
await server?.shutdown();
});
const makeActor = async (): Promise<{ actor: Actor; username: string }> => {
const username = `thumb-${Math.random().toString(36).slice(2, 10)}`;
const user = await server.stores.user.create({
username,
uuid: crypto.randomUUID(),
password: null,
email: `${username}@test.local`,
free_storage: 100 * 1024 * 1024,
requires_email_confirmation: false,
});
await generateDefaultFsentries(
server.clients.db,
server.stores.user,
user,
);
return {
username,
actor: {
user: {
id: user.id,
uuid: user.uuid,
username,
email: user.email ?? null,
email_confirmed: true,
} as Actor['user'],
},
};
};
const call = async <B>(
actor: Actor,
handler: (req: Request, res: Response) => Promise<void>,
body: B,
): Promise<unknown> => {
let captured: unknown;
const res = {
json: (value: unknown) => {
captured = value;
return res;
},
status: () => res,
setHeader: () => res,
};
const req = {
body,
query: {},
headers: { 'content-type': 'application/json' },
actor,
user: { id: actor.user!.id, username: actor.user!.username },
};
await runWithContext({ actor }, () =>
handler.call(
controller,
req as unknown as Request,
res as unknown as Response,
),
);
return captured;
};
const upload = async (
actor: Actor,
path: string,
thumbnailData: (
started: ClientSignedWriteResponse,
) => string | undefined,
) => {
const [started] = (await call<SignedWriteRequest[]>(
actor,
controller.startBatchWrites,
[
{
fileMetadata: { path, size: 4 },
thumbnailMetadata: {
contentType: 'image/png',
size: png.length,
},
},
],
)) as ClientSignedWriteResponse[];
await fetch(started!.url!, { method: 'PUT', body: 'abcd' });
if (started!.thumbnailUploadUrl) {
await fetch(started!.thumbnailUploadUrl, {
method: 'PUT',
body: png,
headers: { 'content-type': 'image/png' },
});
}
const thumbnail = thumbnailData(started!);
await call<CompleteWriteRequest[]>(
actor,
controller.completeBatchWrites,
[
{
uploadId: started!.sessionId,
...(thumbnail === undefined
? {}
: { thumbnailData: thumbnail }),
},
],
);
return {
started: started!,
entry: (await server.stores.fsEntry.getEntryByPath(path))!,
};
};
it('stores the pointer minted for the uploaded entry', async () => {
const { actor, username } = await makeActor();
const { started, entry } = await upload(
actor,
`/${username}/Documents/photo.png`,
(s) => s.thumbnailUrl,
);
expect(started.thumbnailUrl).toMatch(
new RegExp(`^s3://${BUCKET}/thumbnails/${entry.uuid}/`),
);
expect(entry.thumbnail).toBe(started.thumbnailUrl);
});
it("drops a pointer minted for another user's entry", async () => {
const victim = await makeActor();
const { entry: victimEntry } = await upload(
victim.actor,
`/${victim.username}/Documents/photo.png`,
(s) => s.thumbnailUrl,
);
const attacker = await makeActor();
const { entry } = await upload(
attacker.actor,
`/${attacker.username}/Documents/decoy.png`,
() => victimEntry.thumbnail!,
);
expect(entry.thumbnail).toBeNull();
});
it('keeps an external image URL as given', async () => {
const { actor, username } = await makeActor();
const { entry } = await upload(
actor,
`/${username}/Documents/linked.png`,
() => 'https://example.com/thumb.png',
);
expect(entry.thumbnail).toBe('https://example.com/thumb.png');
});
});
+51 -144
View File
@@ -2,13 +2,11 @@ import {
CopyObjectCommand,
DeleteObjectCommand,
GetObjectCommand,
HeadObjectCommand,
PutObjectCommand,
S3Client,
} from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { extension } from '@heyputer/backend/src/extensions';
import { isMissingObjectError } from '@heyputer/backend/src/stores/fs/S3ObjectStore';
import crypto from 'node:crypto';
import sharp from 'sharp';
const clients = extension.import('client');
@@ -24,13 +22,8 @@ const THUMBNAIL_KEY_PREFIX = 'thumbnails/';
const UUID_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
const isUuid = (value: unknown): value is string =>
typeof value === 'string' && UUID_PATTERN.test(value);
// `thumbnails/<entry uuid>/<random uuid>`: the entry segment binds the object
// to the entry it was minted for, which is what writes and deletes check.
const mintThumbnailKey = (entryUuid: string): string =>
`${THUMBNAIL_KEY_PREFIX}${entryUuid}/${crypto.randomUUID()}`;
const mintThumbnailKey = (): string =>
`${THUMBNAIL_KEY_PREFIX}${crypto.randomUUID()}`;
/**
* Extract the object key from a stored thumbnail pointer, or null when the
@@ -38,13 +31,12 @@ const mintThumbnailKey = (entryUuid: string): string =>
*
* `fsentries.thumbnail` is writable through the FS API, so neither half of the
* stored string is trusted: the bucket is discarded (callers always pass their
* own) and the key must sit under {@link THUMBNAIL_KEY_PREFIX} as either
* `<uuid>` (minted before keys were bound to entries) or `<entry uuid>/<uuid>`.
* Honouring an arbitrary key would lend this extension's storage credentials to
* whatever object the caller named — in the shared-bucket layout that is every
* user's file, since an fs object's key is its fsentry uuid. Bare-uuid keys
* from before the prefix fail the check and are treated as absent; they are
* indistinguishable from a planted pointer, so there is nothing safer to do
* own) and the key must sit under {@link THUMBNAIL_KEY_PREFIX} with a random
* uuid. Honouring an arbitrary key would lend this extension's storage
* credentials to whatever object the caller named — in the shared-bucket layout
* that is every user's file, since an fs object's key is its fsentry uuid.
* Legacy bare-uuid thumbnails fail the check and are treated as absent; they
* are indistinguishable from a planted pointer, so there is nothing safer to do
* with them than stop signing them.
*/
const resolveThumbnailKey = (pointer: string): string | null => {
@@ -66,26 +58,10 @@ const resolveThumbnailKey = (pointer: string): string | null => {
key = segments.join('/');
}
if (!key.startsWith(THUMBNAIL_KEY_PREFIX)) return null;
const segments = key.slice(THUMBNAIL_KEY_PREFIX.length).split('/');
return segments.length <= 2 && segments.every(isUuid) ? key : null;
const id = key.slice(THUMBNAIL_KEY_PREFIX.length);
return UUID_PATTERN.test(id) ? key : null;
};
// The entry a resolved key was minted for. Null for an unbound key, whose
// holder can't be told apart from a row that copied or planted the pointer.
const thumbnailKeyOwner = (key: string): string | null => {
const segments = key.slice(THUMBNAIL_KEY_PREFIX.length).split('/');
return segments.length === 2 ? segments[0] : null;
};
// Whether a stored thumbnail names an object in our storage. Anything else is
// an external image URL the client supplied: shown as given, never copied or
// deleted.
const isStoragePointer = (pointer: string, bucketEndpoint: string): boolean =>
pointer.startsWith('s3://') ||
// Legacy format — remove after full migration
(pointer.startsWith('https') &&
pointer.includes(new URL(bucketEndpoint).hostname));
// S3 client + bucket config — lazily resolved after boot from config.
let s3Client: S3Client | null = null;
let s3PresignClient: S3Client | null = null;
@@ -168,73 +144,16 @@ async function decodeAndValidateThumbnail(
return { mimeType, data };
}
// A storage pointer a client hands back after a presigned upload is kept only
// if it names a key minted for this entry and an object within the size bound.
async function vetUploadedThumbnail(
pointer: string,
entryUuid: unknown,
deps: { s3: S3Client; bucketName: string },
): Promise<string | null> {
const key = resolveThumbnailKey(pointer);
if (!key || thumbnailKeyOwner(key) !== entryUuid) return null;
let size: number | undefined;
try {
const head = await deps.s3.send(
new HeadObjectCommand({ Bucket: deps.bucketName, Key: key }),
);
size = head.ContentLength;
} catch (err) {
if (!isMissingObjectError(err)) {
console.warn(
'[thumbnails] failed to check uploaded thumbnail',
err,
);
}
return null;
}
if (typeof size !== 'number') return null;
if (size > MAX_THUMBNAIL_BYTES) {
// Only reachable on a store that doesn't enforce the signed length.
// The key is this entry's own, so removing it is safe.
try {
await deps.s3.send(
new DeleteObjectCommand({ Bucket: deps.bucketName, Key: key }),
);
} catch (err) {
console.warn(
'[thumbnails] failed to remove oversized thumbnail',
err,
);
}
return null;
}
return `s3://${deps.bucketName}/${key}`;
}
// -- thumbnail.created -----------------------------------------------
// Intercept data-URL thumbnails before they hit the DB: upload to S3
// and replace the URL with an s3:// pointer bound to the entry.
// and replace the URL with an s3:// pointer.
export async function handleThumbnailCreated(
event: Record<string, unknown>,
deps: { s3: S3Client; bucketName: string; bucketEndpoint: string },
deps: { s3: S3Client; bucketName: string },
): Promise<void> {
const url = event.url;
if (typeof url !== 'string') return;
if (!url.startsWith('data:')) {
if (isStoragePointer(url, deps.bucketEndpoint)) {
event.url = await vetUploadedThumbnail(url, event.uuid, deps);
}
return;
}
const entryUuid = event.uuid;
if (!isUuid(entryUuid)) {
event.url = null;
return;
}
if (typeof url !== 'string' || !url.startsWith('data:')) return;
const decoded = await decodeAndValidateThumbnail(url);
if (!decoded) {
@@ -242,7 +161,7 @@ export async function handleThumbnailCreated(
return;
}
const key = mintThumbnailKey(entryUuid);
const key = mintThumbnailKey();
event.url = `s3://${deps.bucketName}/${key}`;
await deps.s3.send(
@@ -271,24 +190,21 @@ export const handleThumbnailUploadPrepare = async (
typeof item.contentType === 'string' ? item.contentType.trim() : '';
if (!contentType) continue;
const size = item.size;
if (
typeof size !== 'number' ||
!Number.isInteger(size) ||
size < 0 ||
size > MAX_THUMBNAIL_BYTES
)
continue;
if (!isUuid(item.item_uid)) continue;
if (item.size !== undefined) {
const size = Number(item.size);
if (
!Number.isFinite(size) ||
size < 0 ||
size > MAX_THUMBNAIL_BYTES
)
continue;
}
const key = mintThumbnailKey(item.item_uid);
const key = mintThumbnailKey();
const command = new PutObjectCommand({
Bucket: deps.bucketName,
Key: key,
ContentType: contentType,
// Signing covers `content-length`, so a body of any other size
// fails the signature instead of landing.
ContentLength: size,
});
item.uploadUrl = await getSignedUrl(presignClient, command, {
expiresIn: 900,
@@ -311,7 +227,12 @@ export const handleThumbnailRead = async (
if (typeof thumb !== 'string' || !thumb) return;
const presignClient = deps.s3Presign;
if (isStoragePointer(thumb, deps.bucketEndpoint)) {
if (
thumb.startsWith('s3://') ||
// Legacy format — remove after full migration
(thumb.startsWith('https') &&
thumb.includes(new URL(deps.bucketEndpoint).hostname))
) {
const key = resolveThumbnailKey(thumb);
if (!key) {
// Not a pointer we minted — refuse to sign it rather than hand
@@ -326,9 +247,7 @@ export const handleThumbnailRead = async (
);
} else if (thumb.startsWith('data')) {
// Inline data-URL migration: upload to S3 and update the DB entry.
const uuid = entry.uuid ?? entry.uid;
if (!isUuid(uuid)) return;
const key = mintThumbnailKey(uuid);
const key = mintThumbnailKey();
const { mimeType, data } = base64ParseDataUrl(thumb);
const newUrl = `s3://${deps.bucketName}/${key}`;
@@ -342,6 +261,7 @@ export const handleThumbnailRead = async (
);
// Best-effort async DB update
const uuid = entry.uuid ?? entry.uid;
if (uuid) {
deps.db
.write(
@@ -366,36 +286,24 @@ export const handleFsCopyNodeThumbnail = async (
deps: {
s3: S3Client;
bucketName: string;
bucketEndpoint: string;
db: { write: (sql: string, params: unknown[]) => Promise<unknown> };
},
): Promise<void> => {
const copy = payload.copy;
const thumbnailUrl = copy?.thumbnail;
if (!copy || !isUuid(copy.uuid) || typeof thumbnailUrl !== 'string') return;
if (!isStoragePointer(thumbnailUrl, deps.bucketEndpoint)) return;
if (!copy || !copy.uuid || typeof thumbnailUrl !== 'string') return;
// Same trust rule as the read and remove paths: only touch objects this
// extension minted.
const sourceKey = resolveThumbnailKey(thumbnailUrl);
if (!sourceKey) return;
// The copied row points at the SAME object as its source. Give the copy
// an object of its own, bound to it, so removing either entry leaves the
// other's thumbnail alone.
const newKey = mintThumbnailKey(copy.uuid);
// The copied row points at the SAME S3 object as its source, and
// fs.remove.node deletes the pointed-to object — so the first removal
// among the sharers (an overwrite, a trash purge) would break every
// other sharer's thumbnail. Give the copy an object of its own.
const newKey = mintThumbnailKey();
try {
// Nothing larger than an upload may produce gets duplicated; such an
// object came in through an unbounded upload URL.
const head = await deps.s3.send(
new HeadObjectCommand({ Bucket: deps.bucketName, Key: sourceKey }),
);
if (
typeof head.ContentLength !== 'number' ||
head.ContentLength > MAX_THUMBNAIL_BYTES
) {
throw new Error('thumbnail exceeds the size bound');
}
await deps.s3.send(
new CopyObjectCommand({
Bucket: deps.bucketName,
@@ -404,8 +312,10 @@ export const handleFsCopyNodeThumbnail = async (
}),
);
} catch (err) {
// The shared object is gone or oversized — drop the pointer rather
// than leave the row advertising a thumbnail it doesn't have.
// The shared object is already gone (e.g. a sharer was removed
// before this fix existed) — the pointer is dead either way, so
// drop it rather than leave the row advertising a thumbnail it
// doesn't have.
await deps.db.write(
'UPDATE `fsentries` SET `thumbnail` = NULL WHERE `uuid` = ?',
[copy.uuid],
@@ -421,18 +331,18 @@ export const handleFsCopyNodeThumbnail = async (
};
export const handleFsRemoveNodeThumbnail = async (
payload: { target: { thumbnail?: string | null; uuid?: string } },
deps: { s3: S3Client; bucketName: string; bucketEndpoint: string },
payload: { target: { thumbnail?: string | null } },
deps: { s3: S3Client; bucketName: string },
): Promise<void> => {
const { thumbnail: thumbnailUrl, uuid } = payload.target;
if (!thumbnailUrl || !isStoragePointer(thumbnailUrl, deps.bucketEndpoint))
return;
const thumbnailUrl = payload.target.thumbnail;
if (!thumbnailUrl) return;
// The pointer decides which object gets deleted, and a key is visible to
// anyone who can list the entry holding it. Only a key minted for this
// entry is deleted; an unbound one may be shared or planted, so it stays.
// Same trust rule as the read path, and load-bearing for the same reason:
// the pointer decides which object gets deleted, so a key we didn't mint
// would let the owner of one file destroy an object belonging to someone
// else just by naming it here.
const key = resolveThumbnailKey(thumbnailUrl);
if (!key || thumbnailKeyOwner(key) !== uuid) return;
if (!key) return;
await deps.s3.send(
new DeleteObjectCommand({ Bucket: deps.bucketName, Key: key }),
@@ -445,7 +355,6 @@ extension.on(
await handleThumbnailCreated(event, {
s3: getClient(),
bucketName: thumbnailBucketName,
bucketEndpoint: extensionBucketEndpoint,
});
},
);
@@ -488,7 +397,6 @@ extension.on('fs.copy.node', async (_key, payload) => {
{
s3: getClient(),
bucketName: thumbnailBucketName,
bucketEndpoint: extensionBucketEndpoint,
db: clients.db,
},
);
@@ -501,6 +409,5 @@ extension.on('fs.remove.node', async (_key, payload) => {
await handleFsRemoveNodeThumbnail(payload, {
s3: getClient(),
bucketName: thumbnailBucketName,
bucketEndpoint: extensionBucketEndpoint,
});
});
-355
View File
@@ -1,355 +0,0 @@
import { EventEmitter } from 'node:events';
import { Readable } from 'node:stream';
import type { Request, Response } from 'express';
import { beforeEach, describe, expect, test, vi } from 'vitest';
const SECRET = 'ingress-secret';
// Only `extension` is stubbed; the secret comparison and the routing are the
// real code paths, and the fake service/store layer is the boundary.
const state = vi.hoisted(() => ({
config: { userEmail: { secret: 'ingress-secret' } } as Record<
string,
unknown
>,
routes: [] as Array<{
path: string;
handler: (req: unknown, res: unknown) => unknown;
}>,
users: new Map<string, Record<string, unknown>>(),
writes: [] as Array<{
userId: number;
path: string;
homeRegion?: string;
content?: unknown;
}>,
feedback: [] as Array<{ to: string; from: string | null; raw: Buffer }>,
takeFeedback: true,
}));
vi.mock('@heyputer/backend/src/extensions', () => ({
extension: {
get config() {
return state.config;
},
post(
path: string,
_opts: unknown,
handler: (req: unknown, res: unknown) => unknown,
) {
state.routes.push({ path, handler });
},
import(layer: string) {
if (layer === 'client') {
return {
event: {
async emitAndWait(
key: string,
data: {
to: string;
from: string | null;
raw: Buffer;
handled: boolean;
},
) {
if (key !== 'email.ingress.feedback') return;
state.feedback.push(data);
if (state.takeFeedback) data.handled = true;
},
},
};
}
if (layer === 'store') {
return {
user: {
getByUsername: async (username: string) =>
state.users.get(username) ?? null,
},
};
}
return {
fs: {
async write(
userId: number,
{
fileMetadata,
fileContent,
}: {
fileMetadata: { path: string };
fileContent: unknown;
},
_uploadTracker?: unknown,
_storageAllowanceMax?: number,
homeRegion?: string,
) {
state.writes.push({
userId,
path: fileMetadata.path,
homeRegion,
content: fileContent,
});
return { fsEntry: { path: fileMetadata.path } };
},
},
};
},
},
}));
import './index.js';
const permanent = (id: number, username: string) => ({
id,
username,
email: `${username}@example.com`,
password: '$2b$10$hash',
});
/** No password and no email is what makes an account temporary. */
const temp = (id: number, username: string) => ({
id,
username,
email: null,
password: null,
});
const ingress = async (
to: string,
over: { secret?: string; contentLength?: string | null } = {},
) => {
const route = state.routes.find((r) => r.path === '/email/ingress');
if (!route) throw new Error('route not registered');
const body = Buffer.from('From: [email protected]\r\n\r\nhello\r\n');
const req = Readable.from([body], { objectMode: false }) as unknown as {
headers: Record<string, string>;
query: Record<string, string>;
};
// `contentLength: null` sends no length header at all, which is the one
// framing the HTTP parser does not bound for us.
req.headers =
over.contentLength === null
? {}
: {
'content-length': over.contentLength ?? String(body.length),
};
req.query = {
SECRET: over.secret ?? SECRET,
to,
from: '[email protected]',
subject: 'hi',
};
let status = 200;
let ended = false;
// `end` takes (chunk, encoding) and no callback, the way the compression
// middleware leaves it, and emits `finish` — which is what the refusal
// paths hang their drain/destroy on. A stub that accepts a callback
// instead hides the bug that took the 413 path to production as a 500.
const emitter = new EventEmitter();
const res = {
status(code: number) {
status = code;
return res;
},
end(chunk?: unknown, encoding?: BufferEncoding) {
if (chunk) Buffer.byteLength(chunk as string, encoding);
ended = true;
emitter.emit('finish');
},
once(event: string, listener: () => void) {
emitter.once(event, listener);
return res;
},
} as unknown as Response;
await route.handler(req as unknown as Request, res);
return {
status,
ended,
drained: (req as unknown as Readable).readableFlowing,
destroyed: (req as unknown as Readable).destroyed,
};
};
beforeEach(() => {
state.users.clear();
state.writes.length = 0;
state.feedback.length = 0;
state.takeFeedback = true;
});
describe('temporary accounts cannot receive', () => {
test('mail for a temp account is accepted and dropped', async () => {
state.users.set('tempy', temp(1, 'tempy'));
const { status, ended, drained } = await ingress('[email protected]');
// Accepted, so the relay neither retries nor bounces to the sender.
expect(status).toBe(200);
expect(ended).toBe(true);
// The body is read off the wire rather than left unconsumed.
expect(drained).toBe(true);
expect(state.writes).toEqual([]);
});
test('mail for a permanent account still lands in the mailbox', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status } = await ingress('[email protected]');
expect(status).toBe(200);
expect(state.writes).toHaveLength(1);
expect(state.writes[0].userId).toBe(2);
expect(state.writes[0].path).toMatch(/^\/bob\/\.mail\/objects\//);
});
test('an unknown address is still rejected outright', async () => {
const { status } = await ingress('[email protected]');
expect(status).toBe(404);
expect(state.writes).toEqual([]);
});
});
describe('inbound mail is placed in the recipient home region', () => {
/** A permanent account with explicit placement columns. */
const placed = (
id: number,
username: string,
over: { home?: string | null; signup_server?: string | null },
) => ({
...permanent(id, username),
home: null,
signup_server: null,
...over,
});
test("uses the recipient's own home region when they have one", async () => {
state.users.set('bob', placed(2, 'bob', { home: 'frankfurt' }));
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('frankfurt');
});
test('falls back to the server that served their signup', async () => {
// Every account that predates the `home` column is this case.
state.users.set(
'bob',
placed(2, 'bob', { home: null, signup_server: 'london' }),
);
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('london');
});
test('falls back to the primary region when neither is set', async () => {
state.users.set('bob', placed(2, 'bob', {}));
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('oregon');
});
test('home wins over signup_server when both are set', async () => {
state.users.set(
'bob',
placed(2, 'bob', { home: 'sydney', signup_server: 'london' }),
);
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('sydney');
});
});
describe('refusals dispose of the body nobody read', () => {
test('an unknown recipient drains it and keeps the connection', async () => {
// The secret checked out and the length is under the cap, so the body
// was ours to read: discard it rather than resetting a usable socket.
const { status, drained, destroyed } =
await ingress('[email protected]');
expect(status).toBe(404);
expect(drained).toBe(true);
expect(destroyed).toBe(false);
});
test('a bad secret closes rather than reading the upload', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status, destroyed } = await ingress('[email protected]', {
secret: 'not-the-secret',
});
expect(status).toBe(403);
// Nothing authenticated this caller; their bytes are not worth reading.
expect(destroyed).toBe(true);
expect(state.writes).toEqual([]);
});
test('no content-length is refused with 411', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status, destroyed } = await ingress('[email protected]', {
contentLength: null,
});
expect(status).toBe(411);
expect(destroyed).toBe(true);
expect(state.writes).toEqual([]);
});
test('a declared length over the cap is refused with 413', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status, destroyed } = await ingress('[email protected]', {
contentLength: String(25 * 1024 * 1024 + 1),
});
expect(status).toBe(413);
expect(destroyed).toBe(true);
expect(state.writes).toEqual([]);
});
});
describe('feedback addresses', () => {
test('are handed to a listener and reach no mailbox', async () => {
const { status, ended } = await ingress('[email protected]');
expect(status).toBe(200);
expect(ended).toBe(true);
expect(state.feedback).toHaveLength(1);
expect(state.feedback[0].to).toBe('[email protected]');
expect(state.feedback[0].from).toBe('[email protected]');
expect(state.feedback[0].raw.toString()).toContain('hello');
expect(state.writes).toEqual([]);
});
test('match the local part case-insensitively', async () => {
await ingress('[email protected]');
expect(state.feedback).toHaveLength(1);
});
test('fall through to normal routing when nobody takes them', async () => {
state.takeFeedback = false;
state.users.set('abuse', permanent(3, 'abuse'));
const { status } = await ingress('[email protected]');
expect(status).toBe(200);
expect(state.feedback).toHaveLength(1);
// The body was already read, so the buffered copy is what is stored.
expect(state.writes).toHaveLength(1);
expect(Buffer.isBuffer(state.writes[0].content)).toBe(true);
});
test('an unclaimed one with no such account is still a 404', async () => {
state.takeFeedback = false;
const { status } = await ingress('[email protected]');
expect(status).toBe(404);
});
test('only count on a Puter mail domain', async () => {
state.users.set('abuse', permanent(3, 'abuse'));
await ingress('[email protected]');
expect(state.feedback).toEqual([]);
});
test('are refused over the feedback size cap without being read', async () => {
const { status, destroyed } = await ingress('[email protected]', {
contentLength: String(5 * 1024 * 1024 + 1),
});
expect(status).toBe(413);
expect(destroyed).toBe(true);
expect(state.feedback).toEqual([]);
});
});
-161
View File
@@ -1,161 +0,0 @@
import { extension } from '@heyputer/backend/src/extensions';
import { createHmac, randomBytes, timingSafeEqual } from 'node:crypto';
import type { Request } from 'express';
import {
MAX_MESSAGE_BYTES,
isPuterEmailAddress,
isTempUser,
puterEmailUsername,
storeInboxMessage,
} from '@heyputer/backend/src/services/email/mailbox.js';
// copied form peer secrets checker
const COMPARE_KEY = randomBytes(32);
const secretsEqual = (a: string, b: string): boolean =>
timingSafeEqual(
createHmac('sha256', COMPARE_KEY).update(a).digest(),
createHmac('sha256', COMPARE_KEY).update(b).digest(),
);
const USER_EMAIL_CONFIG = (extension.config as Record<string, unknown>)
.userEmail as { secret?: string; feedbackAddresses?: unknown } | undefined;
const INGRESS_SECRET = USER_EMAIL_CONFIG?.secret;
/**
* Local parts that take complaint feedback rather than naming a mailbox. The
* same names are reserved at signup, so no account can be handed them.
*/
const DEFAULT_FEEDBACK_ADDRESSES = ['fbl', 'abuse', 'postmaster'];
const FEEDBACK_ADDRESSES = new Set(
(Array.isArray(USER_EMAIL_CONFIG?.feedbackAddresses)
? USER_EMAIL_CONFIG.feedbackAddresses
: DEFAULT_FEEDBACK_ADDRESSES
).map((local) => String(local).toLowerCase()),
);
/**
* Feedback is handed over whole, so it is read into memory; a complaint report
* quotes one message and has no business near the mailbox ceiling.
*/
const MAX_FEEDBACK_BYTES = 5 * 1024 * 1024;
const isFeedbackAddress = (to: string): boolean =>
isPuterEmailAddress(to) &&
FEEDBACK_ADDRESSES.has(puterEmailUsername(to).toLowerCase());
const readBody = async (req: Request): Promise<Buffer> => {
const chunks: Buffer[] = [];
for await (const chunk of req) chunks.push(chunk as Buffer);
return Buffer.concat(chunks);
};
if (!INGRESS_SECRET) {
console.warn(
'[useremailrx] config.userEmail.secret unset - email ingress disabled',
);
} else {
extension.post(
'/email/ingress',
{
subdomain: 'api',
},
async (req, res) => {
// Answer first, then deal with the body nobody read. Touching the
// request before the response has flushed takes the response down
// with the connection and the caller sees a reset with no status,
// so this hangs off `finish` — and it has to be the event rather
// than `res.end(cb)`, because the compression middleware replaces
// `res.end` with an `(chunk, encoding)` signature that has no
// callback parameter, so a callback lands in `chunk` and dies in
// `Buffer.byteLength`.
//
// `drain` discards the rest of a body we were entitled to read and
// leaves the connection reusable. `close` is for bodies we do not
// want to read at all: an unauthenticated caller's, or one whose
// declared length is missing or over the cap.
const refuse = (status: number, how: 'drain' | 'close') => {
res.once('finish', () =>
how === 'drain' ? req.resume() : req.destroy(),
);
res.status(status).end();
};
const presented = req.query.SECRET;
if (
typeof presented !== 'string' ||
!secretsEqual(presented, INGRESS_SECRET)
) {
// Nothing has authenticated this caller, so refuse to spend
// bandwidth reading whatever they were uploading.
refuse(403, 'close');
return;
}
const size = Number(req.headers['content-length']);
if (!Number.isInteger(size) || size <= 0) {
refuse(411, 'close');
return;
}
if (size > MAX_MESSAGE_BYTES) {
refuse(413, 'close');
return;
}
const stores = extension.import('store');
const services = extension.import('service');
const to = String(req.query.to ?? '');
let content: Buffer | Request = req;
if (isFeedbackAddress(to)) {
if (size > MAX_FEEDBACK_BYTES) {
refuse(413, 'close');
return;
}
const raw = await readBody(req);
const event = {
to,
from:
typeof req.query.from === 'string'
? req.query.from
: null,
raw,
handled: false,
};
await extension
.import('client')
.event.emitAndWait('email.ingress.feedback', event, {});
if (event.handled) {
res.end();
return;
}
// Nobody took it: route it like any other message.
content = raw;
}
const user = await stores.user.getByUsername(
puterEmailUsername(to),
);
if (!user) {
refuse(404, 'drain');
return;
}
if (isTempUser(user)) {
if (content === req) req.resume();
res.end();
return;
}
await storeInboxMessage(services.fs, user, {
subject: req.query.subject,
content,
size: Buffer.isBuffer(content) ? content.length : size,
});
const { SECRET: _secret, ..._loggableQuery } = req.query;
// console.log('got email: ', puterUser, loggableQuery, size);
res.end();
},
);
}
-12
View File
@@ -1,12 +0,0 @@
{
"name": "emailrx",
"version": "1.0.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "emailrx",
"version": "1.0.0"
}
}
}
-6
View File
@@ -1,6 +0,0 @@
{
"name": "emailrx",
"version": "1.0.0",
"main": "index.ts",
"description": "Internal service which handles routing puter emails to users"
}
+4 -302
View File
@@ -8,9 +8,7 @@ import {
it,
vi,
} from 'vitest';
import { makeActor, type Actor } from '../src/backend/core/actor.ts';
import { runWithContext } from '../src/backend/core/context.ts';
import { configContainer } from '../src/backend/exports.ts';
import { PuterServer } from '../src/backend/server.ts';
import { setupTestServer } from '../src/backend/testUtil.ts';
import { handleWhoami } from './whoami.ts';
@@ -49,7 +47,6 @@ beforeAll(async () => {
create_shortcut: true,
payment_bypass: true,
},
teams_enabled: true,
} as never);
});
@@ -68,81 +65,6 @@ const seedUser = async () => {
};
describe('whoami extension — handleWhoami', () => {
// The sidebar label needs this at boot, which is why it rides whoami
// rather than a call of its own.
describe('the team an account belongs to', () => {
// A seat is created, never adopted, so it must have no password.
const seedSeat = async () => {
const slug = Math.random().toString(36).slice(2, 8);
return server.stores.user.create({
username: `wseat_${slug}`,
uuid: uuidv4(),
password: null,
email: null,
});
};
const seatOf = async (teamName: string) => {
const owner = await seedUser();
const seat = await seedSeat();
const team = await server.stores.team.create({
ownerUserId: owner.id as number,
name: teamName,
handle: `wt-${Math.random().toString(36).slice(2, 9)}`,
});
await server.stores.team.addMember(team.uid, seat.id as number, {
orgOwned: true,
});
return { seat, team };
};
it('names the team for a seat', async () => {
const { seat, team } = await seatOf('Acme Corp');
const { res, captured } = makeRes();
await runWithContext(
{ actor: { user: { uuid: seat.uuid, id: seat.id as number } } },
() => handleWhoami(makeReq(), res),
);
expect((captured.body as { team?: unknown }).team).toEqual({
uid: team.uid,
name: 'Acme Corp',
});
});
it('says nothing for an account that is not a seat', async () => {
const user = await seedUser();
const { res, captured } = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleWhoami(makeReq(), res),
);
expect(captured.body).not.toHaveProperty('team');
});
it('withholds it from an app actor', async () => {
// Same class as the phone number: a seat's employer is not an
// app's business.
const { seat } = await seatOf('Acme Corp');
const { res, captured } = makeRes();
await runWithContext(
{
actor: makeActor({
user: { uuid: seat.uuid, id: seat.id as number },
app: { uid: 'app-1' },
}),
},
() => handleWhoami(makeReq(), res),
);
expect(captured.body).not.toHaveProperty('team');
});
});
it('returns 401 when no actor is on the context', async () => {
const { res, captured } = makeRes();
@@ -196,50 +118,6 @@ describe('whoami extension — handleWhoami', () => {
expect(body).toHaveProperty('taskbar_items');
});
it('reports the SMS-to-card fallback only once a send has opened it', async () => {
const user = await server.stores.user.create({
username: `wuser_${Math.random().toString(36).slice(2, 8)}`,
uuid: uuidv4(),
password: 'hashedpw',
email: `${Math.random().toString(36).slice(2, 8)}@example.com`,
requires_phone_verification: true,
} as never);
const prev = configContainer.phone_verification_card_fallback;
configContainer.phone_verification_card_fallback = {
enabled: true,
} as never;
try {
const before = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleWhoami(makeReq(), before.res),
);
// Phone-gated, but the user has attempts left — no offer yet.
expect(
before.captured.body as Record<string, unknown>,
).toMatchObject({ card_fallback_available: false });
// Exhausting SMS attempts stamps this flag; whoami is then the only
// thing that can still tell a reloading GUI about the offer, since
// further sends are rejected by the route's own rate limit.
await server.stores.kv.set({
key: `card-fallback-open:${user.id}`,
value: true,
});
const after = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleWhoami(makeReq(), after.res),
);
expect(
after.captured.body as Record<string, unknown>,
).toMatchObject({ card_fallback_available: true });
} finally {
configContainer.phone_verification_card_fallback = prev;
}
});
it('only forwards allow-listed feature flags', async () => {
const user = await seedUser();
const { res, captured } = makeRes();
@@ -263,10 +141,10 @@ describe('whoami extension — handleWhoami', () => {
await runWithContext(
{
actor: makeActor({
actor: {
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
},
},
() => handleWhoami(makeReq(), res),
);
@@ -285,104 +163,6 @@ describe('whoami extension — handleWhoami', () => {
expect(body.directories).toBeUndefined();
});
describe('fields added by whoami.details listeners', () => {
const addAccountDetails = (
_key: unknown,
event: { details: Record<string, unknown> },
) => {
Object.assign(event.details, {
subscribed: true,
paid_storage: 1024,
subscription: { tier: 'pro', status: 'trialing' },
accountEligible: true,
});
};
const whoamiWithListener = async (
actor: Actor,
listener: typeof addAccountDetails = addAccountDetails,
) => {
const { res, captured } = makeRes();
server.clients.event.on('whoami.details', listener);
try {
await runWithContext({ actor }, () =>
handleWhoami(makeReq(), res),
);
} finally {
server.clients.event.off('whoami.details', listener);
}
return captured.body as Record<string, unknown>;
};
it('reach an app actor only as the documented subscribed / paid_storage', async () => {
const user = await seedUser();
const body = await whoamiWithListener(
makeActor({
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
);
expect(body.subscribed).toBe(true);
expect(body.paid_storage).toBe(1024);
expect(body).not.toHaveProperty('subscription');
expect(body).not.toHaveProperty('accountEligible');
// Fields the handler built itself still ship.
expect(body.username).toBe(user.username);
expect(body.app_name).toBe('app-test-actor');
});
it('are withheld from an app reached through an access token it issued', async () => {
const user = await seedUser();
const owner = { uuid: user.uuid, id: user.id as number };
const body = await whoamiWithListener(
makeActor({
user: owner,
accessToken: {
uid: 'tok-1',
issuer: makeActor({
user: owner,
app: { uid: 'app-issuer' },
}),
},
}),
);
expect(body.subscribed).toBe(true);
expect(body).not.toHaveProperty('subscription');
expect(body).not.toHaveProperty('accountEligible');
});
it('all reach a user actor', async () => {
const user = await seedUser();
const body = await whoamiWithListener(
makeActor({ user: { uuid: user.uuid, id: user.id as number } }),
);
expect(body).toMatchObject({
subscribed: true,
paid_storage: 1024,
subscription: { tier: 'pro', status: 'trialing' },
accountEligible: true,
});
});
it('cannot add taskbar_items for an app actor, which core omits', async () => {
const user = await seedUser();
const body = await whoamiWithListener(
makeActor({
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
(_key, event) => {
event.details.taskbar_items = [{ name: 'injected' }];
},
);
expect(body).not.toHaveProperty('taskbar_items');
});
});
it('redacts tmp_password from metadata for user actors', async () => {
const user = await seedUser();
await server.stores.user.updateMetadata(user.id as number, {
@@ -413,10 +193,10 @@ describe('whoami extension — handleWhoami', () => {
const { res, captured } = makeRes();
await runWithContext(
{
actor: makeActor({
actor: {
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
},
},
() => handleWhoami(makeReq(), res),
);
@@ -426,84 +206,6 @@ describe('whoami extension — handleWhoami', () => {
expect(JSON.stringify(body)).not.toContain('bootstrap-secret');
});
it('never exposes phone, card fingerprint or signup identity', async () => {
const user = await seedUser();
await server.stores.user.update(user.id as number, {
phone: '+15551234567',
card_fingerprint: 'fp_ABC123',
requires_phone_verification: false,
requires_card_verification: false,
});
// Guard against a vacuous assertion below: the columns really do hold
// the values we then expect never to see on the wire.
const stored = await server.stores.user.getById(user.id as number, {
cached: false,
force: true,
});
expect(stored?.phone).toBe('+15551234567');
expect(stored?.card_fingerprint).toBe('fp_ABC123');
for (const actor of [
makeActor({ user: { uuid: user.uuid, id: user.id as number } }),
makeActor({
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
]) {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleWhoami(makeReq(), res),
);
const body = captured.body as Record<string, unknown>;
expect(body.phone).toBeUndefined();
expect(body.card_fingerprint).toBeUndefined();
expect(body.password).toBeUndefined();
expect(body.otp_secret).toBeUndefined();
expect(body.signup_ip).toBeUndefined();
// Not just absent as a key — the values must not appear anywhere
// in the payload (nested under metadata, taskbar items, …).
const serialized = JSON.stringify(body);
expect(serialized).not.toContain('5551234567');
expect(serialized).not.toContain('fp_ABC123');
// The verification flags, which the GUI acts on, still ship.
expect(body).toHaveProperty('requires_phone_verification');
expect(body).toHaveProperty('requires_card_verification');
}
});
it('scrubs sensitive keys added to metadata, and leaves the cached row intact', async () => {
const user = await seedUser();
await server.stores.user.updateMetadata(user.id as number, {
tmp_password: 'bootstrap-secret',
billing: { card_fingerprint: 'fp_NESTED', tier: 'pro' },
});
const { res, captured } = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleWhoami(makeReq(), res),
);
const body = captured.body as Record<string, unknown>;
const metadata = body.metadata as Record<string, unknown>;
expect(metadata.tmp_password).toBeUndefined();
// Nested sensitive keys are removed too; siblings survive.
expect(metadata.billing).toEqual({ tier: 'pro' });
expect(JSON.stringify(body)).not.toContain('fp_NESTED');
// The scrub works on a copy — server-side state is untouched.
const fresh = await server.stores.user.getById(user.id as number, {
cached: false,
force: true,
});
expect(fresh?.metadata?.tmp_password).toBe('bootstrap-secret');
expect(
(fresh?.metadata?.billing as Record<string, unknown>)
?.card_fingerprint,
).toBe('fp_NESTED');
});
it('marks the user as oidc_only when password is null', async () => {
const slug = Math.random().toString(36).slice(2, 8);
const oidcUser = await server.stores.user.create({
+20 -159
View File
@@ -1,10 +1,5 @@
import { Context } from '@heyputer/backend/src/core';
import { extension } from '@heyputer/backend/src/extensions';
import {
cardFallbackDepsFrom,
isCardFallbackEligible,
} from '@heyputer/backend/src/util/cardFallback.js';
import { APP_ICON_SIZES } from '@heyputer/backend/src/util/appIcon.js';
import { getTaskbarItems } from '@heyputer/backend/src/util/taskbarItems.js';
import type { Request, Response } from 'express';
import TimeAgo from 'javascript-time-ago';
@@ -38,75 +33,6 @@ const CLIENT_VISIBLE_FEATURE_FLAGS: ReadonlySet<string> = new Set([
'prompt_user_when_navigation_away_from_puter',
]);
// Keys that must never leave the server, whoever put them on the response.
// `details` is an explicit pick, but the `whoami.details` event hands
// listeners the full UserRow next to the object they may write to, and
// `metadata` is a free-form blob — so any of these can arrive on the
// response without an edit to the pick above. The scrub runs last, over the
// whole payload, and is the one place that decides what "sensitive" means.
//
// Credentials and single-use tokens, the payment/phone identifiers used for
// verification (`card_fingerprint` is the Stripe fingerprint, stable per
// card number), the network identity recorded at signup, and internal
// anti-abuse bookkeeping. `requires_phone_verification` /
// `requires_card_verification` stay: they are the flags the GUI acts on, and
// they carry no identifier.
const SENSITIVE_KEYS: ReadonlySet<string> = new Set([
'password',
'tmp_password',
'pass_recovery_token',
'email_confirm_code',
'email_confirm_token',
'change_email_confirm_token',
'otp_secret',
'otp_recovery_codes',
'card_fingerprint',
'phone',
'clean_email',
'signup_ip',
'signup_ip_forwarded',
'signup_user_agent',
'signup_origin',
'signup_server',
'audit_metadata',
]);
// What an app actor may receive from `whoami.details` listeners. Anything
// else a listener adds describes the account, and only ships to user actors.
const APP_VISIBLE_LISTENER_KEYS: ReadonlySet<string> = new Set([
'subscribed',
'paid_storage',
]);
// Depth-limited, cycle-safe walk deleting every SENSITIVE_KEYS entry it finds
// at any level (`metadata` and `taskbar_items` are both nested structures).
const scrubSensitive = (
value: unknown,
seen: Set<object> = new Set(),
depth = 0,
): void => {
if (depth > 8 || value === null || typeof value !== 'object') return;
if (seen.has(value as object)) return;
seen.add(value as object);
if (Array.isArray(value)) {
for (const entry of value) scrubSensitive(entry, seen, depth + 1);
return;
}
for (const key of Object.keys(value as Record<string, unknown>)) {
if (SENSITIVE_KEYS.has(key)) {
delete (value as Record<string, unknown>)[key];
continue;
}
scrubSensitive(
(value as Record<string, unknown>)[key],
seen,
depth + 1,
);
}
};
export const handleWhoami = async (
req: Request,
res: Response,
@@ -117,7 +43,7 @@ export const handleWhoami = async (
return;
}
const isUser = !actor.effectiveApp;
const isUser = !actor.app;
const user = await stores.user.getById(actor.user.id);
if (!user) {
res.status(404).json({ error: 'User not found' });
@@ -125,12 +51,13 @@ export const handleWhoami = async (
}
const oidcOnly = user.password === null;
const ALLOWED_ICON_SIZES = new Set([16, 32, 64, 128, 256, 512]);
const rawIconSize =
typeof req.query?.icon_size === 'string'
? Number(req.query.icon_size)
: undefined;
const iconSize =
rawIconSize !== undefined && APP_ICON_SIZES.includes(rawIconSize)
rawIconSize !== undefined && ALLOWED_ICON_SIZES.has(rawIconSize)
? rawIconSize
: undefined;
const noIcons = !iconSize;
@@ -147,12 +74,8 @@ export const handleWhoami = async (
}
}
// Deep-copied (it is decoded JSON) so the scrub below edits the response
// and not the cached UserRow. Sensitive keys inside it — tmp_password and
// anything else on the denylist — are removed by scrubSensitive.
const metadata = user.metadata
? structuredClone(user.metadata)
: user.metadata;
const metadata = user.metadata ? { ...user.metadata } : user.metadata;
if (metadata) delete metadata.tmp_password;
const details: Record<string, unknown> = {
username: user.username,
@@ -161,38 +84,27 @@ export const handleWhoami = async (
unconfirmed_email: user.email,
email_confirmed: user.email_confirmed || user.username === 'admin',
requires_email_confirmation: user.requires_email_confirmation,
// The phone number itself is deliberately absent: nothing on the
// client reads it, and it is PII that would otherwise be handed to
// every app actor. Only the verification flag ships.
phone: user.phone,
requires_phone_verification: user.requires_phone_verification,
requires_card_verification: user.requires_card_verification,
// A seat reaches nothing until it replaces its admin's password.
requires_password_change: user.requires_password_change,
// The SMS-to-card escape hatch: true once this user is out of SMS send
// attempts and may verify a card instead. It has to ship from here
// because /send-confirm-phone can no longer say so — by the time the
// fallback opens, further sends are rejected by that route's own rate
// limit before any handler runs, so a page reload would otherwise lose
// an offer that stays valid for 24 hours. Only for user actors, and it
// costs no KV read unless the account is actually phone-gated.
card_fallback_available: isUser
? await isCardFallbackEligible(
extension.config,
user,
async (key) => (await stores.kv.get({ key })).res,
cardFallbackDepsFrom(clients),
)
: false,
desktop_bg_url: user.desktop_bg_url,
desktop_bg_color: user.desktop_bg_color,
desktop_bg_fit: user.desktop_bg_fit,
is_temp: user.password === null && user.email === null,
is_user_token: true,
// Present only once the account has actually asked for a code (see the
// referral extension) — null until then, and never minted from here:
// this endpoint is polled, and a mint is a write.
referral_code: user.referral_code,
oidc_only: oidcOnly,
taskbar_items: isUser
? await getTaskbarItems(
user,
{
clients,
stores,
services,
apiBaseUrl: String(extension.config.api_base_url ?? ''),
},
{ iconSize, noIcons },
)
: undefined,
otp: !!user.otp_enabled,
feature_flags,
created_ts: toUnixSeconds(user.timestamp),
@@ -221,21 +133,6 @@ export const handleWhoami = async (
}
}
// Taskbar items — only sent to user actors
if (isUser) {
details.taskbar_items = await getTaskbarItems(
user,
{
clients,
stores,
services,
apiBaseUrl: String(extension.config.api_base_url ?? ''),
config: extension.config,
},
{ iconSize, noIcons },
);
}
// Directories — only sent to user actors
if (isUser) {
const directories: Record<string, unknown> = {};
@@ -253,23 +150,6 @@ export const handleWhoami = async (
details.directories = directories;
}
// The team an account belongs to, when it is one a team pays for. User
// actors only, and only where teams are on.
if (isUser && extension.config.teams_enabled === true) {
try {
const seat = await stores.team.getOrgSeat(user.id);
if (seat) {
details.team = {
uid: seat.team_uid,
name: seat.team_name ?? null,
};
}
} catch (e) {
// Never fail whoami over this; the account still works without it.
console.warn('[whoami] team lookup failed:', (e as Error).message);
}
}
// Last activity
const lastActivityTs = toUnixSeconds(user.last_activity_ts);
if (lastActivityTs !== undefined) {
@@ -292,21 +172,12 @@ export const handleWhoami = async (
delete details.created_ts;
delete details.is_user_token;
delete details.metadata;
// An app has no business reading the code its user earns credit with.
delete details.referral_code;
}
const app = actor.effectiveApp;
if (app) {
details.app_name = app.uid;
if (actor.app) {
details.app_name = actor.app.uid;
}
// A key core left undefined isn't built, so a listener can't fill it for an app.
const builtKeys = isUser
? null
: new Set(
Object.keys(details).filter((key) => details[key] !== undefined),
);
try {
await clients.event.emitAndWait(
'whoami.details',
@@ -316,13 +187,6 @@ export const handleWhoami = async (
} catch {
/* best-effort */
}
if (builtKeys) {
for (const key of Object.keys(details)) {
if (!builtKeys.has(key) && !APP_VISIBLE_LISTENER_KEYS.has(key)) {
delete details[key];
}
}
}
const subscription = details.subscription as
{ offering?: Record<string, unknown> } | undefined;
@@ -332,9 +196,6 @@ export const handleWhoami = async (
delete subscription.offering.price_id;
}
// Last word on what ships, after every listener has had its say.
scrubSensitive(details);
res.json(details);
};
-9
View File
@@ -131,11 +131,6 @@ S3_SECRET_KEY=$S3_SECRET_KEY
S3_BUCKET=puter-local
EOF
# `meteringEnforcement.subscriptions` is off here because a self-hosted
# install has no paid plans to be on: every account resolves to the free
# policy, so leaving the plan gates armed would 402 the surfaces that
# declare one (the OpenAI/Anthropic-compatible AI endpoints) for everybody.
# The code default is on — this is a property of the install, not of Puter.
log "writing puter/config/config.json"
cat > puter/config/config.json <<EOF
{
@@ -196,10 +191,6 @@ EOF
"ollama": { "enabled": false }
},
"meteringEnforcement": {
"subscriptions": false
},
"trust_proxy": $PUTER_TRUST_PROXY
}
EOF
+3976 -2192
View File
File diff suppressed because it is too large. Load diff
+26 -16
View File
@@ -13,9 +13,14 @@
"@babel/core": "^7.29.7",
"@eslint/js": "^9.35.0",
"@playwright/test": "^1.56.1",
"@stylistic/eslint-plugin": "^5.3.1",
"@types/better-sqlite3": "^7.6.13",
"@types/express": "^5.0.0",
"@types/mime-types": "^3.0.1",
"@typescript-eslint/eslint-plugin": "^8.46.1",
"@typescript-eslint/parser": "^8.46.1",
"@vitest/coverage-v8": "^4.0.14",
"@vitest/ui": "^4.0.14",
"babel-loader": "^10.1.1",
"babel-plugin-istanbul": "^8.0.0",
"chalk": "^4.1.0",
@@ -25,19 +30,25 @@
"eslint": "^9.35.0",
"eslint-config-prettier": "^10.1.8",
"eslint-plugin-prettier": "^5.5.5",
"eslint-rule-composer": "^0.3.0",
"globals": "^15.15.0",
"html-webpack-plugin": "^5.6.0",
"husky": "^9.1.7",
"istanbul-lib-coverage": "^3.2.2",
"istanbul-lib-report": "^3.0.1",
"istanbul-reports": "^3.2.0",
"license-check-and-add": "^4.0.5",
"nodemon": "^3.1.0",
"prettier": "^3.8.3",
"prettier-plugin-jsdoc": "^1.8.1",
"simple-git": "^3.32.3",
"typescript": "^5.4.5",
"uglify-js": "^3.17.4",
"vite-plugin-static-copy": "^3.3.0",
"vitest": "^4.1.5",
"webpack": "^5.88.2",
"webpack-cli": "^5.1.1"
"webpack-cli": "^5.1.1",
"yaml": "^2.8.1"
},
"scripts": {
"test:backend": "npm run setupExtensions && npm run build:workerLib && vitest run --config src/backend/vitest.config.ts ",
@@ -50,7 +61,6 @@
"build:workerLib:coverage": "cd src/puter-js && npm run build:coverage && cd ../worker && npm run build",
"start:gui": "nodemon --exec \"node dev-server.js\" ",
"start": "node ./tools/start.mjs",
"start:smtp": "node ./tools/start.mjs --smtp",
"dev": "npm start",
"build": "npm run setupExtensions && npm run build:ts && cd src/gui && node ./build.js && cd ../puter-js && npm run build",
"build:workerLib": "cd src/puter-js && npm run build && cd ../worker && npm run build",
@@ -63,7 +73,9 @@
"setupExtensions": "node ./tools/extensionSetup.mjs"
},
"workspaces": [
"src/*"
"src/*",
"tools/*",
"experiments/js-parse-and-output"
],
"nodemonConfig": {
"ext": "js, json, mjs, jsx, svg, css",
@@ -73,11 +85,14 @@
]
},
"dependencies": {
"@ai-sdk/openai": "^3.0.25",
"@aws-sdk/client-s3": "^3.1020.0",
"@aws-sdk/s3-request-presigner": "^3.1028.0",
"@heyputer/putility": "^1.0.2",
"ai": "^6.0.73",
"dedent": "^1.5.3",
"javascript-time-ago": "^2.5.11",
"libphonenumber-js": "1.13.14",
"libphonenumber-js": "1.13.6",
"miniflare": "^4.20260617.1",
"open": "^10.1.0"
},
@@ -86,17 +101,12 @@
},
"allowScripts": {
"@google/genai": false,
"bcrypt": true,
"better-sqlite3": true,
"browser-tabs-lock": true,
"classic-level": true,
"esbuild": true,
"fsevents": true,
"protobufjs": true,
"sharp": true,
"workerd": true
},
"overrides": {
"undici": "^7.30.0"
"bcrypt@5.1.1": true,
"better-sqlite3@12.11.1": true,
"browser-tabs-lock@1.3.0": true,
"classic-level@3.0.0": true,
"[email protected]": true,
"[email protected]": true,
"[email protected]": true
}
}
+4 -54
View File
@@ -375,18 +375,18 @@ describe('AlarmClient alarm registry', () => {
const client = makeClient();
const seen = capture(client);
for (let i = 0; i < 500; i++) {
for (let i = 0; i < 50; i++) {
client.create('hot', `occurrence ${i}`, { i });
}
const alarm = client.get('hot')!;
expect(alarm.count).toBe(500);
expect(alarm.count).toBe(50);
expect(alarm.occurrences).toHaveLength(20);
expect(alarm.timestamps).toHaveLength(20);
// The window kept is the most recent one, not the oldest.
expect(alarm.occurrences[19].message).toBe('occurrence 499');
expect(alarm.occurrences[19].message).toBe('occurrence 49');
// Trimming history must not rewind what the transports are told.
expect(seen.at(-1)).toMatchObject({ repeatCount: 500, isRepeat: true });
expect(seen[49]).toMatchObject({ repeatCount: 50, isRepeat: true });
});
it('names anonymous handlers by their registration order', () => {
@@ -519,53 +519,3 @@ describe('AlarmClient known-error rules', () => {
expect(chat[0].severity).toBe('info');
});
});
describe('AlarmClient repeat reporting', () => {
beforeEach(() => {
vi.spyOn(console, 'error').mockImplementation(() => {});
vi.spyOn(console, 'warn').mockImplementation(() => {});
vi.spyOn(console, 'log').mockImplementation(() => {});
pdEvent.mockClear();
});
it('reports every occurrence up to the burst, then backs off', () => {
const client = makeClient();
const seen = capture(client);
for (let i = 0; i < 12; i++) client.create('flap', 'same fault');
expect(seen).toHaveLength(10);
expect(seen[9].repeatCount).toBe(10);
});
it('reports again once the interval is reached', () => {
const client = makeClient();
const seen = capture(client);
for (let i = 0; i < 500; i++) client.create('flap', 'same fault');
expect(seen).toHaveLength(11);
expect(seen[10].repeatCount).toBe(500);
});
it('keeps a fast-repeating fault out of the log after the burst', () => {
const client = makeClient();
capture(client);
const warn = vi.mocked(console.warn);
warn.mockClear();
for (let i = 0; i < 100; i++) client.create('flap', 'same fault');
// Occurrences 2-10 only; the first is an ACTIVE line on console.error.
expect(warn.mock.calls).toHaveLength(9);
});
it('skips the payload entirely when no transport accepts the severity', () => {
const client = makeClient();
const seen = capture(client, 'critical');
client.create('quiet', 'nobody is listening', {}, 'info');
expect(seen).toHaveLength(0);
});
});
-27
View File
@@ -69,15 +69,6 @@ const FALLBACK_SEVERITY: PagerSeverity = 'critical';
* and `count` keeps that.
*/
const OCCURRENCE_HISTORY_LIMIT = 20;
// A fast-recurring fault repeats far faster than it is useful to report, and
// every report allocates a log line and a payload. Report every occurrence up
// to the burst, then one per interval; `count` still tracks all of them.
const REPEAT_REPORT_BURST = 10;
const REPEAT_REPORT_INTERVAL = 500;
const shouldReportRepeat = (count: number): boolean =>
count <= REPEAT_REPORT_BURST || count % REPEAT_REPORT_INTERVAL === 0;
/** Keeps `info` alarms out of the paging system unless config says otherwise. */
const DEFAULT_PAGERDUTY_MIN_SEVERITY: PagerSeverity = 'warning';
/** Slack's ceiling once a pager exists: chat gets what doesn't page. */
@@ -465,8 +456,6 @@ export class AlarmClient extends PuterClient {
private handleRepeat(alarm: Alarm): void {
this.applyKnownErrors(alarm);
if (!shouldReportRepeat(alarm.count)) return;
console.warn(
`[alarm] REPEAT ${displayId(alarm)} :: ${alarm.message} (${alarm.count})`,
);
@@ -494,18 +483,6 @@ export class AlarmClient extends PuterClient {
);
}
private hasHandlerFor(severity: PagerSeverity): boolean {
for (const { minSeverity, maxSeverity } of this.alertHandlers) {
if (
meetsMinSeverity(severity, minSeverity) &&
withinMaxSeverity(severity, maxSeverity)
) {
return true;
}
}
return false;
}
private dispatchAlert(alarm: Alarm): void {
const resolved = this.resolveSeverity(alarm);
if (resolved === 'mute') {
@@ -517,10 +494,6 @@ export class AlarmClient extends PuterClient {
}
alarm.severity = resolved;
// Build nothing for a severity no transport accepts — the payload
// (cleaned fields plus the stack, twice) is the expensive part.
if (!this.hasHandlerFor(resolved)) return;
const fieldsClean = cleanFields(alarm.fields);
const repeatCount = alarm.count;
@@ -17,9 +17,24 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
// Optional analytics client, registered by an extension. Only the surface the
// app-stats path consumes is declared; extend it rather than widening to `any`.
/**
* Optional ClickHouse client.
*
* ClickHouse is NOT a hard dependency. Core Puter runs without it — the app
* stats read path (open / unique-user counts) falls back to the primary SQL
* database. `this.clients.clickhouse` is therefore typed as optional and is
* `undefined` in default and self-hosted setups; callers MUST branch on its
* presence and fall back to SQL.
*
* A production deployment can register a real ClickHouse client via an
* extension (`extension.registerClient('clickhouse', client)`) to offload the
* analytics queries off the primary database and keep the stats path fast at
* scale. When registered, the instance flows into `this.clients.clickhouse`
* everywhere it's typed.
*
* Only the surface the stats path actually consumes is declared here. Extend
* this interface (don't widen to `any`) when a new query shape is needed.
*/
export interface ClickhouseQueryResult {
json<T = Record<string, unknown>>(): Promise<T[]>;
}
@@ -184,8 +184,7 @@ describe('compareMigrationFilenames', () => {
//
// `read()` normally goes to the replica batcher; when the replica side is
// degraded (batcher load-shed or a transient connection error) and a real
// replica is configured, the read retries once on the primary read batcher.
// The write batcher is left unset, so a failover that reached it would throw.
// replica is configured, the read retries once on the primary batcher.
type Batcher = { execute: ReturnType<typeof vi.fn> };
@@ -201,7 +200,7 @@ const makeClient = (opts: {
// inject the batchers directly. Configuration enum: SINGLE=0, REPLICA=1.
Object.assign(client as unknown as Record<string, unknown>, {
dbReplica: opts.replica,
dbPrimaryRead: opts.primary,
db: opts.primary,
configuration: opts.multiNode === false ? 0 : 1,
});
return client;
@@ -285,10 +284,6 @@ const startClient = async (
// must answer with one row-set per statement plus one for the SELECT 1.
const rowSets = (...sets: unknown[][]) => [[...sets, [{ 1: 1 }]], undefined];
// Writes go out one statement at a time, so the driver answers with just the
// result header.
const writeHeader = (header: Record<string, unknown>) => [header, undefined];
beforeEach(() => {
createdPools.length = 0;
createPoolMock.mockClear();
@@ -308,9 +303,6 @@ describe('MySQLDatabaseClient — pool construction', () => {
expect(createdPools[0].poolConfig).toEqual({
maxPreparedStatements: 900,
connectionLimit: 30,
maxIdle: 29,
idleTimeout: 60_000,
gracefulEnd: true,
enableKeepAlive: true,
host: 'db.internal',
port: 3307,
@@ -321,32 +313,6 @@ describe('MySQLDatabaseClient — pool construction', () => {
});
});
// mysql2 skips its idle reaper unless maxIdle < connectionLimit, and an
// unreaped connection is closed by the server under the pool's feet.
it('keeps the idle reaper on when a pool sets its own connection limit', async () => {
await startClient({
replica: { host: 'replica.internal', connectionLimit: 10 },
});
expect(createdPools[1].poolConfig).toMatchObject({
connectionLimit: 10,
maxIdle: 9,
idleTimeout: 60_000,
});
});
// mysql2 reads 0 as unlimited; a maxIdle of -1 would crash its reaper.
it('leaves maxIdle to mysql2 for an unlimited pool', async () => {
await startClient({
replica: { host: 'replica.internal', connectionLimit: 0 },
});
expect(createdPools[1].poolConfig).toMatchObject({
connectionLimit: 0,
});
expect(createdPools[1].poolConfig).not.toHaveProperty('maxIdle');
});
it('falls back to loopback defaults when the endpoint is unspecified', async () => {
await startClient();
@@ -435,30 +401,10 @@ describe('MySQLDatabaseClient — query interface', () => {
]);
});
it('sends a write on its own, outside a transaction', async () => {
const client = await startClient();
const pool = createdPools[0];
pool.respond = () => writeHeader({ affectedRows: 1 });
await client.write('UPDATE `user` SET `username` = ? WHERE `id` = ?', [
'ada',
1,
]);
expect(pool.calls).toEqual([
{
sql: 'UPDATE `user` SET `username` = ? WHERE `id` = ?',
values: ['ada', 1],
},
]);
expect(pool.connections[0].beginTransaction).not.toHaveBeenCalled();
expect(pool.connections[0].commit).not.toHaveBeenCalled();
});
it('maps the mysql result header onto the shared write result', async () => {
const client = await startClient();
createdPools[0].respond = () =>
writeHeader({ insertId: 42, affectedRows: 2 });
rowSets({ insertId: 42, affectedRows: 2 } as unknown as unknown[]);
await expect(
client.write('UPDATE `user` SET `username` = ? WHERE `id` = ?', [
@@ -474,7 +420,7 @@ describe('MySQLDatabaseClient — query interface', () => {
it('reports no rows affected when the header omits the counters', async () => {
const client = await startClient();
createdPools[0].respond = () => writeHeader({});
createdPools[0].respond = () => rowSets({} as unknown as unknown[]);
await expect(client.write('DELETE FROM `user`')).resolves.toEqual({
insertId: 0,
@@ -486,14 +432,17 @@ describe('MySQLDatabaseClient — query interface', () => {
it('builds an INSERT with quoted identifiers and positional params', async () => {
const client = await startClient();
const pool = createdPools[0];
pool.respond = () => writeHeader({ insertId: 7, affectedRows: 1 });
pool.respond = () =>
rowSets({ insertId: 7, affectedRows: 1 } as unknown as unknown[]);
await expect(
client.insert('user', { username: 'ada', email: null }),
).resolves.toMatchObject({ insertId: 7 });
expect(pool.calls[0]).toEqual({
sql: 'INSERT INTO `user` (`username`, `email`) VALUES (?, ?)',
sql:
'INSERT INTO `user` (`username`, `email`) VALUES (?, ?); ' +
'SELECT 1',
values: ['ada', null],
});
});
@@ -744,19 +693,31 @@ describe('MySQLDatabaseClient — pool lifecycle', () => {
expect(createdPools[1].ended).toBe(true);
});
it('leaves the pools open through prepare-shutdown; only onServerShutdown closes them', async () => {
it('closes the pools once the drain window elapses', async () => {
vi.useFakeTimers();
try {
const client = await startClient();
await client.onServerPrepareShutdown();
expect(createdPools[0].ended).toBe(false);
// No timer closes the pools on its own — the layers above are
// still draining through them for as long as the process drains.
await vi.advanceTimersByTimeAsync(60_000);
expect(createdPools[0].ended).toBe(false);
expect(createdPools[0].ended).toBe(true);
} finally {
vi.useRealTimers();
}
});
it('cancels the drain timer when shutdown arrives first', async () => {
vi.useFakeTimers();
try {
const client = await startClient();
await client.onServerPrepareShutdown();
// A second prepare is a no-op — only one drain timer may exist.
await client.onServerPrepareShutdown();
expect(vi.getTimerCount()).toBe(1);
await client.onServerShutdown();
expect(vi.getTimerCount()).toBe(0);
expect(createdPools[0].ended).toBe(true);
} finally {
vi.useRealTimers();
@@ -20,7 +20,7 @@
import { readdirSync, readFileSync } from 'fs';
import { isAbsolute, resolve as resolvePath } from 'path';
import { metrics } from '@opentelemetry/api';
import { createPool, ExecuteValues, type Pool } from 'mysql2';
import { createPool, type Pool } from 'mysql2';
import { Span } from '../../util/span.js';
import { AbstractDatabaseClient, type WriteResult } from './DatabaseClient';
import { SQLBatcher } from './SQLBatcher.js';
@@ -30,8 +30,6 @@ import { compareMigrationFilenames } from './migrationFilenames.js';
import type { IConfig } from '../../types';
const DEFAULT_SELECT_TIMEOUT_MS = 30_000;
const DEFAULT_CONNECTION_LIMIT = 30;
const IDLE_TIMEOUT_MS = 60_000;
const replicaFailoverCounter = metrics
.getMeter('puter-backend')
@@ -56,10 +54,9 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
private replicaPool!: Pool;
private db!: SQLBatcher;
private dbReplica!: SQLBatcher;
/** Primary pool, SELECT-only: same rows as `db`, coalesced like the replica. */
private dbPrimaryRead!: SQLBatcher;
private configuration = Configuration.SINGLE;
private shutdownStarted = false;
private shutdownTimer: ReturnType<typeof setTimeout> | null = null;
constructor(config: IConfig) {
super(config);
@@ -82,7 +79,6 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
console.log('[mysql] connected to primary');
this.db = this.createPrimaryBatcher(this.primaryPool);
this.dbPrimaryRead = this.createPrimaryReadBatcher(this.primaryPool);
if (dbConf.replica) {
this.replicaPool = this.createPool(dbConf.replica);
@@ -102,12 +98,29 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
if (this.shutdownStarted) return;
this.shutdownStarted = true;
// Blocks reinitPrimary/reinitReplica from here on. Pools stay open;
// onServerShutdown closes them after the layers above have drained.
console.log('[mysql] entering drain mode');
// Allow in-flight queries to drain before closing pools
const drainMs = 60_000;
console.log(
`[mysql] draining in-flight queries (${drainMs}ms) before closing pools`,
);
this.shutdownTimer = setTimeout(() => {
this.shutdownTimer = null;
this.closeCurrentPools('drain').catch((e) =>
console.error('[mysql] error closing pools after drain', e),
);
}, drainMs);
if (typeof this.shutdownTimer.unref === 'function') {
this.shutdownTimer.unref();
}
}
override async onServerShutdown(): Promise<void> {
if (this.shutdownTimer) {
clearTimeout(this.shutdownTimer);
this.shutdownTimer = null;
}
await this.closeCurrentPools('shutdown');
}
@@ -138,7 +151,7 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
throw error;
}
replicaFailoverCounter.add(1);
result = await this.dbPrimaryRead.execute(query, params);
result = await this.db.execute(query, params);
}
if (!result) return [];
return (result[0] as Record<string, unknown>[]) ?? [];
@@ -149,7 +162,7 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
const result = await this.dbPrimaryRead.execute(query, params);
const result = await this.db.execute(query, params);
if (!result) return [];
return (result[0] as Record<string, unknown>[]) ?? [];
}
@@ -188,7 +201,7 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
await conn.beginTransaction();
try {
for (const { statement, values } of entries) {
await conn.execute(statement, values as ExecuteValues);
await conn.execute(statement, values);
}
await conn.commit();
} catch (err) {
@@ -211,7 +224,7 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
// Run both reads in parallel — prefer replica when it returns rows,
// otherwise fall back to primary to handle replication lag.
const primaryPromise = this.dbPrimaryRead.execute(query, params);
const primaryPromise = this.db.execute(query, params);
try {
const replicaResult = await this.dbReplica.execute(query, params);
if (
@@ -302,19 +315,9 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
// ------------------------------------------------------------------
private createPool(poolConfig: PoolConfig): Pool {
const connectionLimit =
poolConfig.connectionLimit ?? DEFAULT_CONNECTION_LIMIT;
const pool = createPool({
maxPreparedStatements: 900,
connectionLimit,
// mysql2 only reaps idle connections when maxIdle < connectionLimit.
// Unreaped, they sit until the server's wait_timeout closes them,
// and the next statement sent on one fails. 0 means unlimited,
// and a negative maxIdle would crash the reaper.
...(connectionLimit > 1 && { maxIdle: connectionLimit - 1 }),
idleTimeout: IDLE_TIMEOUT_MS,
// Reaped connections say goodbye instead of dropping the socket.
gracefulEnd: true,
connectionLimit: 30,
enableKeepAlive: true,
...poolConfig,
multipleStatements: true,
@@ -342,26 +345,11 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
return pool;
}
/**
* One write per flush. Coalesced writes share a transaction, so unrelated
* callers hold each other's row locks until COMMIT and deadlock; a lone
* statement needs no BEGIN/COMMIT round trips.
*/
private createPrimaryBatcher(pool: Pool): SQLBatcher {
return new SQLBatcher(pool, {
maxBatchSize: 1,
poolLabel: 'primary',
acquireTimeoutMs: this.config.database?.acquireTimeoutMs,
});
}
/** Reads that must see the primary, still coalesced. */
private createPrimaryReadBatcher(pool: Pool): SQLBatcher {
return new SQLBatcher(pool, {
maxTimeInQueue: 30,
maxBatchSize: 5,
poolLabel: 'primary',
readOnly: true,
acquireTimeoutMs: this.config.database?.acquireTimeoutMs,
});
}
@@ -390,7 +378,6 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
database: dbConf.database ?? 'puter',
});
this.db = this.createPrimaryBatcher(this.primaryPool);
this.dbPrimaryRead = this.createPrimaryReadBatcher(this.primaryPool);
if (this.configuration === Configuration.SINGLE) {
this.replicaPool = this.primaryPool;
@@ -18,7 +18,7 @@
*/
import type { FieldDef, QueryResult } from 'pg';
import { describe, expect, it, vi } from 'vitest';
import { describe, expect, it } from 'vitest';
import type { IConfig } from '../../types';
import { DatabaseClientFactory } from './index.js';
import {
@@ -357,33 +357,4 @@ describe('PostgresDatabaseClient', () => {
]);
expect(conn.released).toBe(true);
});
it('leaves the pool open through prepare-shutdown; only onServerShutdown closes it', async () => {
vi.useFakeTimers();
try {
const pool = new RecordingPool();
let ended = false;
pool.end = async () => {
ended = true;
};
const client = new PostgresDatabaseClient(
postgresConfig(),
() => pool,
);
await client.onServerStart();
await client.onServerPrepareShutdown();
expect(ended).toBe(false);
// No timer closes the pool on its own — the layers above are
// still draining through it for as long as the process drains.
await vi.advanceTimersByTimeAsync(60_000);
expect(ended).toBe(false);
await client.onServerShutdown();
expect(ended).toBe(true);
} finally {
vi.useRealTimers();
}
});
});
@@ -138,6 +138,7 @@ export class PostgresDatabaseClient extends AbstractDatabaseClient {
private replicaPool!: PostgresPool;
private configuration = Configuration.SINGLE;
private shutdownStarted = false;
private shutdownTimer: ReturnType<typeof setTimeout> | null = null;
constructor(
config: IConfig,
@@ -171,12 +172,28 @@ export class PostgresDatabaseClient extends AbstractDatabaseClient {
if (this.shutdownStarted) return;
this.shutdownStarted = true;
// Pools stay open; onServerShutdown closes them after the layers above
// have drained through them.
console.log('[postgres] entering drain mode');
const drainMs = 60_000;
console.log(
`[postgres] draining in-flight queries (${drainMs}ms) before closing pools`,
);
this.shutdownTimer = setTimeout(() => {
this.shutdownTimer = null;
this.closeCurrentPools().catch((e) =>
console.error('[postgres] error closing pools after drain', e),
);
}, drainMs);
if (typeof this.shutdownTimer.unref === 'function') {
this.shutdownTimer.unref();
}
}
override async onServerShutdown(): Promise<void> {
if (this.shutdownTimer) {
clearTimeout(this.shutdownTimer);
this.shutdownTimer = null;
}
await this.closeCurrentPools();
}
+34 -76
View File
@@ -258,11 +258,16 @@ export class SQLBatcher {
}
if (batch.length === 0) return;
const query = `${batch.map((b) => b.sql.replace(/;+\s*$/, '')).join(';')}; SELECT 1`; // SELECT 1 forces mysql2 to return array
const values = batch.map((b) => b.values ?? []).flat();
let connection;
try {
connection = await this.#acquireConnection();
} catch (error) {
this.#recordFlushFailure();
this.#consecutiveFailures++;
this.#lastFailureAt = Date.now();
flushFailureCounter.add(1, this.#metricAttrs);
console.warn(
'SQLBatcher could not acquire connection for flush:',
error,
@@ -273,30 +278,17 @@ export class SQLBatcher {
return;
}
if (batch.length === 1 && !this.readOnly) {
await this.#flushLoneWrite(batch[0], connection);
return;
}
const query = `${batch.map((b) => b.sql.replace(/;+\s*$/, '')).join(';')}; SELECT 1`; // SELECT 1 forces mysql2 to return array
const values = batch.map((b) => b.values ?? []).flat();
// Run the coalesced multi-statement inside an explicit transaction so
// a single bad statement (e.g. a duplicate-key INSERT) rolls back the
// whole batch atomically, leaving us free to re-run each item
// individually below. Without this, MySQL would commit every
// statement up to the failure point and a per-item retry would
// misreport already-committed inserts as duplicate-key failures.
//
// A read-only batch has nothing to roll back, and the wrapper is not
// free: BEGIN and COMMIT are round trips of their own, which triples
// the cost of a read against a pool in another region.
const wrapInTransaction = !this.readOnly;
let batchSucceeded = false;
try {
if (wrapInTransaction) await connection.beginTransaction();
await connection.beginTransaction();
const [results, fields] = await connection.query(query, values);
if (wrapInTransaction) await connection.commit();
await connection.commit();
batchSucceeded = true;
this.#consecutiveFailures = 0;
for (let i = 0; i < batch.length; i++) {
@@ -304,12 +296,10 @@ export class SQLBatcher {
b.resolve([results[i], fields?.[i]]);
}
} catch (batchError) {
if (wrapInTransaction) {
try {
await connection.rollback();
} catch (rollbackError) {
console.warn('SQLBatcher rollback failed:', rollbackError);
}
try {
await connection.rollback();
} catch (rollbackError) {
console.warn('SQLBatcher rollback failed:', rollbackError);
}
console.warn(
'SQLBatcher batch failed; retrying items individually:',
@@ -321,11 +311,10 @@ export class SQLBatcher {
if (batchSucceeded) return;
// Per-item fallback. Nothing from the batch is live — it was rolled
// back, or carried only SELECTs — so re-running each item
// independently produces clean success/failure outcomes for each
// caller. Concurrency is capped to avoid briefly saturating the pool
// when a large batch fails.
// Per-item fallback. The transaction was rolled back so no statement
// committed; re-running each item independently produces clean
// success/failure outcomes for each caller. Concurrency is capped to
// avoid briefly saturating the pool when a large batch fails.
flushFailureCounter.add(1, this.#metricAttrs);
fallbackInvocationsCounter.add(1, this.#metricAttrs);
@@ -336,7 +325,7 @@ export class SQLBatcher {
async () => {
while (cursor < batch.length) {
const i = cursor++;
settled[i] = await this.#runItem(batch[i]);
settled[i] = await this.#runFallbackItem(batch[i]);
}
},
);
@@ -370,34 +359,7 @@ export class SQLBatcher {
}
}
// A single statement is atomic under autocommit, so it skips BEGIN/COMMIT
// and their round trips. With no transaction to roll back, a failure is
// not re-run wholesale; only the per-item retry policy applies.
async #flushLoneWrite(item, connection) {
const r = await this.#runItem(item, connection);
if (r.ok) {
this.#consecutiveFailures = 0;
item.resolve(r.value);
return;
}
// An error the server answered with (duplicate key, bad SQL) means it
// is up; only failing to reach it counts toward the breaker.
if (isRetriableError(r.error)) {
this.#recordFlushFailure();
} else {
this.#consecutiveFailures = 0;
flushFailureCounter.add(1, this.#metricAttrs);
}
item.reject(r.error);
}
#recordFlushFailure() {
this.#consecutiveFailures++;
this.#lastFailureAt = Date.now();
flushFailureCounter.add(1, this.#metricAttrs);
}
// Run one item on its own, retrying transient failures with backoff.
// Run one fallback item, retrying transient failures with backoff.
// A read-only batcher may retry anything transient; a batcher that
// carries writes only retries failures where the statement provably
// did not apply — either it never reached the server, or the server
@@ -408,37 +370,33 @@ export class SQLBatcher {
// than surfacing: an item is a single statement, so a deadlock victim has
// been fully undone, and the caller sees an unhandled 500 for what the
// database is telling us to just run again.
async #runItem(b, connection) {
async #runFallbackItem(b) {
let attempt = 0;
while (true) {
if (!connection) {
try {
connection = await this.#acquireConnection();
} catch (error) {
return { ok: false, error };
}
let connection;
try {
connection = await this.#acquireConnection();
} catch (error) {
return { ok: false, error };
}
let error;
try {
return {
ok: true,
value: await connection.query(b.sql, b.values ?? []),
};
} catch (e) {
error = e;
} catch (error) {
const canRetry = this.readOnly
? isRetriableError(error) || isRolledBackError(error)
: isNeverSentError(error) || isRolledBackError(error);
if (!canRetry || attempt >= ITEM_RETRY_ATTEMPTS) {
return { ok: false, error };
}
attempt++;
fallbackItemRetriesCounter.add(1, this.#metricAttrs);
await sleep(RETRY_BASE_BACKOFF_MS * attempt);
} finally {
connection.release();
connection = undefined;
}
const canRetry = this.readOnly
? isRetriableError(error) || isRolledBackError(error)
: isNeverSentError(error) || isRolledBackError(error);
if (!canRetry || attempt >= ITEM_RETRY_ATTEMPTS) {
return { ok: false, error };
}
attempt++;
fallbackItemRetriesCounter.add(1, this.#metricAttrs);
await sleep(RETRY_BASE_BACKOFF_MS * attempt);
}
}
}
@@ -86,47 +86,8 @@ describe('SQLBatcher', () => {
expect(conn.release).toHaveBeenCalledTimes(1);
});
it('skips the transaction wrapper on a readOnly batcher', async () => {
const conn = makeConnection(happyBatch);
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxTimeInQueue: 5,
readOnly: true,
});
const [a, b] = await Promise.all([
batcher.query('SELECT a', []),
batcher.query('SELECT b', []),
]);
expect(a[0]).toEqual([{ n: 0 }]);
expect(b[0]).toEqual([{ n: 1 }]);
// BEGIN and COMMIT are round trips; a SELECT-only batch has nothing to
// roll back, so it must not pay for them.
expect(conn.beginTransaction).not.toHaveBeenCalled();
expect(conn.commit).not.toHaveBeenCalled();
expect(conn.release).toHaveBeenCalledTimes(1);
});
it('does not roll back a failed readOnly batch', async () => {
const conn = makeConnection((sql: string) => {
if (isBatchQuery(sql)) throw makeError('ER_LOCK_DEADLOCK');
return [[{ n: 0 }], undefined];
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxTimeInQueue: 5,
readOnly: true,
});
await Promise.all([
batcher.query('SELECT a', []),
batcher.query('SELECT b', []),
]);
expect(conn.rollback).not.toHaveBeenCalled();
});
it('drops the oldest item with reason queueOverflow at the high-water mark', async () => {
const conn = makeConnection(() => [[{ n: 0 }], undefined]);
const conn = makeConnection(happyBatch);
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxTimeInQueue: 5,
@@ -277,65 +238,6 @@ describe('SQLBatcher', () => {
);
});
it('sends a lone write as-is, outside a transaction', async () => {
const conn = makeConnection((sql) => {
if (isBatchQuery(sql)) throw new Error('unexpected batch query');
return [{ affectedRows: 1 }, undefined];
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, { maxBatchSize: 1 });
const result = await batcher.query('UPDATE t SET x = ?', [1]);
expect(result[0]).toEqual({ affectedRows: 1 });
expect(conn.query).toHaveBeenCalledTimes(1);
expect(conn.query).toHaveBeenCalledWith('UPDATE t SET x = ?', [1]);
expect(conn.beginTransaction).not.toHaveBeenCalled();
expect(conn.commit).not.toHaveBeenCalled();
expect(conn.release).toHaveBeenCalledTimes(1);
});
it('runs a lone write the server rejects once, without tripping the breaker', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
const conn = makeConnection((sql) => {
if (sql === 'INSERT dup') throw makeError('ER_DUP_ENTRY');
return [{ affectedRows: 1 }, undefined];
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxBatchSize: 1,
failureThreshold: 1,
cooldownMs: 60_000,
});
await expect(batcher.query('INSERT dup', [])).rejects.toMatchObject({
code: 'ER_DUP_ENTRY',
});
expect(conn.query).toHaveBeenCalledTimes(1);
expect(conn.rollback).not.toHaveBeenCalled();
expect(warnSpy).not.toHaveBeenCalled();
// The server answered, so it is up: the next write goes through.
await expect(batcher.query('INSERT fine', [])).resolves.toBeTruthy();
warnSpy.mockRestore();
});
it('retries a lone write on a connection the server closed for inactivity', async () => {
let attempts = 0;
const conn = makeConnection(() => {
attempts++;
if (attempts === 1) {
throw makeError('ER_CLIENT_INTERACTION_TIMEOUT');
}
return [{ affectedRows: 1 }, undefined];
});
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, { maxBatchSize: 1 });
const result = await batcher.query('INSERT x', []);
expect(result[0]).toEqual({ affectedRows: 1 });
expect(attempts).toBe(2);
expect(getConnection).toHaveBeenCalledTimes(2);
});
it('never retries deterministic row-level errors and does not escalate the breaker', async () => {
let fallbackAttempts = 0;
const conn = makeConnection((sql) => {
@@ -18,7 +18,7 @@
*/
import Database from 'better-sqlite3';
import { existsSync, mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { existsSync, mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
@@ -27,15 +27,7 @@ import { DatabaseClientFactory } from './index.js';
import { SqliteDatabaseClient } from './SqliteDatabaseClient.js';
/** Highest schema version the migration table can reach. */
const CURRENT_SCHEMA_VERSION = 86;
/**
* These suites migrate real files on disk. Idle they finish in well under a
* second, but the whole migration chain runs against the filesystem — enough
* that a loaded runner blows the 5s default and reports a timeout where there
* is no fault.
*/
const DISK_MIGRATION_TIMEOUT_MS = 30_000;
const CURRENT_SCHEMA_VERSION = 61;
const SYSTEM_USER_UUID = '5d4adce0-a381-4982-9c02-6e2540026238';
const sqliteConfig = (
@@ -60,7 +52,7 @@ const userVersionOf = async (client: SqliteDatabaseClient): Promise<number> => {
return row.user_version as number;
};
describe('SqliteDatabaseClient — boot and migrations', { timeout: DISK_MIGRATION_TIMEOUT_MS }, () => {
describe('SqliteDatabaseClient — boot and migrations', () => {
let client: SqliteDatabaseClient;
beforeEach(async () => {
@@ -81,312 +73,6 @@ describe('SqliteDatabaseClient — boot and migrations', { timeout: DISK_MIGRATI
expect(await userVersionOf(client)).toBe(CURRENT_SCHEMA_VERSION);
});
it('stamps a version whose migration did not run (known off-by-one)', async () => {
// Existing behaviour, not an endorsement: the stamp overshoots the
// applied work by one entry, so 70 is reported without 0074 running.
const partial = await bootClient({ targetVersion: 70 });
try {
expect(await userVersionOf(partial)).toBe(70);
await expect(
partial.read(
"SELECT 1 FROM pragma_table_info('group') WHERE name = 'handle'",
),
).resolves.toEqual([]);
} finally {
partial.onServerShutdown();
}
});
it('applies the team columns and indexes from 0076', async () => {
const columnsOf = async (table: string) =>
(
(await client.read(
`SELECT name FROM pragma_table_info('${table}')`,
)) as { name: string }[]
).map((r) => r.name);
expect(await columnsOf('group')).toEqual(
expect.arrayContaining([
'kind',
'name',
'handle',
'deleted_at',
]),
);
expect(await columnsOf('jct_user_group')).toContain('org_owned');
expect(await columnsOf('user')).toContain('requires_password_change');
const indexes = (await client.read(
"SELECT name FROM sqlite_master WHERE type = 'index' AND name IN (?, ?, ?)",
['idx_group_handle', 'idx_group_owner', 'idx_jct_user_group_group'],
)) as { name: string }[];
expect(indexes.map((r) => r.name).sort()).toEqual([
'idx_group_handle',
'idx_group_owner',
'idx_jct_user_group_group',
]);
});
it('applies the event_subscriptions indexes from 0081', async () => {
const indexes = (await client.read(
"SELECT name FROM sqlite_master WHERE type = 'index' AND name IN (?, ?, ?)",
[
'idx_event_subscriptions_app_handler',
'idx_event_subscriptions_expires',
'idx_event_subscriptions_suspended',
],
)) as { name: string }[];
expect(indexes.map((r) => r.name).sort()).toEqual([
'idx_event_subscriptions_app_handler',
'idx_event_subscriptions_expires',
'idx_event_subscriptions_suspended',
]);
});
it('keeps `handle` unique but lets the seeded groups share a NULL one', async () => {
const seeded = (await client.read(
'SELECT COUNT(*) AS n FROM `group` WHERE `handle` IS NULL',
)) as { n: number }[];
expect(seeded[0].n).toBeGreaterThan(1);
await client.write(
'UPDATE `group` SET `handle` = ? WHERE `id` = (SELECT MIN(`id`) FROM `group`)',
['taken'],
);
await expect(
client.write(
'UPDATE `group` SET `handle` = ? WHERE `id` = (SELECT MAX(`id`) FROM `group`)',
['taken'],
),
).rejects.toThrow(/UNIQUE/iu);
});
it('treats `handle` case-insensitively, as usernames are treated', async () => {
await client.write(
'UPDATE `group` SET `handle` = ? WHERE `id` = (SELECT MIN(`id`) FROM `group`)',
['design-team'],
);
// Without NOCASE this differs per engine: mysql rejects, the others accept.
await expect(
client.write(
'UPDATE `group` SET `handle` = ? WHERE `id` = (SELECT MAX(`id`) FROM `group`)',
['Design-Team'],
),
).rejects.toThrow(/UNIQUE/iu);
});
it('matches a handle case-insensitively on lookup, not just on insert', async () => {
// Index-only NOCASE would leave `WHERE handle = ?` case-sensitive here while
// mysql's utf8mb4_unicode_ci column matched -- one query, two behaviours.
await client.write(
'UPDATE `group` SET `handle` = ? WHERE `id` = (SELECT MIN(`id`) FROM `group`)',
['design-team'],
);
await expect(
client.read('SELECT `handle` FROM `group` WHERE `handle` = ?', [
'Design-Team',
]),
).resolves.toEqual([{ handle: 'design-team' }]);
});
it('rejects a duplicate membership pair after 0078', async () => {
const [{ user_id, group_id }] = (await client.read(
'SELECT (SELECT MIN(`id`) FROM `user`) AS user_id, ' +
'(SELECT MIN(`id`) FROM `group`) AS group_id',
)) as { user_id: number; group_id: number }[];
const insert = () =>
client.write(
'INSERT INTO `jct_user_group` (`user_id`, `group_id`) VALUES (?, ?)',
[user_id, group_id],
);
await insert();
// `GroupStore.addUsers` has no conflict clause, so before 0075 this
// second call silently doubled every group permission the user reads.
await expect(insert()).rejects.toThrow(/UNIQUE/iu);
});
it('applies the audit table and group-share columns from 0079', async () => {
const columns = (await client.read(
"SELECT name FROM pragma_table_info('audit_team_membership')",
)) as { name: string }[];
expect(columns.map((r) => r.name)).toEqual([
'id',
'group_id',
'group_id_keep',
'user_id',
'user_id_keep',
'actor_user_id',
'action',
'reason',
'created_at',
]);
const shareColumns = (await client.read(
"SELECT name FROM pragma_table_info('share')",
)) as { name: string }[];
expect(shareColumns.map((r) => r.name)).toContain('holder_group_id');
const indexes = (await client.read(
"SELECT name FROM sqlite_master WHERE type = 'index' " +
'AND (name LIKE ? OR name LIKE ?)',
['idx_audit_team_membership%', 'idx_share_holder_group%'],
)) as { name: string }[];
expect(indexes.map((r) => r.name).sort()).toEqual([
// The `_fk` three stop a user or group delete scanning the audit table.
'idx_audit_team_membership_actor_fk',
'idx_audit_team_membership_group',
'idx_audit_team_membership_group_fk',
'idx_audit_team_membership_user',
'idx_audit_team_membership_user_fk',
'idx_share_holder_group',
'idx_share_holder_group_entry_issuer',
]);
});
it('keeps an audit row after its group is deleted, blanking only the FK', async () => {
// 0043 leaves `PRAGMA foreign_keys = ON`, so this is real behaviour.
const [group] = (await client.read(
'SELECT MIN(`id`) AS id FROM `group`',
)) as { id: number }[];
const [user] = (await client.read(
'SELECT MIN(`id`) AS id FROM `user`',
)) as { id: number }[];
await client.write(
'INSERT INTO `audit_team_membership` ' +
'(`group_id`, `group_id_keep`, `user_id`, `user_id_keep`, ' +
'`actor_user_id`, `action`) VALUES (?, ?, ?, ?, ?, ?)',
[group.id, group.id, user.id, user.id, user.id, 'reset_member_password'],
);
await client.write('DELETE FROM `group` WHERE `id` = ?', [group.id]);
await expect(
client.read(
'SELECT `group_id`, `group_id_keep`, `action` FROM `audit_team_membership`',
),
).resolves.toEqual([
{
group_id: null,
group_id_keep: group.id,
action: 'reset_member_password',
},
]);
});
it('deletes the subdomain rows an app owns when the app is deleted', async () => {
const [user] = (await client.read(
'SELECT MIN(`id`) AS id FROM `user`',
)) as { id: number }[];
await client.write(
'INSERT INTO `apps` (`uid`, `owner_user_id`, `name`, `title`, `index_url`) ' +
'VALUES (?, ?, ?, ?, ?)',
[
'app-cascade-test',
user.id,
'cascade-test',
'cascade-test',
'https://cascade.test/',
],
);
const [app] = (await client.read(
'SELECT `id` FROM `apps` WHERE `uid` = ?',
['app-cascade-test'],
)) as { id: number }[];
const insertSubdomain = (
uuid: string,
name: string,
appOwner: number | null,
) =>
client.write(
'INSERT INTO `subdomains` (`uuid`, `subdomain`, `user_id`, `app_owner`) ' +
'VALUES (?, ?, ?, ?)',
[uuid, name, user.id, appOwner],
);
await insertSubdomain('sd-cascade-owned', 'cascade-owned', app.id);
await insertSubdomain('sd-cascade-unowned', 'cascade-unowned', null);
await client.write('DELETE FROM `apps` WHERE `id` = ?', [app.id]);
await expect(
client.read(
'SELECT `subdomain` FROM `subdomains` WHERE `subdomain` LIKE ? ORDER BY `id`',
['cascade-%'],
),
).resolves.toEqual([{ subdomain: 'cascade-unowned' }]);
});
it('keeps every subdomains column through the 0086 rebuild', async () => {
const columns = (await client.read(
"SELECT `name` FROM pragma_table_info('subdomains') ORDER BY `cid`",
)) as { name: string }[];
expect(columns.map((c) => c.name)).toEqual([
'id',
'uuid',
'subdomain',
'user_id',
'root_dir_id',
'associated_app_id',
'ts',
'app_owner',
'protected',
'domain',
'database_id',
'preamble_version',
]);
});
it('constrains team shares that the user-holder index cannot', async () => {
const [user] = (await client.read(
'SELECT MIN(`id`) AS id FROM `user`',
)) as { id: number }[];
const groups = (await client.read(
'SELECT `id` FROM `group` ORDER BY `id` LIMIT 2',
)) as { id: number }[];
await client.write(
'INSERT INTO `fsentries` (`uuid`, `name`, `user_id`, `modified`) ' +
'VALUES (?, ?, ?, ?)',
['11111111-1111-4111-8111-111111111111', 'shared.txt', user.id, 0],
);
const [entry] = (await client.read(
'SELECT `id` FROM `fsentries` WHERE `uuid` = ?',
['11111111-1111-4111-8111-111111111111'],
)) as { id: number }[];
const insertShare = (uid: string, groupId: number) =>
client.write(
'INSERT INTO `share` ' +
'(`uid`, `issuer_user_id`, `recipient_email`, `holder_user_id`, ' +
'`holder_group_id`, `fsentry_id`) VALUES (?, ?, ?, NULL, ?, ?)',
[uid, user.id, '[email protected]', groupId, entry.id],
);
await insertShare('share-team-a', groups[0].id);
// Different group, same file and issuer -- allowed.
await insertShare('share-team-b', groups[1].id);
// Both have `holder_user_id` NULL, so the user-holder index permitted them.
await expect(insertShare('share-team-c', groups[0].id)).rejects.toThrow(
/UNIQUE/iu,
);
});
it('matches a handle case-insensitively on lookup, not just on insert', async () => {
// Index-only NOCASE would leave `WHERE handle = ?` case-sensitive here while
// mysql's utf8mb4_unicode_ci column matched -- one query, two behaviours.
await client.write(
'UPDATE `group` SET `handle` = ? WHERE `id` = (SELECT MIN(`id`) FROM `group`)',
['design-team'],
);
await expect(
client.read('SELECT `handle` FROM `group` WHERE `handle` = ?', [
'Design-Team',
]),
).resolves.toEqual([{ handle: 'design-team' }]);
});
it('runs the javascript migrations, not just the .sql ones', async () => {
// The `system` user only exists because 0025 (a .dbmig.js file) ran
// inside the migration VM.
@@ -397,289 +83,6 @@ describe('SqliteDatabaseClient — boot and migrations', { timeout: DISK_MIGRATI
expect(rows).toEqual([{ username: 'system' }]);
});
// 0070 sweeps three by-hand groups no code reads. Both tables it can reach
// cascade on delete, so the guard is the point: a group that still carries
// permissions or members has to survive, or the sweep silently revokes
// them. Re-running the statement is safe — it is a plain conditional DELETE.
describe('0070 drop-orphaned-default-groups', () => {
const MIGRATION = readFileSync(
new URL(
'./migrations/sqlite/0070_drop-orphaned-default-groups.sql',
import.meta.url,
),
'utf8',
);
const seedGroup = async (name: string): Promise<number> => {
const [system] = await client.read(
'SELECT `id` FROM `user` WHERE `uuid` = ?',
[SYSTEM_USER_UUID],
);
const uid = `grp-${name}-${Math.random().toString(36).slice(2, 10)}`;
await client.write(
'INSERT INTO `group` (`uid`, `owner_user_id`, `extra`, `metadata`) ' +
'VALUES (?, ?, ?, ?)',
[uid, system.id, JSON.stringify({ name }), '{}'],
);
const [row] = await client.read(
'SELECT `id` FROM `group` WHERE `uid` = ?',
[uid],
);
return Number(row.id);
};
const stillThere = async (id: number): Promise<boolean> => {
const rows = await client.read(
'SELECT `id` FROM `group` WHERE `id` = ?',
[id],
);
return rows.length > 0;
};
it('drops an orphaned freeai/experimental/dangerous group', async () => {
const ids = await Promise.all(
['freeai', 'experimental', 'dangerous'].map(seedGroup),
);
await client.write(MIGRATION);
for (const id of ids) expect(await stillThere(id)).toBe(false);
});
it('spares one that still carries a permission row', async () => {
const id = await seedGroup('freeai');
const [system] = await client.read(
'SELECT `id` FROM `user` WHERE `uuid` = ?',
[SYSTEM_USER_UUID],
);
await client.write(
'INSERT INTO `user_to_group_permissions` ' +
'(`user_id`, `group_id`, `permission`, `extra`) VALUES (?, ?, ?, ?)',
[system.id, id, 'service:some-paid-thing', '{}'],
);
await client.write(MIGRATION);
expect(await stillThere(id)).toBe(true);
// And the grant it carries is still intact, not cascaded away.
const perms = await client.read(
'SELECT `permission` FROM `user_to_group_permissions` WHERE `group_id` = ?',
[id],
);
expect(perms).toEqual([{ permission: 'service:some-paid-thing' }]);
});
it('spares one that still has members', async () => {
const id = await seedGroup('experimental');
const [system] = await client.read(
'SELECT `id` FROM `user` WHERE `uuid` = ?',
[SYSTEM_USER_UUID],
);
await client.write(
'INSERT INTO `jct_user_group` (`user_id`, `group_id`) VALUES (?, ?)',
[system.id, id],
);
await client.write(MIGRATION);
expect(await stillThere(id)).toBe(true);
});
it('leaves the groups the platform actually depends on alone', async () => {
// system, admin, user and temp are named by config or code.
await client.write(MIGRATION);
const rows = await client.read(
"SELECT json_extract(`extra`, '$.name') AS name FROM `group`",
);
const names = rows.map((r) => String(r.name));
for (const name of ['system', 'admin', 'user']) {
expect(names).toContain(name);
}
});
});
// 0072 rebuilds `notification` for the scope tuple, and picks up the
// user_id index and the delete cascade mysql and postgres already had.
describe('0072 notification-scope', () => {
// Only the backfill re-runs: the rebuild half is once-only by
// construction, but mysql and postgres replay their whole file on
// every boot, so these statements have to be safe twice.
const BACKFILL = readFileSync(
new URL(
'./migrations/sqlite/0072_notification-scope.sql',
import.meta.url,
),
'utf8',
)
.split(/;\s*\n/)
.map((s) => s.trim())
.filter((s) => /(^|\n)UPDATE /.test(s));
const seedUser = async (): Promise<number> => {
const name = `notif-${Math.random().toString(36).slice(2, 10)}`;
await client.write(
'INSERT INTO `user` (`username`, `uuid`) VALUES (?, ?)',
[name, `${name}-uuid`],
);
const [row] = await client.read(
'SELECT `id` FROM `user` WHERE `username` = ?',
[name],
);
return Number(row.id);
};
const seedLegacy = async (
userId: number,
value: Record<string, unknown> | string,
): Promise<string> => {
const uid = `n-${Math.random().toString(36).slice(2, 12)}`;
await client.write(
'INSERT INTO `notification` (`uid`, `user_id`, `value`) VALUES (?, ?, ?)',
[
uid,
userId,
typeof value === 'string' ? value : JSON.stringify(value),
],
);
return uid;
};
const scopeOf = async (uid: string) => {
const [row] = await client.read(
'SELECT `type`, `audience`, `app_uid` FROM `notification` WHERE `uid` = ?',
[uid],
);
return row;
};
const runBackfill = async () => {
for (const stmt of BACKFILL) await client.write(stmt);
};
it('carries every backfill statement', () => {
expect(BACKFILL).toHaveLength(3);
});
it('classifies the markers legacy rows actually carried', async () => {
const userId = await seedUser();
const received = await seedLegacy(userId, {
source: 'sharing',
template: 'file-shared-with-you',
});
const claimed = await seedLegacy(userId, {
source: 'sharing',
template: 'file-shared-before-you-joined',
});
const deployed = await seedLegacy(userId, {
source: 'worker',
title: 'Successfully deployed https://x.puter.work',
template: 'user-requesting-share',
});
const failed = await seedLegacy(userId, {
source: 'worker',
title: 'Failed to deploy x! boom',
template: 'user-requesting-share',
});
await runBackfill();
expect(await scopeOf(received)).toEqual({
type: 'share.received',
audience: 'account',
app_uid: null,
});
expect(await scopeOf(claimed)).toEqual({
type: 'share.claimed',
audience: 'account',
app_uid: null,
});
// The app is unrecoverable — the payload only ever named a worker.
expect(await scopeOf(deployed)).toEqual({
type: 'app.worker.deployed',
audience: 'developer',
app_uid: null,
});
expect(await scopeOf(failed)).toEqual({
type: 'app.worker.deployFailed',
audience: 'developer',
app_uid: null,
});
});
it('leaves anything it does not recognise reading as legacy', async () => {
const userId = await seedUser();
const unknown = await seedLegacy(userId, {
source: 'sharing',
template: 'something-else',
});
const bare = await seedLegacy(userId, { title: 'hi' });
const notJson = await seedLegacy(userId, 'plain text');
await runBackfill();
for (const uid of [unknown, bare, notJson]) {
expect(await scopeOf(uid)).toEqual({
type: '',
audience: 'account',
app_uid: null,
});
}
});
it('does not reclassify on replay', async () => {
const userId = await seedUser();
const uid = await seedLegacy(userId, {
source: 'sharing',
template: 'file-shared-with-you',
});
await runBackfill();
// A row already classified as something else must survive a
// second pass untouched.
await client.write(
'UPDATE `notification` SET `audience` = ?, `app_uid` = ? WHERE `uid` = ?',
['app-user', 'app-1234', uid],
);
await runBackfill();
expect(await scopeOf(uid)).toEqual({
type: 'share.received',
audience: 'app-user',
app_uid: 'app-1234',
});
});
it('indexes user_id and the scope tuple', async () => {
const rows = await client.read('PRAGMA index_list(`notification`)');
const names = rows.map((r) => String(r.name));
expect(names).toContain('idx_notification_user_id');
expect(names).toContain('idx_notification_scope');
});
it('retires a deleted user rows', async () => {
const [{ foreign_keys: enforcing }] = await client.read(
'PRAGMA foreign_keys',
);
expect(enforcing).toBe(1);
const userId = await seedUser();
const uid = await seedLegacy(userId, { title: 'orphan-to-be' });
await client.write('DELETE FROM `user` WHERE `id` = ?', [userId]);
expect(
await client.read(
'SELECT `uid` FROM `notification` WHERE `uid` = ?',
[uid],
),
).toEqual([]);
});
});
it('indexes notification.created_at for the retention sweep', async () => {
const rows = await client.read('PRAGMA index_list(`notification`)');
expect(rows.map((r) => String(r.name))).toContain(
'idx_notification_created_at',
);
});
it('leaves an already-migrated database untouched on a second boot', async () => {
const dir = mkdtempSync(join(tmpdir(), 'puter-sqlite-'));
const path = join(dir, 'nested', 'puter.sqlite');
@@ -713,66 +116,6 @@ describe('SqliteDatabaseClient — boot and migrations', { timeout: DISK_MIGRATI
}
});
it('deduplicates pre-existing membership rows when 0078 applies', async () => {
const dir = mkdtempSync(join(tmpdir(), 'puter-sqlite-dedup-'));
const path = join(dir, 'puter.sqlite');
try {
// Stop at 67 so the duplicates exist the way a live database's do:
// written before the unique index, not after it.
const before = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path, targetVersion: 67 }),
);
await before.onServerStart();
const [{ user_id, group_id }] = (await before.read(
'SELECT (SELECT MIN(`id`) FROM `user`) AS user_id, ' +
'(SELECT MIN(`id`) FROM `group`) AS group_id',
)) as { user_id: number; group_id: number }[];
for (const pair of [
[user_id, group_id],
[user_id, group_id],
[user_id, group_id],
[user_id, group_id + 1],
]) {
await before.write(
'INSERT INTO `jct_user_group` (`user_id`, `group_id`) VALUES (?, ?)',
pair,
);
}
const seeded = (await before.read(
'SELECT `id`, `group_id` FROM `jct_user_group` ORDER BY `id`',
)) as { id: number; group_id: number }[];
expect(seeded).toHaveLength(4);
before.onServerShutdown();
const after = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path }),
);
await after.onServerStart();
expect(await userVersionOf(after)).toBe(CURRENT_SCHEMA_VERSION);
// Three copies collapse to the lowest id; the distinct pair is left
// alone. Losing the higher ids costs nothing -- `addUsers` writes
// only the two id columns, so `extra` and `metadata` are NULL.
await expect(
after.read(
'SELECT `id`, `group_id` FROM `jct_user_group` ORDER BY `id`',
),
).resolves.toEqual([seeded[0], seeded[3]]);
await expect(
after.write(
'INSERT INTO `jct_user_group` (`user_id`, `group_id`) VALUES (?, ?)',
[user_id, group_id],
),
).rejects.toThrow(/UNIQUE/iu);
after.onServerShutdown();
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it('stops early at a configured target version', async () => {
const partial = await bootClient({ targetVersion: 5 });
try {
@@ -794,10 +137,7 @@ describe('SqliteDatabaseClient — boot and migrations', { timeout: DISK_MIGRATI
});
});
describe(
'SqliteDatabaseClient — legacy version inference',
{ timeout: DISK_MIGRATION_TIMEOUT_MS },
() => {
describe('SqliteDatabaseClient — legacy version inference', () => {
let dir: string;
beforeEach(() => {
@@ -95,31 +95,6 @@ const AVAILABLE_MIGRATIONS: [number, string[]][] = [
[58, ['0063_add_suspended_reason.sql']],
[59, ['0064_abuse-moderation-events.sql']],
[60, ['0065_app-feedback.sql']],
[61, ['0066_owned-email-unique.sql']],
[62, ['0067_share_entries.sql']],
[63, ['0068_referral-code-unique.sql']],
[64, ['0069_user-block.sql']],
[65, ['0070_drop-orphaned-default-groups.sql']],
[66, ['0071_share_issuer_index.sql']],
[67, ['0072_notification-scope.sql']],
[68, ['0073_notification-created-at.sql']],
[69, ['0074_add_user_home.sql']],
[70, ['0075_event-subscriptions.sql']],
[71, ['0076_event-handlers.sql']],
[72, ['0077_teams.sql']],
[73, ['0078_jct-user-group-pair-unique.sql']],
[74, ['0079_team-audit-and-group-shares.sql']],
[75, ['0080_kv-share-handles.sql']],
[76, ['0081_event-subscriptions-indexes.sql']],
[77, ['0082_temp-password-expiry.sql']],
[78, ['0083_team-directory.sql']],
[79, ['0084_share-anyone-with-link.sql']],
[80, ['0085_event-subscriptions-include-value.sql']],
[81, ['0086_subdomains-app-owner-cascade.sql']],
[82, ['0087_feedback-attachments.sql']],
[83, ['0088_apps-index-url.sql']],
[84, ['0089_team-require-2fa.sql']],
[85, ['0090_user-app-permission-index.sql']],
];
export class SqliteDatabaseClient extends AbstractDatabaseClient {
@@ -1060,7 +1060,6 @@ CREATE TABLE IF NOT EXISTS `user` (
`signup_server` varchar(255) DEFAULT NULL,
`metadata` json DEFAULT (json_object()),
`reputation` smallint DEFAULT '100',
`home` varchar(64) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `uid` (`uuid`),
UNIQUE KEY `username` (`username`),
@@ -1152,7 +1151,6 @@ CALL _puter_add_col('user', 'signup_user_agent', '`signup_user_agent` varchar(51
CALL _puter_add_col('user', 'signup_origin', '`signup_origin` varchar(255) DEFAULT NULL');
CALL _puter_add_col('user', 'signup_server', '`signup_server` varchar(255) DEFAULT NULL');
CALL _puter_add_col('user', 'metadata', '`metadata` json DEFAULT (json_object())');
CALL _puter_add_col('user', 'home', '`home` varchar(64) DEFAULT NULL');
CALL _puter_add_col('user', 'reputation', '`reputation` smallint DEFAULT ''100''');
--
@@ -1,128 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Enforce "at most one account owns an email address". Mirrors SQLite
-- migration 0066.
--
-- `user.email` is deliberately not UNIQUE: several rows may legitimately hold
-- the same address while unconfirmed (admin-provisioned placeholders, signups
-- that were never confirmed, temp accounts on their way to becoming real). What
-- must never happen is two rows both *owning* an address — owning meaning the
-- row is confirmed, or holds a password and so can drive password recovery for
-- that inbox.
--
-- Signup, save-account, change-email, OIDC and admin provisioning each check for
-- an owner before writing, but a check and an insert are not one operation: two
-- requests can both read "free" and both write. This index is what actually
-- holds the invariant; the application checks just produce a nicer error most of
-- the time.
--
-- SQLite expresses that with a partial index. MySQL has none, so the predicate
-- lives in a generated column that evaluates to NULL for every row that does not
-- own its address — and NULLs do not collide in an InnoDB unique index, which is
-- exactly the "unlimited unconfirmed placeholders" behaviour we need.
--
-- The column is VIRTUAL, not STORED, on purpose: adding a stored generated
-- column rebuilds the table, while a virtual one is a metadata-only change and
-- the index that follows builds INPLACE. On a `user` table of any size with
-- read replicas attached, that is the difference between a routine change and an
-- outage. The ALGORITHM/LOCK clauses are spelled out so a server that cannot
-- honour them refuses the statement instead of quietly copying the table.
--
-- Matching is on the canonical address so provider aliases
-- (`[email protected]` vs `[email protected]`) collide. `clean_email` is
-- written on every modern write path; the COALESCE covers rows old enough to
-- predate the column. Run the `clean_email` backfill before this migration or
-- alias collisions among those rows go unnoticed.
--
-- Idempotent: both steps are guarded on INFORMATION_SCHEMA so the directory
-- replays safely.
--
-- If the index creation fails with ER_DUP_ENTRY, the DB already contains
-- duplicate owners. Collapse them first (admin → One-off Jobs → Collapse
-- Duplicate Emails) — there is no safe automatic merge of two accounts.
DROP PROCEDURE IF EXISTS _puter_add_owned_email;
DELIMITER //
CREATE PROCEDURE _puter_add_owned_email()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user'
AND COLUMN_NAME = 'owned_email'
) THEN
ALTER TABLE `user`
ADD COLUMN `owned_email` VARCHAR(256)
CHARACTER SET latin1 COLLATE latin1_swedish_ci
GENERATED ALWAYS AS (
CASE
WHEN `email` IS NOT NULL
AND (`email_confirmed` = 1 OR `password` IS NOT NULL)
THEN COALESCE(`clean_email`, LOWER(`email`))
ELSE NULL
END
) VIRTUAL,
ALGORITHM=INSTANT;
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user'
AND INDEX_NAME = 'idx_user_owned_email'
) THEN
ALTER TABLE `user`
ADD UNIQUE KEY `idx_user_owned_email` (`owned_email`),
ALGORITHM=INPLACE, LOCK=NONE;
END IF;
END//
DELIMITER ;
CALL _puter_add_owned_email();
DROP PROCEDURE IF EXISTS _puter_add_owned_email;
-- One Puter account per external identity. OIDCStore.link already assumes this
-- constraint exists — it catches the unique violation to tell "re-linking the
-- same account" apart from "this sub belongs to someone else" — but the table
-- never actually had it, so two concurrent first-time logins could each create
-- an account and each link the same sub. Subsequent logins then resolved to
-- whichever row came back first.
--
-- Dedupe `user_oidc_providers` before applying this: keep the lowest `id` per
-- (provider, provider_sub) and point it at the account the collapse job kept.
DROP PROCEDURE IF EXISTS _puter_add_oidc_sub_unique;
DELIMITER //
CREATE PROCEDURE _puter_add_oidc_sub_unique()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user_oidc_providers'
AND INDEX_NAME = 'idx_user_oidc_provider_sub'
) THEN
ALTER TABLE `user_oidc_providers`
ADD UNIQUE KEY `idx_user_oidc_provider_sub` (`provider`, `provider_sub`);
END IF;
END//
DELIMITER ;
CALL _puter_add_oidc_sub_unique();
DROP PROCEDURE IF EXISTS _puter_add_oidc_sub_unique;
@@ -1,106 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Grow `share` from a pending-email-invite table into the index of active
-- shares. See sqlite/0067_share_entries.sql for the column rationale.
--
-- Idempotent: columns go through _puter_add_col (from mig_1), indexes and
-- foreign keys through the guarded procedure below. There is no per-file
-- applied-state tracking, so every statement must tolerate a re-run.
CALL _puter_add_col('share', 'holder_user_id', '`holder_user_id` int unsigned DEFAULT NULL');
CALL _puter_add_col('share', 'fsentry_id', '`fsentry_id` int unsigned DEFAULT NULL');
CALL _puter_add_col('share', 'mode', '`mode` varchar(20) DEFAULT NULL');
CALL _puter_add_col('share', 'applied_at', '`applied_at` timestamp NULL DEFAULT NULL');
DROP PROCEDURE IF EXISTS _puter_add_share_index_constraints;
DELIMITER //
CREATE PROCEDURE _puter_add_share_index_constraints()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_holder'
) THEN
ALTER TABLE `share` ADD INDEX `idx_share_holder` (`holder_user_id`, `id`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_fsentry'
) THEN
ALTER TABLE `share` ADD INDEX `idx_share_fsentry` (`fsentry_id`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_holder_entry_issuer'
) THEN
ALTER TABLE `share` ADD UNIQUE INDEX `idx_share_holder_entry_issuer`
(`holder_user_id`, `fsentry_id`, `issuer_user_id`);
END IF;
-- Retiring a deleted node's grants looks them up by permission text
-- (`fs:<uuid>` and `fs:<uuid>:%`); the subject lives in that text rather
-- than a column, so no foreign key can cascade it. The primary key is
-- (issuer, holder, permission), which never puts `permission` first, so
-- without this both the equality and the left-anchored LIKE degrade to a
-- full scan on the delete path. The column is ascii varchar(255), well
-- inside InnoDB's key limit, so it is indexed whole.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user_to_user_permissions'
AND INDEX_NAME = 'idx_user_to_user_permissions_permission'
) THEN
ALTER TABLE `user_to_user_permissions`
ADD INDEX `idx_user_to_user_permissions_permission` (`permission`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND CONSTRAINT_NAME = 'share_holder_user_fk'
) THEN
ALTER TABLE `share` ADD CONSTRAINT `share_holder_user_fk`
FOREIGN KEY (`holder_user_id`) REFERENCES `user` (`id`)
ON DELETE CASCADE ON UPDATE CASCADE;
END IF;
-- The cascade that retires a share with its file.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND CONSTRAINT_NAME = 'share_fsentry_fk'
) THEN
ALTER TABLE `share` ADD CONSTRAINT `share_fsentry_fk`
FOREIGN KEY (`fsentry_id`) REFERENCES `fsentries` (`id`)
ON DELETE CASCADE ON UPDATE CASCADE;
END IF;
END//
DELIMITER ;
CALL _puter_add_share_index_constraints();
DROP PROCEDURE IF EXISTS _puter_add_share_index_constraints;
@@ -1,35 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- One user refusing contact from another. See sqlite/0068_user-block.sql for
-- the rationale. `created_at` is unix seconds.
--
-- Idempotent: `CREATE TABLE IF NOT EXISTS` with the indexes declared inline,
-- so the directory can replay safely.
CREATE TABLE IF NOT EXISTS `user_block` (
`id` INT NOT NULL AUTO_INCREMENT,
`blocker_user_id` INT UNSIGNED NOT NULL,
`blocked_user_id` INT UNSIGNED NOT NULL,
`created_at` BIGINT NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_user_block_pair` (`blocker_user_id`, `blocked_user_id`),
CONSTRAINT `fk_user_block_blocker` FOREIGN KEY (`blocker_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_user_block_blocked` FOREIGN KEY (`blocked_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
@@ -1,61 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Widen `notification.shown` and `notification.acknowledged` to hold the unix
-- second the store has been writing to them since the backend rework.
--
-- Both arrived from the v1 schema as `tinyint(1)`, where they were flags. The
-- rework changed the writes to timestamps and sqlite (`INTEGER`) and postgres
-- (`bigint`) took them; only mysql was left too narrow, so every `markShown`
-- and `markAcknowledged` there fails with ER_WARN_DATA_OUT_OF_RANGE and the
-- column stays NULL. Dismissing a notification therefore never sticks, and one
-- already delivered is re-sent on every reconnect.
--
-- No backfill: every reader tests `IS NULL` / `IS NOT NULL` only, so a legacy
-- `1` keeps meaning "yes" once widened.
--
-- Guarded on the current type rather than run unconditionally. Changing a
-- column type copies the table, and this one grows without bound — a migration
-- directory that replays on every boot must not rebuild it every time.
DROP PROCEDURE IF EXISTS _puter_widen_notification_stamps;
DELIMITER //
CREATE PROCEDURE _puter_widen_notification_stamps()
BEGIN
IF EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'notification'
AND COLUMN_NAME = 'shown'
AND DATA_TYPE = 'tinyint'
) THEN
ALTER TABLE `notification` MODIFY `shown` BIGINT DEFAULT NULL;
END IF;
IF EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'notification'
AND COLUMN_NAME = 'acknowledged'
AND DATA_TYPE = 'tinyint'
) THEN
ALTER TABLE `notification` MODIFY `acknowledged` BIGINT DEFAULT NULL;
END IF;
END //
DELIMITER ;
CALL _puter_widen_notification_stamps();
DROP PROCEDURE IF EXISTS _puter_widen_notification_stamps;
@@ -1,22 +0,0 @@
-- Drops three groups that no code has ever read: `freeai`, `experimental` and
-- `dangerous`. No migration creates them — they were added by hand back when
-- hardcoded permissions were keyed by group name. That map is now a flat
-- per-user floor (`default_user_permissions` in data/hardcoded-permissions.js),
-- so a group nothing looks up grants nothing, and the rows are dead weight.
--
-- Guarded, and deliberately so: `user_to_group_permissions.group_id` and
-- `jct_user_group.group_id` both cascade on delete, so removing a group that
-- still carries permissions or members would silently revoke them from every
-- member. Only a group with neither is dropped. A group that survives this
-- migration has dependents and needs a deliberate decision, not a sweep --
-- query `user_to_group_permissions` by `group_id` to see what it holds.
--
-- The joins rather than `NOT IN` subqueries: MySQL will not read from the table
-- it is deleting from inside a subquery on the same statement.
DELETE g FROM `group` g
LEFT JOIN `user_to_group_permissions` p ON p.`group_id` = g.`id`
LEFT JOIN `jct_user_group` j ON j.`group_id` = g.`id`
WHERE JSON_UNQUOTE(JSON_EXTRACT(g.`extra`, '$.name'))
IN ('freeai', 'experimental', 'dangerous')
AND p.`group_id` IS NULL
AND j.`group_id` IS NULL;
@@ -1,80 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Give `notification` a scope tuple: what the row is (`type`), the app it is
-- about (`app_uid`, NULL for platform), and who the recipient is in relation
-- to it (`audience`). Attribution lived inside the `value` JSON as two
-- free-form markers, `source` and `template`, that could disagree; these
-- columns replace them and are what a scoped list can be indexed on.
--
-- `app_uid` is char(40) to match `apps`.`uid`, which is `app-` plus a uuid.
--
-- Idempotent: columns go through `_puter_add_col` (mig_1), the index through
-- the guarded procedure. There is no per-file applied-state tracking, so every
-- statement here tolerates a re-run.
CALL _puter_add_col('notification', 'app_uid', '`app_uid` char(40) COLLATE utf8mb4_unicode_ci DEFAULT NULL');
CALL _puter_add_col('notification', 'audience', '`audience` varchar(16) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT ''account''');
CALL _puter_add_col('notification', 'type', '`type` varchar(64) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT ''''');
DROP PROCEDURE IF EXISTS _puter_add_notification_scope_index;
DELIMITER //
CREATE PROCEDURE _puter_add_notification_scope_index()
BEGIN
-- Serves both the scoped list and the unacknowledged count.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'notification'
AND INDEX_NAME = 'idx_notification_scope'
) THEN
ALTER TABLE `notification` ADD INDEX `idx_notification_scope`
(`user_id`, `audience`, `app_uid`, `acknowledged`);
END IF;
END //
DELIMITER ;
CALL _puter_add_notification_scope_index();
DROP PROCEDURE IF EXISTS _puter_add_notification_scope_index;
-- Backfill from the markers the payload actually carried. `type = ''` is the
-- not-yet-classified state, so re-running matches nothing the first pass
-- claimed, and a row matching no marker keeps the defaults (`''` / 'account')
-- and reads as legacy.
UPDATE `notification`
SET `type` = 'share.received', `audience` = 'account', `app_uid` = NULL
WHERE `type` = ''
AND JSON_UNQUOTE(JSON_EXTRACT(`value`, '$.template')) = 'file-shared-with-you';
UPDATE `notification`
SET `type` = 'share.claimed', `audience` = 'account', `app_uid` = NULL
WHERE `type` = ''
AND JSON_UNQUOTE(JSON_EXTRACT(`value`, '$.template')) = 'file-shared-before-you-joined';
-- Worker rows carried only `source: 'worker'` and a title, so the app they
-- belong to is unrecoverable and they stay unattributed. The title prefix is
-- the sole surviving signal of which way the deploy went.
UPDATE `notification`
SET `audience` = 'developer',
`app_uid` = NULL,
`type` = CASE
WHEN JSON_UNQUOTE(JSON_EXTRACT(`value`, '$.title')) LIKE 'Successfully deployed %'
THEN 'app.worker.deployed'
ELSE 'app.worker.deployFailed'
END
WHERE `type` = ''
AND JSON_UNQUOTE(JSON_EXTRACT(`value`, '$.source')) = 'worker';
@@ -1,42 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Index `notification`.`created_at`. The retention sweep selects by age alone
-- and by nothing else, so no existing index on the table narrows it — every
-- pass would otherwise scan the whole table to find the oldest few hundred
-- rows.
--
-- Idempotent: the guarded procedure, as mig_26. There is no per-file
-- applied-state tracking, so a replay has to be a no-op.
DROP PROCEDURE IF EXISTS _puter_add_notification_created_at_index;
DELIMITER //
CREATE PROCEDURE _puter_add_notification_created_at_index()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'notification'
AND INDEX_NAME = 'idx_notification_created_at'
) THEN
ALTER TABLE `notification` ADD INDEX `idx_notification_created_at`
(`created_at`);
END IF;
END //
DELIMITER ;
CALL _puter_add_notification_created_at_index();
DROP PROCEDURE IF EXISTS _puter_add_notification_created_at_index;
@@ -1,25 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Home region column. Mirrors SQLite migration 0074. Home region this
-- account's data belongs in — a node id, not a storage region. NULL for
-- accounts that predate the column, which fall back to `signup_server`.
--
-- Idempotent: the column add uses _puter_add_col (defined in mig_1, which
-- leaves it resident for later migrations).
CALL _puter_add_col('user', 'home', '`home` VARCHAR(64) DEFAULT NULL');
@@ -1,59 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Subscriptions that outlive the connection that made them. See
-- sqlite/0075_event-subscriptions.sql for the column rationale.
--
-- Idempotent: `CREATE TABLE IF NOT EXISTS` with the indexes declared inline,
-- as mig_23. There is no per-file applied-state tracking, so a replay has to
-- be a no-op.
CREATE TABLE IF NOT EXISTS `event_subscriptions` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`sub_id` varchar(80) NOT NULL,
`token` varchar(255) NOT NULL,
`owner_user_id` int unsigned NOT NULL,
`holder_user_id` int unsigned NOT NULL,
-- Matches `apps`.`uid` exactly, charset included, so an equality against
-- one never falls back to a conversion. No foreign key: a row outlives the
-- app that made it, which is what keeps it revocable afterwards.
`app_uid` char(40) CHARACTER SET latin1 COLLATE latin1_swedish_ci
DEFAULT NULL,
`subject` varchar(4096) NOT NULL,
`anchor_uid` char(36) NOT NULL,
`anchor_path` varchar(4096) NOT NULL,
`match` varchar(1024) DEFAULT NULL,
`delivery` varchar(16) NOT NULL,
`ops` varchar(64) DEFAULT NULL,
`handler_name` varchar(128) DEFAULT NULL,
`targets` json NOT NULL,
`context` text,
`permission` varchar(32) NOT NULL,
`suspended_at` bigint DEFAULT NULL,
`suspended_reason` varchar(64) DEFAULT NULL,
`expires_at` bigint DEFAULT NULL,
`created_at` bigint NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_event_subscriptions_sub_id` (`sub_id`),
KEY `idx_event_subscriptions_token` (`token`),
KEY `idx_event_subscriptions_holder` (`holder_user_id`, `app_uid`),
KEY `idx_event_subscriptions_owner` (`owner_user_id`),
CONSTRAINT `fk_event_subscriptions_owner` FOREIGN KEY (`owner_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_event_subscriptions_holder` FOREIGN KEY (`holder_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
@@ -1,43 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- `event_subscriptions`.`anchor_uid` was sized for a filesystem node uuid.
-- A key-value subscription anchors on an app instead, and `apps`.`uid` is
-- `char(40)`, so the column has to hold one.
--
-- Idempotent: the guarded procedure, as mig_27. There is no per-file
-- applied-state tracking, so a replay has to be a no-op.
DROP PROCEDURE IF EXISTS _puter_widen_event_subscriptions_anchor_uid;
DELIMITER //
CREATE PROCEDURE _puter_widen_event_subscriptions_anchor_uid()
BEGIN
IF EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'event_subscriptions'
AND COLUMN_NAME = 'anchor_uid'
AND CHARACTER_MAXIMUM_LENGTH < 40
) THEN
ALTER TABLE `event_subscriptions`
MODIFY COLUMN `anchor_uid` varchar(40) NOT NULL;
END IF;
END //
DELIMITER ;
CALL _puter_widen_event_subscriptions_anchor_uid();
DROP PROCEDURE IF EXISTS _puter_widen_event_subscriptions_anchor_uid;
@@ -1,37 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Named handlers an app deploys once. See sqlite/0076_event-handlers.sql for
-- the column rationale.
--
-- Idempotent: `CREATE TABLE IF NOT EXISTS` with the indexes declared inline,
-- as mig_29. There is no per-file applied-state tracking, so a replay has to
-- be a no-op.
CREATE TABLE IF NOT EXISTS `event_handlers` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
-- Matches `apps`.`uid` exactly, charset included, so an equality against
-- one never falls back to a conversion.
`app_uid` char(40) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL,
`name` varchar(128) NOT NULL,
`source` mediumtext NOT NULL,
`source_hash` char(64) NOT NULL,
`created_at` bigint NOT NULL,
`updated_at` bigint NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_event_handlers_app_name` (`app_uid`, `name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
@@ -1,58 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Teams. See sqlite/0077_teams.sql for the column rationale.
-- No per-file applied-state tracking, so every statement must tolerate a re-run:
-- columns go through _puter_add_col, indexes through the guarded procedure below.
CALL _puter_add_col('group', 'kind', '`kind` varchar(20) DEFAULT NULL');
CALL _puter_add_col('group', 'name', '`name` varchar(255) DEFAULT NULL');
CALL _puter_add_col('group', 'handle', '`handle` varchar(64) DEFAULT NULL');
CALL _puter_add_col('group', 'deleted_at', '`deleted_at` timestamp NULL DEFAULT NULL');
CALL _puter_add_col('jct_user_group', 'org_owned', '`org_owned` tinyint(1) DEFAULT NULL');
CALL _puter_add_col('user', 'requires_password_change', '`requires_password_change` tinyint(1) NOT NULL DEFAULT ''0''');
DROP PROCEDURE IF EXISTS _puter_add_team_indexes;
DELIMITER //
CREATE PROCEDURE _puter_add_team_indexes()
BEGIN
-- Already case-insensitive: the table is utf8mb4_unicode_ci. NULLs stay distinct.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'group'
AND INDEX_NAME = 'idx_group_handle'
) THEN
ALTER TABLE `group` ADD UNIQUE INDEX `idx_group_handle` (`handle`);
END IF;
-- Composite, unlike the existing single-column `group_id` key.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'jct_user_group'
AND INDEX_NAME = 'idx_jct_user_group_group'
) THEN
ALTER TABLE `jct_user_group`
ADD INDEX `idx_jct_user_group_group` (`group_id`, `user_id`);
END IF;
END//
DELIMITER ;
CALL _puter_add_team_indexes();
DROP PROCEDURE IF EXISTS _puter_add_team_indexes;
@@ -1,47 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Deduplicate then constrain. See sqlite/0075 for why this is lossless.
-- Both steps sit behind the index check: mysql re-runs this file on every boot, and
-- guarding them together stops a rolling deploy deleting and indexing concurrently.
DROP PROCEDURE IF EXISTS _puter_add_membership_pair_index;
DELIMITER //
CREATE PROCEDURE _puter_add_membership_pair_index()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'jct_user_group'
AND INDEX_NAME = 'idx_jct_user_group_pair'
) THEN
-- Self-join because mysql refuses `NOT IN (SELECT ... FROM jct_user_group)` (1093).
DELETE dup FROM `jct_user_group` dup
JOIN `jct_user_group` keep
ON dup.`user_id` = keep.`user_id`
AND dup.`group_id` = keep.`group_id`
AND dup.`id` > keep.`id`;
ALTER TABLE `jct_user_group`
ADD UNIQUE INDEX `idx_jct_user_group_pair` (`user_id`, `group_id`);
END IF;
END//
DELIMITER ;
CALL _puter_add_membership_pair_index();
DROP PROCEDURE IF EXISTS _puter_add_membership_pair_index;
@@ -1,84 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0076. Types follow mysql_mig_1; the guarded procedure follows mysql_mig_22.
CREATE TABLE IF NOT EXISTS `audit_team_membership` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`group_id` int unsigned DEFAULT NULL,
`group_id_keep` int unsigned NOT NULL,
`user_id` int unsigned DEFAULT NULL,
`user_id_keep` int unsigned NOT NULL,
`actor_user_id` int unsigned DEFAULT NULL,
`action` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`reason` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_audit_team_membership_group` (`group_id_keep`, `id`),
KEY `idx_audit_team_membership_user` (`user_id_keep`, `id`),
KEY `idx_audit_team_membership_group_fk` (`group_id`),
KEY `idx_audit_team_membership_user_fk` (`user_id`),
KEY `idx_audit_team_membership_actor_fk` (`actor_user_id`),
-- SET NULL, never CASCADE: deleting an account must not erase what was done to it.
CONSTRAINT `fk_audit_team_membership_group` FOREIGN KEY (`group_id`)
REFERENCES `group` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
CONSTRAINT `fk_audit_team_membership_user` FOREIGN KEY (`user_id`)
REFERENCES `user` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
CONSTRAINT `fk_audit_team_membership_actor` FOREIGN KEY (`actor_user_id`)
REFERENCES `user` (`id`) ON DELETE SET NULL ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
CALL _puter_add_col('share', 'holder_group_id', '`holder_group_id` int unsigned DEFAULT NULL');
DROP PROCEDURE IF EXISTS _puter_add_group_share_constraints;
DELIMITER //
CREATE PROCEDURE _puter_add_group_share_constraints()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_holder_group'
) THEN
ALTER TABLE `share` ADD INDEX `idx_share_holder_group` (`holder_group_id`, `id`);
END IF;
-- Team shares leave `holder_user_id` NULL, so the existing unique index binds none.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_holder_group_entry_issuer'
) THEN
ALTER TABLE `share` ADD UNIQUE INDEX `idx_share_holder_group_entry_issuer`
(`holder_group_id`, `fsentry_id`, `issuer_user_id`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND CONSTRAINT_NAME = 'share_holder_group_fk'
) THEN
ALTER TABLE `share` ADD CONSTRAINT `share_holder_group_fk`
FOREIGN KEY (`holder_group_id`) REFERENCES `group` (`id`)
ON DELETE CASCADE ON UPDATE CASCADE;
END IF;
END//
DELIMITER ;
CALL _puter_add_group_share_constraints();
DROP PROCEDURE IF EXISTS _puter_add_group_share_constraints;
@@ -1,68 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Opaque names for shared regions of a user's key-value namespace. See
-- sqlite/0080_kv-share-handles.sql for the column rationale.
--
-- Idempotent: `CREATE TABLE IF NOT EXISTS` with the indexes declared inline,
-- as mig_29. There is no per-file applied-state tracking, so a replay has to
-- be a no-op.
CREATE TABLE IF NOT EXISTS `kv_share_handles` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`handle` varchar(64) NOT NULL,
`owner_user_id` int unsigned NOT NULL,
`grantee_user_id` int unsigned NOT NULL,
-- Matches `apps`.`uid` exactly, charset included, so an equality against
-- one never falls back to a conversion. No foreign key: a handle outlives
-- the app it was minted against, which is what keeps it revocable.
`app_uid` char(40) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL,
`key_prefix` varchar(1024) NOT NULL,
`permission` varchar(1024) NOT NULL,
`created_at` bigint NOT NULL,
`revoked_at` bigint DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_kv_share_handles_handle` (`handle`),
KEY `idx_kv_share_handles_owner` (`owner_user_id`, `id`),
CONSTRAINT `fk_kv_share_handles_owner` FOREIGN KEY (`owner_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_kv_share_handles_grantee` FOREIGN KEY (`grantee_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- A subscription made through a handle stores the grant string it was
-- authorized under rather than an access mode, and a grant string carries a
-- user uuid, an app uid and a key prefix. Guarded on the current length, as
-- mig_24: replaying a column change on a growing table rebuilds it every boot.
DROP PROCEDURE IF EXISTS _puter_widen_subscription_permission;
DELIMITER //
CREATE PROCEDURE _puter_widen_subscription_permission()
BEGIN
IF EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'event_subscriptions'
AND COLUMN_NAME = 'permission'
AND CHARACTER_MAXIMUM_LENGTH < 1024
) THEN
ALTER TABLE `event_subscriptions` MODIFY `permission` varchar(1024) NOT NULL;
END IF;
END //
DELIMITER ;
CALL _puter_widen_subscription_permission();
DROP PROCEDURE IF EXISTS _puter_widen_subscription_permission;
@@ -1,60 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- `event_subscriptions` grew three hot queries with no index behind them:
-- a handler's own rows, the expiry reaper and the suspended reaper. See
-- sqlite/0081_event-subscriptions-indexes.sql for the query shapes.
--
-- Idempotent: the guarded procedure, as mig_30/mig_34/mig_35. There is no
-- per-file applied-state tracking, so a replay has to be a no-op.
DROP PROCEDURE IF EXISTS _puter_add_event_subscriptions_indexes;
DELIMITER //
CREATE PROCEDURE _puter_add_event_subscriptions_indexes()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'event_subscriptions'
AND INDEX_NAME = 'idx_event_subscriptions_app_handler'
) THEN
ALTER TABLE `event_subscriptions`
ADD INDEX `idx_event_subscriptions_app_handler` (`app_uid`, `handler_name`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'event_subscriptions'
AND INDEX_NAME = 'idx_event_subscriptions_expires'
) THEN
ALTER TABLE `event_subscriptions`
ADD INDEX `idx_event_subscriptions_expires` (`expires_at`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'event_subscriptions'
AND INDEX_NAME = 'idx_event_subscriptions_suspended'
) THEN
ALTER TABLE `event_subscriptions`
ADD INDEX `idx_event_subscriptions_suspended` (`suspended_at`, `id`);
END IF;
END//
DELIMITER ;
CALL _puter_add_event_subscriptions_indexes();
DROP PROCEDURE IF EXISTS _puter_add_event_subscriptions_indexes;
@@ -1,21 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0082_temp-password-expiry.sql for the column rationale.
-- No per-file applied-state tracking, so the column goes through _puter_add_col.
CALL _puter_add_col('user', 'temp_password_expires_at', '`temp_password_expires_at` bigint DEFAULT NULL');
@@ -1,21 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0083_team-directory.sql for the column rationale.
-- No per-file applied-state tracking, so the column goes through _puter_add_col.
CALL _puter_add_col('group', 'directory_enabled', '`directory_enabled` tinyint(1) NOT NULL DEFAULT 0');
@@ -1,42 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0084_share-anyone-with-link.sql for the column rationale.
-- No per-file applied-state tracking, so the column goes through _puter_add_col
-- and the index through a guarded procedure, as mysql_mig_34 does.
CALL _puter_add_col('share', 'anyone', '`anyone` tinyint(1) DEFAULT NULL');
DROP PROCEDURE IF EXISTS _puter_add_share_anyone_index;
DELIMITER //
CREATE PROCEDURE _puter_add_share_anyone_index()
BEGIN
-- Every other row leaves `anyone` NULL, so this binds only link shares.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_anyone'
) THEN
ALTER TABLE `share` ADD UNIQUE INDEX `idx_share_anyone` (`fsentry_id`, `anyone`);
END IF;
END//
DELIMITER ;
CALL _puter_add_share_anyone_index();
DROP PROCEDURE IF EXISTS _puter_add_share_anyone_index;
@@ -1,22 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0085_event-subscriptions-include-value.sql for the column
-- rationale. No per-file applied-state tracking, so it goes through
-- _puter_add_col as mysql_mig_39 does.
CALL _puter_add_col('event_subscriptions', 'include_value', '`include_value` tinyint(1) NOT NULL DEFAULT 0');
@@ -1,75 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0086: `subdomains.app_owner` cascades on app
-- delete instead of nulling. The existing constraint is found by column
-- rather than by name, since an older database need not carry the name
-- mysql_mig_1 declares. Guarded procedure as mysql_mig_34: a replay sees
-- DELETE_RULE already CASCADE and does nothing.
DROP PROCEDURE IF EXISTS _puter_subdomains_app_owner_cascade;
DELIMITER //
CREATE PROCEDURE _puter_subdomains_app_owner_cascade()
BEGIN
DECLARE fk_name VARCHAR(64) DEFAULT NULL;
-- A subquery, not SELECT ... INTO: no matching row yields NULL quietly.
SET fk_name = (
SELECT rc.CONSTRAINT_NAME
FROM INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS rc
JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE kcu
ON kcu.CONSTRAINT_SCHEMA = rc.CONSTRAINT_SCHEMA
AND kcu.CONSTRAINT_NAME = rc.CONSTRAINT_NAME
AND kcu.TABLE_NAME = rc.TABLE_NAME
WHERE rc.CONSTRAINT_SCHEMA = DATABASE()
AND rc.TABLE_NAME = 'subdomains'
AND rc.REFERENCED_TABLE_NAME = 'apps'
AND kcu.COLUMN_NAME = 'app_owner'
AND rc.DELETE_RULE <> 'CASCADE'
LIMIT 1
);
IF fk_name IS NOT NULL THEN
SET @s := CONCAT('ALTER TABLE `subdomains` DROP FOREIGN KEY `', fk_name, '`');
PREPARE stmt FROM @s;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END IF;
IF NOT EXISTS (
SELECT 1
FROM INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS rc
JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE kcu
ON kcu.CONSTRAINT_SCHEMA = rc.CONSTRAINT_SCHEMA
AND kcu.CONSTRAINT_NAME = rc.CONSTRAINT_NAME
AND kcu.TABLE_NAME = rc.TABLE_NAME
WHERE rc.CONSTRAINT_SCHEMA = DATABASE()
AND rc.TABLE_NAME = 'subdomains'
AND rc.REFERENCED_TABLE_NAME = 'apps'
AND kcu.COLUMN_NAME = 'app_owner'
AND rc.DELETE_RULE = 'CASCADE'
) THEN
ALTER TABLE `subdomains` ADD CONSTRAINT `fk_subdomains_app_owner`
FOREIGN KEY (`app_owner`) REFERENCES `apps` (`id`)
ON DELETE CASCADE ON UPDATE CASCADE;
END IF;
END//
DELIMITER ;
CALL _puter_subdomains_app_owner_cascade();
DROP PROCEDURE IF EXISTS _puter_subdomains_app_owner_cascade;
@@ -1,39 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0087: Contact Us attachment metadata, a JSON array
-- of `{name, type, size}` (NULL when none). Guarded by a throwaway procedure,
-- as in mysql_mig_21.
DROP PROCEDURE IF EXISTS _puter_add_feedback_attachments;
DELIMITER //
CREATE PROCEDURE _puter_add_feedback_attachments()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'feedback'
AND COLUMN_NAME = 'attachments'
) THEN
ALTER TABLE `feedback` ADD COLUMN `attachments` text DEFAULT NULL;
END IF;
END//
DELIMITER ;
CALL _puter_add_feedback_attachments();
DROP PROCEDURE IF EXISTS _puter_add_feedback_attachments;
@@ -1,44 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0088: index `apps.index_url` for origin lookups.
-- TEXT needs a prefix; 191 characters keeps the key under 767 bytes in
-- utf8mb4 whatever the table's row format, and URLs diverge well before it.
--
-- Idempotent: the guarded procedure, as mig_36. The 1061 handler covers two
-- nodes booting at once, where the loser finds the index already built.
DROP PROCEDURE IF EXISTS _puter_add_apps_index_url_index;
DELIMITER //
CREATE PROCEDURE _puter_add_apps_index_url_index()
BEGIN
DECLARE CONTINUE HANDLER FOR 1061 BEGIN END;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'apps'
AND INDEX_NAME = 'idx_apps_index_url'
) THEN
ALTER TABLE `apps`
ADD INDEX `idx_apps_index_url` (`index_url`(191)),
ALGORITHM=INPLACE, LOCK=NONE;
END IF;
END//
DELIMITER ;
CALL _puter_add_apps_index_url_index();
DROP PROCEDURE IF EXISTS _puter_add_apps_index_url_index;
@@ -1,21 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0089: a team may require 2FA of the accounts it
-- provisioned. Added through `_puter_add_col` as mysql_mig_32 does, so a
-- replay finds the column and does nothing.
CALL _puter_add_col('group', 'require_2fa', '`require_2fa` tinyint(1) NOT NULL DEFAULT 0');
@@ -1,39 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0090. Guarded like mysql_mig_22, because there is
-- no per-file applied-state tracking and a replay must do nothing.
DROP PROCEDURE IF EXISTS _puter_add_user_app_permission_index;
DELIMITER //
CREATE PROCEDURE _puter_add_user_app_permission_index()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user_to_app_permissions'
AND INDEX_NAME = 'idx_user_to_app_permissions_permission'
) THEN
ALTER TABLE `user_to_app_permissions`
ADD INDEX `idx_user_to_app_permissions_permission` (`permission`);
END IF;
END//
DELIMITER ;
CALL _puter_add_user_app_permission_index();
DROP PROCEDURE IF EXISTS _puter_add_user_app_permission_index;
@@ -72,8 +72,7 @@ CREATE TABLE IF NOT EXISTS "user" (
signup_origin varchar(255),
signup_server varchar(255),
metadata jsonb DEFAULT '{}'::jsonb,
reputation smallint DEFAULT 100,
home varchar(64)
reputation smallint DEFAULT 100
);
CREATE INDEX IF NOT EXISTS idx_user_email ON "user" (email);
@@ -1,40 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Enforce "at most one account owns an email address". Mirrors SQLite
-- migration 0066 and MySQL migration 21; see those for the full rationale.
--
-- In short: `user.email` is deliberately not UNIQUE because several rows may
-- hold the same address while unconfirmed. What must not happen is two rows
-- both owning it — confirmed, or holding a password and so able to drive
-- password recovery for that inbox. The application checks for an owner before
-- every write, but a check and a write are not one operation.
--
-- If this fails, the DB already contains duplicate owners; collapse them first
-- (admin → One-off Jobs → Collapse Duplicate Emails).
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_owned_email
ON "user" (COALESCE(clean_email, LOWER(email)))
WHERE email IS NOT NULL
AND (email_confirmed = TRUE OR password IS NOT NULL);
-- One Puter account per external identity. OIDCStore.link already assumes this
-- constraint exists — it catches the unique violation to tell "re-linking the
-- same account" apart from "this sub belongs to someone else" — but the table
-- never actually had it.
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_oidc_provider_sub
ON user_oidc_providers (provider, provider_sub);
@@ -1,42 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Grow share from a pending-email-invite table into the index of active
-- shares. See sqlite/0067_share_entries.sql for the column rationale.
ALTER TABLE share
ADD COLUMN IF NOT EXISTS holder_user_id integer
REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
ADD COLUMN IF NOT EXISTS fsentry_id integer
REFERENCES fsentries (id) ON DELETE CASCADE ON UPDATE CASCADE,
ADD COLUMN IF NOT EXISTS mode varchar(20),
ADD COLUMN IF NOT EXISTS applied_at timestamp;
CREATE INDEX IF NOT EXISTS idx_share_holder
ON share (holder_user_id, id);
CREATE INDEX IF NOT EXISTS idx_share_fsentry
ON share (fsentry_id);
CREATE UNIQUE INDEX IF NOT EXISTS idx_share_holder_entry_issuer
ON share (holder_user_id, fsentry_id, issuer_user_id);
-- Retiring a deleted node's grants looks them up by permission text, and the
-- primary key is (issuer, holder, permission) — `permission` is never the
-- leading column, so the delete path would otherwise scan the table. The
-- text_pattern_ops class is what makes the left-anchored LIKE a range scan
-- under a non-C collation; plain equality still uses it.
CREATE INDEX IF NOT EXISTS idx_user_to_user_permissions_permission
ON user_to_user_permissions (permission text_pattern_ops);
@@ -1,33 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- One user refusing contact from another. See sqlite/0068_user-block.sql for
-- the rationale. `created_at` is unix seconds.
--
-- Idempotent via IF NOT EXISTS.
CREATE TABLE IF NOT EXISTS user_block (
id BIGSERIAL PRIMARY KEY,
blocker_user_id integer NOT NULL
REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
blocked_user_id integer NOT NULL
REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
created_at BIGINT NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_block_pair
ON user_block (blocker_user_id, blocked_user_id);
@@ -1,20 +0,0 @@
-- Drops three groups that no code has ever read: `freeai`, `experimental` and
-- `dangerous`. No migration creates them — they were added by hand back when
-- hardcoded permissions were keyed by group name. That map is now a flat
-- per-user floor (`default_user_permissions` in data/hardcoded-permissions.js),
-- so a group nothing looks up grants nothing, and the rows are dead weight.
--
-- Guarded, and deliberately so: user_to_group_permissions.group_id and
-- jct_user_group.group_id both cascade on delete, so removing a group that
-- still carries permissions or members would silently revoke them from every
-- member. Only a group with neither is dropped. A group that survives this
-- migration has dependents and needs a deliberate decision, not a sweep --
-- query user_to_group_permissions by group_id to see what it holds.
DELETE FROM "group" g
WHERE g.extra ->> 'name' IN ('freeai', 'experimental', 'dangerous')
AND NOT EXISTS (
SELECT 1 FROM user_to_group_permissions p WHERE p.group_id = g.id
)
AND NOT EXISTS (
SELECT 1 FROM jct_user_group j WHERE j.group_id = g.id
);
@@ -1,23 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- "Shared by me" pages by keyset: issuer equality, then `id` as the cursor.
-- The single-column issuer index (postgres_mig_1) serves the equality but not
-- the ordering; this composite lets the issued half of that listing range-scan
-- without a sort.
CREATE INDEX IF NOT EXISTS idx_share_issuer
ON share (issuer_user_id, id);
@@ -1,55 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Scope tuple for `notification`. See mysql/mysql_mig_26.sql for the rationale.
-- `app_uid` is varchar(40) — the width of `apps`.`uid` — and varchar rather
-- than char for the same reason `uid` is here: blank padding is not what the
-- column means.
--
-- Idempotent via IF NOT EXISTS and the `type = ''` guard on the backfill.
ALTER TABLE notification ADD COLUMN IF NOT EXISTS app_uid varchar(40);
ALTER TABLE notification ADD COLUMN IF NOT EXISTS audience varchar(16) NOT NULL DEFAULT 'account';
ALTER TABLE notification ADD COLUMN IF NOT EXISTS type varchar(64) NOT NULL DEFAULT '';
-- Serves both the scoped list and the unacknowledged count.
CREATE INDEX IF NOT EXISTS idx_notification_scope
ON notification (user_id, audience, app_uid, acknowledged);
UPDATE notification
SET type = 'share.received', audience = 'account', app_uid = NULL
WHERE type = ''
AND value->>'template' = 'file-shared-with-you';
UPDATE notification
SET type = 'share.claimed', audience = 'account', app_uid = NULL
WHERE type = ''
AND value->>'template' = 'file-shared-before-you-joined';
-- Worker rows carried only `source: 'worker'` and a title, so the app they
-- belong to is unrecoverable and they stay unattributed. The title prefix is
-- the sole surviving signal of which way the deploy went.
UPDATE notification
SET audience = 'developer',
app_uid = NULL,
type = CASE
WHEN value->>'title' LIKE 'Successfully deployed %'
THEN 'app.worker.deployed'
ELSE 'app.worker.deployFailed'
END
WHERE type = ''
AND value->>'source' = 'worker';
@@ -1,22 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Index `notification`.`created_at` for the retention sweep. See
-- mysql/mysql_mig_27.sql for the rationale.
CREATE INDEX IF NOT EXISTS idx_notification_created_at
ON notification (created_at);
@@ -1,23 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Home region column. Mirrors SQLite migration 0074. Home region this
-- account's data belongs in — a node id, not a storage region. NULL for
-- accounts that predate the column, which fall back to `signup_server`.
-- Idempotent via IF NOT EXISTS.
ALTER TABLE "user" ADD COLUMN IF NOT EXISTS home varchar(64);
@@ -1,53 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Subscriptions that outlive the connection that made them. See
-- sqlite/0075_event-subscriptions.sql for the column rationale.
--
-- Idempotent via IF NOT EXISTS.
CREATE TABLE IF NOT EXISTS event_subscriptions (
id BIGSERIAL PRIMARY KEY,
sub_id VARCHAR(80) NOT NULL UNIQUE,
token VARCHAR(255) NOT NULL,
owner_user_id INTEGER NOT NULL
REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
holder_user_id INTEGER NOT NULL
REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
app_uid VARCHAR(40) DEFAULT NULL,
subject VARCHAR(4096) NOT NULL,
anchor_uid VARCHAR(36) NOT NULL,
anchor_path VARCHAR(4096) NOT NULL,
"match" VARCHAR(1024) DEFAULT NULL,
delivery VARCHAR(16) NOT NULL,
ops VARCHAR(64) DEFAULT NULL,
handler_name VARCHAR(128) DEFAULT NULL,
targets JSONB NOT NULL,
context TEXT DEFAULT NULL,
permission VARCHAR(32) NOT NULL,
suspended_at BIGINT DEFAULT NULL,
suspended_reason VARCHAR(64) DEFAULT NULL,
expires_at BIGINT DEFAULT NULL,
created_at BIGINT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_event_subscriptions_token
ON event_subscriptions (token);
CREATE INDEX IF NOT EXISTS idx_event_subscriptions_holder
ON event_subscriptions (holder_user_id, app_uid);
CREATE INDEX IF NOT EXISTS idx_event_subscriptions_owner
ON event_subscriptions (owner_user_id);
@@ -1,35 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- `event_subscriptions.anchor_uid` was sized for a filesystem node uuid. A
-- key-value subscription anchors on an app instead, and an app uid is 40
-- characters. See mysql/mysql_mig_30.sql.
--
-- Guarded: an unconditional ALTER COLUMN TYPE takes an ACCESS EXCLUSIVE lock
-- on every replay, even widening to the same type. Skipped once the column is
-- already wide enough.
DO $$
BEGIN
IF (
SELECT character_maximum_length FROM information_schema.columns
WHERE table_name = 'event_subscriptions' AND column_name = 'anchor_uid'
) < 40 THEN
ALTER TABLE event_subscriptions
ALTER COLUMN anchor_uid TYPE VARCHAR(40);
END IF;
END $$;
@@ -1,34 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Named handlers an app deploys once. See sqlite/0076_event-handlers.sql for
-- the column rationale.
--
-- Idempotent via IF NOT EXISTS.
CREATE TABLE IF NOT EXISTS event_handlers (
id BIGSERIAL PRIMARY KEY,
app_uid VARCHAR(40) NOT NULL,
name VARCHAR(128) NOT NULL,
source TEXT NOT NULL,
source_hash VARCHAR(64) NOT NULL,
created_at BIGINT NOT NULL,
updated_at BIGINT NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_event_handlers_app_name
ON event_handlers (app_uid, name);
@@ -1,38 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Teams. See sqlite/0077_teams.sql for the column rationale.
-- Idempotent via IF NOT EXISTS; there is no per-file applied-state tracking.
ALTER TABLE "group" ADD COLUMN IF NOT EXISTS kind text;
ALTER TABLE "group" ADD COLUMN IF NOT EXISTS name text;
ALTER TABLE "group" ADD COLUMN IF NOT EXISTS handle text;
ALTER TABLE "group" ADD COLUMN IF NOT EXISTS deleted_at timestamp;
-- On lower(handle): text is case-sensitive here, unlike MySQL's utf8mb4_unicode_ci.
-- Handle lookups must therefore compare lower(handle) to hit this index.
CREATE UNIQUE INDEX IF NOT EXISTS idx_group_handle ON "group" (lower(handle));
-- No idx_group_owner here: idx_group_owner_user_id already covers it.
ALTER TABLE jct_user_group ADD COLUMN IF NOT EXISTS org_owned smallint;
-- Composite, unlike the existing single-column idx_jct_user_group_group_id.
CREATE INDEX IF NOT EXISTS idx_jct_user_group_group
ON jct_user_group (group_id, user_id);
ALTER TABLE "user" ADD COLUMN IF NOT EXISTS requires_password_change smallint NOT NULL DEFAULT 0;
@@ -1,34 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Deduplicate then constrain. See sqlite/0075 for why this is lossless.
-- Guarded because postgres re-runs every file on each boot; `to_regclass` follows
-- search_path, so it resolves under a test schema too.
DO $mig15$
BEGIN
IF to_regclass('idx_jct_user_group_pair') IS NULL THEN
DELETE FROM jct_user_group
WHERE id NOT IN (
SELECT MIN(id) FROM jct_user_group GROUP BY user_id, group_id
);
END IF;
END
$mig15$;
CREATE UNIQUE INDEX IF NOT EXISTS idx_jct_user_group_pair
ON jct_user_group (user_id, group_id);
@@ -1,57 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0076. Types follow `audit_user_to_group_permissions` in postgres_mig_1.
CREATE TABLE IF NOT EXISTS audit_team_membership (
id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
-- SET NULL, never CASCADE: deleting an account must not erase what was done to it.
group_id integer REFERENCES "group" (id) ON DELETE SET NULL ON UPDATE CASCADE,
group_id_keep integer NOT NULL,
user_id integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
user_id_keep integer NOT NULL,
actor_user_id integer REFERENCES "user" (id) ON DELETE SET NULL ON UPDATE CASCADE,
action varchar(255) NOT NULL,
reason varchar(255),
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_audit_team_membership_group
ON audit_team_membership (group_id_keep, id);
CREATE INDEX IF NOT EXISTS idx_audit_team_membership_user
ON audit_team_membership (user_id_keep, id);
-- SET NULL has to find the child rows; postgres does not index FK columns for you.
CREATE INDEX IF NOT EXISTS idx_audit_team_membership_group_fk
ON audit_team_membership (group_id);
CREATE INDEX IF NOT EXISTS idx_audit_team_membership_user_fk
ON audit_team_membership (user_id);
CREATE INDEX IF NOT EXISTS idx_audit_team_membership_actor_fk
ON audit_team_membership (actor_user_id);
ALTER TABLE share ADD COLUMN IF NOT EXISTS holder_group_id integer
REFERENCES "group" (id) ON DELETE CASCADE ON UPDATE CASCADE;
CREATE INDEX IF NOT EXISTS idx_share_holder_group
ON share (holder_group_id, id);
-- Default NULLS DISTINCT, matching the other engines, so this binds only team shares.
CREATE UNIQUE INDEX IF NOT EXISTS idx_share_holder_group_entry_issuer
ON share (holder_group_id, fsentry_id, issuer_user_id);
@@ -1,64 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Opaque names for shared regions of a user's key-value namespace. See
-- sqlite/0080_kv-share-handles.sql for the column rationale.
--
-- Idempotent via IF NOT EXISTS.
CREATE TABLE IF NOT EXISTS kv_share_handles (
id BIGSERIAL PRIMARY KEY,
handle VARCHAR(64) NOT NULL,
owner_user_id INTEGER NOT NULL
REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
grantee_user_id INTEGER NOT NULL
REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
app_uid VARCHAR(40) NOT NULL,
key_prefix VARCHAR(1024) NOT NULL,
permission VARCHAR(1024) NOT NULL,
created_at BIGINT NOT NULL,
revoked_at BIGINT DEFAULT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_kv_share_handles_handle
ON kv_share_handles (handle);
CREATE INDEX IF NOT EXISTS idx_kv_share_handles_owner
ON kv_share_handles (owner_user_id, id);
-- A cascading delete has to find the child rows, and postgres does not index
-- FK columns for you.
CREATE INDEX IF NOT EXISTS idx_kv_share_handles_grantee_fk
ON kv_share_handles (grantee_user_id);
-- A subscription made through a handle stores the grant string it was
-- authorized under rather than an access mode, and a grant string carries a
-- user uuid, an app uid and a key prefix.
--
-- Guarded: an unconditional ALTER COLUMN TYPE takes an ACCESS EXCLUSIVE lock
-- on every replay, even widening to the same type. Skipped once the column is
-- already wide enough.
DO $$
BEGIN
IF (
SELECT character_maximum_length FROM information_schema.columns
WHERE table_name = 'event_subscriptions' AND column_name = 'permission'
) < 1024 THEN
ALTER TABLE event_subscriptions
ALTER COLUMN permission TYPE VARCHAR(1024);
END IF;
END $$;
@@ -1,31 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- `event_subscriptions` grew three hot queries with no index behind them:
-- a handler's own rows, the expiry reaper and the suspended reaper. See
-- sqlite/0081_event-subscriptions-indexes.sql for the query shapes.
--
-- Idempotent via IF NOT EXISTS.
CREATE INDEX IF NOT EXISTS idx_event_subscriptions_app_handler
ON event_subscriptions (app_uid, handler_name);
CREATE INDEX IF NOT EXISTS idx_event_subscriptions_expires
ON event_subscriptions (expires_at);
CREATE INDEX IF NOT EXISTS idx_event_subscriptions_suspended
ON event_subscriptions (suspended_at, id);
@@ -1,21 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0082_temp-password-expiry.sql for the column rationale.
-- Idempotent via IF NOT EXISTS; there is no per-file applied-state tracking.
ALTER TABLE "user" ADD COLUMN IF NOT EXISTS temp_password_expires_at bigint DEFAULT NULL;
@@ -1,21 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0083_team-directory.sql for the column rationale.
-- Idempotent via IF NOT EXISTS; there is no per-file applied-state tracking.
ALTER TABLE "group" ADD COLUMN IF NOT EXISTS directory_enabled smallint NOT NULL DEFAULT 0;
@@ -1,24 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0084_share-anyone-with-link.sql for the column rationale.
-- Idempotent via IF NOT EXISTS; there is no per-file applied-state tracking.
ALTER TABLE share ADD COLUMN IF NOT EXISTS anyone smallint;
-- Default NULLS DISTINCT, matching the other engines, so this binds only link shares.
CREATE UNIQUE INDEX IF NOT EXISTS idx_share_anyone ON share (fsentry_id, anyone);
@@ -1,22 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- See sqlite/0085_event-subscriptions-include-value.sql for the column
-- rationale. Idempotent via IF NOT EXISTS; there is no per-file applied-state
-- tracking.
ALTER TABLE event_subscriptions ADD COLUMN IF NOT EXISTS include_value smallint NOT NULL DEFAULT 0;
@@ -1,59 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0086: `subdomains.app_owner` cascades on app
-- delete instead of nulling. The existing constraint is found by column, not
-- by name. Idempotent: there is no per-file applied-state tracking, and a
-- replay sees the cascade already in place and does nothing.
DO $$
DECLARE
fk_name text;
BEGIN
SELECT c.conname INTO fk_name
FROM pg_constraint c
JOIN pg_attribute a
ON a.attrelid = c.conrelid
AND a.attnum = ANY (c.conkey)
WHERE c.conrelid = 'subdomains'::regclass
AND c.contype = 'f'
AND c.confrelid = 'apps'::regclass
AND a.attname = 'app_owner'
AND c.confdeltype <> 'c'
LIMIT 1;
IF fk_name IS NOT NULL THEN
EXECUTE format('ALTER TABLE subdomains DROP CONSTRAINT %I', fk_name);
END IF;
IF NOT EXISTS (
SELECT 1
FROM pg_constraint c
JOIN pg_attribute a
ON a.attrelid = c.conrelid
AND a.attnum = ANY (c.conkey)
WHERE c.conrelid = 'subdomains'::regclass
AND c.contype = 'f'
AND c.confrelid = 'apps'::regclass
AND a.attname = 'app_owner'
AND c.confdeltype = 'c'
) THEN
ALTER TABLE subdomains ADD CONSTRAINT subdomains_app_owner_fkey
FOREIGN KEY (app_owner) REFERENCES apps (id)
ON DELETE CASCADE ON UPDATE CASCADE;
END IF;
END $$;
@@ -1,21 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0087: Contact Us attachment metadata, a JSON array
-- of `{name, type, size}` (NULL when none).
ALTER TABLE feedback ADD COLUMN IF NOT EXISTS attachments text DEFAULT NULL;
@@ -1,24 +0,0 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0088: index `apps.index_url` for origin lookups.
-- Hash rather than btree: `index_url` allows 3000 characters, past btree's
-- ~2.7 KB row limit, and every lookup is equality (`=` / `IN`).
--
-- Idempotent via IF NOT EXISTS.
CREATE INDEX IF NOT EXISTS idx_apps_index_url ON apps USING hash (index_url);
Loaded 100 of 1399 files, more files were not shown because too many files have changed in this diff. Show more