Compare commits

...
Author SHA1 Message Date
Nariman Jelveh 734715f461 Merge branch 'feat/app-user-feedback' of https://github.com/HeyPuter/puter into feat/app-user-feedback 2026-08-12 13:32:34 -07:00
jelveh b290c1d922 Merge remote-tracking branch 'origin/main' into feat/app-user-feedback 2026-08-12 09:22:00 -07:00
jelveh dd068e31d0 Require verified users for app feedback API
Add `requireVerified: true` to both app feedback routes (`GET /target` and `POST /`) in `AppFeedbackController`. This tightens access control so only verified user accounts can fetch feedback targets or submit app feedback.
2026-08-12 09:13:11 -07:00
jelveh ea63c3093e feat(dashboard): add Publish as website and Properties to folder menu
The Files tab's folder-background context menu (empty listing area,
sidebar folders, breadcrumb segments) now offers Publish as website and
Properties for the folder the menu was opened on, matching the desktop's
folder menus. Both are omitted at the filesystem root and in Trash, and
Publish is omitted inside trashed folders.

The properties modal can now resolve an entry by path when the caller
has no uid, which the folder-background menu doesn't.

The publish gate (account creation + email confirmation) was duplicated
in three places, so it moves to a publish_as_website helper that all
call sites now share.
2026-08-12 08:50:03 -07:00
Daniel Salazar e8236a100c chore: bump versions (#3548) 2026-08-12 01:18:17 -07:00
Daniel Salazar 20b3b88e39 metering: big fixes to metering + jsdoc types (#3547)
Changes are:
- global egress metering
- remove file egress cost
- introduce file op cost for the per request cost s3 has
- enforce fs read/download etc to through 402 when out of usage; allow for subdomains
- enforce kv metering when out of usage through 402; allow for workers
- jsdoc as source of truth for puter.js types
- kv driver caching for get and batchget operations with decreased costs
2026-08-12 01:06:04 -07:00
jelveh 90cffbfcde refactor: prep the feedback cap error once instead of via a factory
Both throw sites are in one call and only one can ever run, so a plain
const reads the same and drops a function that existed only to defer a
constructor.
2026-08-11 22:11:24 -07:00
jelveh 25acf14c8b test: cover the feedback service and store layers directly
The service owns every feedback business rule — target resolution,
eligibility, message normalization, the durable caps, and the owner-email
preconditions — but was only reachable through the controller's tests. Give
it and the store their own suites so a regression names the layer it broke.

Service coverage adds the branches the route tests could not reach: a
blocked origin resolving to null rather than surfacing a 403, owners who are
suspended or unsubscribed, length measured after normalization, the 24h cap
window boundary, subject-header injection via the app title, and the email
links being rooted at config.origin.

The two describes already labelled `AppFeedbackService ...` move out of the
controller test, which keeps only the caller-facing promise that a failed
send still returns success.
2026-08-11 22:10:04 -07:00
Chen, Ting-An cbae39a08f fix(i18n): add Traditional Chinese app management strings (#3541)
Translate the app discovery, app request, desktop shortcut, and app folder controls that currently fall back to English in the Traditional Chinese interface. Preserve the positional placeholders used by accessibility labels.
2026-08-11 21:18:37 -07:00
Nariman Jelveh be781c5f1b fix: restore the feedback Send button's gradient on hover
1c2d33f14 added :not(:disabled) to the base .app-feedback-btn hover
rule, which raised its specificity above the primary button's hover rule
— so hovering Send swapped the blue gradient for the light hover
background, leaving white text on a pale pill. Give the primary hover
rule the same :not(:disabled) so it wins by source order again.
2026-08-11 19:57:56 -07:00
Nariman Jelveh 4cb0730875 feat: point the feedback email footer at the Dev Center
The footer told developers to turn feedback off via a puter.apps.update
one-liner, but the toggle lives in the Dev Center app settings — and
Dev Center is where apps get feedback enabled by default in the first
place. Reword it to 'manage it in the Dev Center' with a link built
from config.origin (like app_link) so it holds on self-hosted
deployments.
2026-08-11 19:19:38 -07:00
Nariman Jelveh 00b2fcdea3 docs: document the COOP false-resolve limitation of showFeedbackDialog
Under COOP the popup's opener link is severed, so the SDK deliberately
resolves false while the popup stays open and the user can still submit
(a feedback submission has no server read-back the way a permission
grant does). The documented contract ('resolves to true if the user
submitted feedback') was silently wrong on cross-origin-isolated pages —
state the limitation in the doc and the SDK jsdoc: false means 'not
confirmed', not 'not sent'.
2026-08-11 18:31:39 -07:00
Nariman Jelveh f70347b579 docs: state exactly what showFeedbackDialog shares with the developer
The doc claimed the dialog 'tells the user their username will be
shared', while the dialog's note talked only about the email address and
the implementation shares the username always plus the email (as
Reply-To) only when verified. Describe the actual disclosure: username
always, email when verified.
2026-08-11 18:31:02 -07:00
Nariman Jelveh 3d9691b11a fix: stop HTML-escaping email subject lines
Subjects were compiled with default Handlebars escaping, so the
app-user-feedback subject rendered a title like "Bob's App & Games" as
"Bob's App & Games" — literal entities in the recipient's mail
client. Subjects are plain-text headers, not HTML; compile them with
noEscape. Header safety is unaffected: the transport encodes newlines
and free-form values collapse whitespace upstream.
2026-08-11 18:30:29 -07:00
Nariman Jelveh c2fe813929 fix: make the feedback dialog's privacy note match what is shared
The note unconditionally said 'Your email address will be shared with
the developer so they can respond', but AppFeedbackService shares the
username always and the email only when it exists and is verified — an
unverified or temp-user sender was promised a reply path that never
materializes, and nobody was told about the username.

Show 'username and email' when the signed-in user's email is verified,
and 'username' otherwise.
2026-08-11 18:29:22 -07:00
Nariman Jelveh ef7fc05dcc fix: reject feedback origins longer than the source_origin column
readTargetParam accepted values up to 3000 chars but the raw origin is
stored verbatim into source_origin VARCHAR(2048) (MySQL/Postgres), so a
2049-3000 char origin passed every validation and then blew up the
INSERT with an HTTP 500 on Postgres/strict MySQL — or was silently
truncated on non-strict MySQL, corrupting the abuse-forensics value the
column exists for. Cap the param at the column size.
2026-08-11 18:27:42 -07:00
Nariman Jelveh 1c2d33f149 fix: gate feedback dialog Cancel/X on an in-flight submit
Escape and backdrop clicks were already ignored while the POST was
pending, but the X, Close, and Cancel buttons weren't — clicking one
mid-send settled the promise false and tore down the overlay while the
submission still landed server-side: the developer got the email, the
app was told sent=false, and a user who resubmitted 'the failed one'
sent a duplicate and burned a daily-cap slot.

Apply the same !sending gate to the buttons and disable them visually
while the send is in flight.
2026-08-11 18:26:47 -07:00
Nariman Jelveh b1fdc8d665 fix: carry feedback_enabled in suggested/recommended app summaries
launch_app uses options.app_obj verbatim when provided, and the
suggested-apps launch paths (open_item.js, UIWindowSearch.js,
UIDesktop.js) pass summaries from toAppSummary — which omitted
feedback_enabled. So an opted-in editor launched by opening a .txt file
showed no Send Feedback control in the dashboard drawer, while the same
app launched from the Apps tab (full puter.apps.get object) did.
2026-08-11 18:23:43 -07:00
Nariman Jelveh 1ecc23cf83 fix: guard app-triggered feedback dialog against desktop-lockout loops
The showFeedbackDialog IPC handler had no re-entry or abuse guard, and
the dialog it opens is a full-viewport overlay above the taskbar and
every window — so 'while (true) await puter.ui.showFeedbackDialog()'
kept the desktop permanently covered (for signed-out users, the same
loop spams the full-page signup window instead). Every dismissal just
settled the promise and let the app immediately reopen it.

Allow one dialog at a time, and back off reopens per app after each
dismissal that sent nothing: 10s, then 60s, then blocked until page
reload. A successful send resets the backoff, and user-initiated paths
(dashboard drawer) are unaffected since they don't go through IPC.
2026-08-11 18:21:43 -07:00
Nariman Jelveh 7230580b18 fix: settle showFeedbackDialog instead of hanging on older host GUIs
In the app environment showFeedbackDialog awaited an IPC reply with no
capability check. A host GUI that predates this feature (self-hosted
Puter running the live js.puter.com SDK) has no handler for the message
and never replies, so the promise documented as 'never rejects' also
never resolved.

The GUI now advertises the IPC dialogs it can answer via a
puter.gui_features param on the app iframe URL, and the SDK resolves
false when 'feedback-dialog' isn't listed. A reply timeout could not
substitute: legitimate replies only arrive when the user closes the
dialog, so any timeout would false-negative while the user is typing.
Older SDKs ignore the extra param.
2026-08-11 18:19:46 -07:00
Nariman Jelveh 1460751b41 fix: refuse feedback when the deployment cannot deliver it
With no email transport configured (the common self-hosted default),
submissions were stored in app_feedback — a table with no read path
beyond the abuse-cap COUNTs — the owner email was silently skipped, and
the sender was still shown 'Feedback sent. Thank you!'. The developer
never learns the feedback exists while the user believes it was
delivered.

Gate acceptsFeedback on clients.email.isConfigured so the pre-flight
reports enabled:false (the dialog shows its 'not accepting feedback'
pane) and submit returns 403 instead of swallowing messages. Owner-level
store-without-email cases (unconfirmed owner email, per-app email cap
overflow) keep their existing deliberate semantics.
2026-08-11 18:16:14 -07:00
Nariman Jelveh 4f96d371f4 fix: don't mint a user-app token as a side effect of the feedback popup
Every embedded_in_popup boot ran the user-app token exchange, and the
exchange is a write: /auth/get-user-app-token bootstraps an app row for
the opener origin, grants flag:app-is-authenticated (what makes the
site count as connected to the account), and creates its AppData dir.
So merely opening — or immediately cancelling — a send-feedback popup
recorded a user<->site relationship the read-only feedback flow never
needs: the server resolves the feedback target from the attested origin
without any of it.

Gate the exchange behind runsUserAppTokenExchange(action) in all three
popup paths that mint (main postAuthActions exchange, temp-user signup
success, manual signup fallback). request-permission keeps the exchange
since grants are written against the app row it bootstraps.
2026-08-11 18:13:13 -07:00
Nariman Jelveh c5cf1dc390 docs: disclose the Dev Center's feedback-on-by-default in SDK docs/types
The Dev Center deliberately creates apps with feedbackEnabled (see
32c950de5), but the SDK docs, apps.d.ts, and AppFeedbackService's class
doc all described feedback as strictly opt-in / default-false with no
qualification — so a Dev Center developer reading them would wrongly
conclude feedback is off for their app. State the Dev Center behavior
alongside the API default, and correct the update-path docs: an omitted
feedbackEnabled leaves the current value unchanged rather than
defaulting to false.
2026-08-11 18:07:47 -07:00
Nariman Jelveh 503a450b40 fix: make feedback daily caps fail closed under concurrent submissions
The per-user and per-user-per-app caps were check-then-insert and the
per-app email cap was count-then-send, so parallel requests (or multiple
nodes, or the route limiter failing open) could all read a stale
under-cap count and push past every limit — the exact scenario the
DB-backed caps exist to stop.

Now the user caps recount after the insert (own row included) and roll
the row back with 429 if a burst breached them, and the email cap claims
its slot (email_sent=1) before sending, recounts, and releases the slot
if over cap or if the send fails.
2026-08-11 18:06:05 -07:00
Nariman Jelveh f4dfbd735b fix: resolve 'app-'-prefixed app names in feedback target lookup
APP_NAME_REGEX allows names beginning with "app-" (e.g. the seeded
app-center), but resolveTargetApp's startsWith('app-') heuristic sent
those to a uid-only lookup with no name fallback, so feedback for such
apps 403'd even when enabled. Use AppStore.resolveApp (uid, then name)
like the rest of the codebase.
2026-08-11 18:03:15 -07:00
Nariman Jelveh 81402918d2 fix: feedback modal polish + share sender email
Address four issues with the app feedback UI:

- Dashboard app-drawer: the extra "feedback" control pushed the close
  button past the drawer's derived width and clipped it. A `has-feedback`
  modifier widens the surface by one button + gap so all three controls
  fit. The control's glyph is now a message bubble with text lines, which
  reads more clearly at 14px than the previous bare speech bubble.

- The feedback dialog is no longer a UIWindow. It's a from-scratch
  overlay modal in the spirit of the dashboard modals (uninstall,
  add-app): a fixed scrim + centered card with self-contained,
  theme-aware color tokens (light default + dark override), a bottom-sheet
  layout on narrow screens, backdrop/Escape close, and an entrance
  transition. This renders consistently across the three contexts it's
  opened from (desktop app-IPC, dashboard drawer, standalone popup), so
  the callers no longer pass UIWindow-specific window_options.

- Feedback now shares the sender's email (not just their username) with
  the developer so they can respond: the owner email sets Reply-To to the
  sender and shows the address in the body — but only when the sender's
  email is verified (an unverified address could be anyone's, so it's
  never used as a reply target). EmailClient.send gains an optional
  replyTo. The dialog note now says the email will be shared.

Tests: e2e updated for the new modal (7 pass); backend feedback suite
covers the verified/unverified sender-email split (sqlite + postgres);
EmailClient + GUI unit suites pass; type-check clean.
2026-08-11 17:11:36 -07:00
Nariman Jelveh 32c950de5b feat: enable app feedback by default in Dev Center
New apps created in Dev Center now have feedbackEnabled set on creation,
so users can send the developer feedback without any extra setup. A "User
Feedback" toggle in the app's settings lets developers turn it off (and
back on); it's wired into the save payload, the dirty-state tracking, and
the reset-to-original path like the neighboring toggles.

The Save update omits feedbackEnabled unless the toggle is present, and
the backend leaves an omitted field untouched, so the default survives
the create-then-save flow Dev Center runs. Add an SDK apps-suite guard
covering that round-trip (create-on -> unrelated update keeps it -> can
be turned off).
2026-08-11 16:47:12 -07:00
Nariman Jelveh 792361507e feat: add feedback control to the dashboard app-drawer
Surface the feedback dialog directly from the app window's chrome in
dashboard mode: apps that opt in (apps.feedbackEnabled) get a "Send
Feedback" button in the dashboard app-drawer, next to minimize/close.
It opens the same UIWindowAppFeedback dialog, targeting this app by uid.

The control is only rendered when the app opted in — feedback_enabled is
threaded from the launched app's metadata into the window options — and
the dialog still re-checks opt-in server-side, so a stale flag can't send
anywhere. Reuses the existing .dashboard-app-drawer-btn styling and the
app_feedback_title i18n string, so no new CSS or strings.

Adds e2e coverage: the control appears and opens the dialog for an
opted-in app, and is absent for an app that hasn't opted in.
2026-08-11 16:39:05 -07:00
Nariman Jelveh 4f267fb83e feat: app user feedback system
Add puter.ui.showFeedbackDialog(), letting users send feedback to an
app's developer. In the app environment the Puter desktop renders the
dialog; on a third-party website a puter.com popup hosts it. The message
is stored in a new app_feedback table and emailed to the app owner's
confirmed email — it never passes through the app's own code.

Feedback is strictly opt-in per app via a new apps.feedback_enabled
column (a real column, not an app-metadata key, so Dev Center's
whole-blob metadata saves can't silently erase it), settable through the
existing puter.apps.update path (feedbackEnabled).

Backend follows the layered stack: AppFeedbackStore (durable count
queries) -> AppFeedbackService (opt-in check, message normalization,
abuse caps, best-effort owner email) -> AppFeedbackController
(POST /app-feedback, GET /app-feedback/target). New app-user-feedback
email template uses the escaping-safe nl2br triple-stash.

Defensive by design:
- requireUserActor blocks app tokens, so feedback can't be submitted
  programmatically; guiOriginOnly keeps cross-origin pages out.
- App identity comes only from the validated IPC sender (desktop) or the
  browser-attested opener origin (popup), never from message contents.
- The send-feedback popup action is in NON_AUTH_POPUP_ACTIONS, so it
  never delivers a token to the opener.
- Layered limits: route rate limits, plus DB-count caps that fail closed
  when the limiter backend is down, plus a per-app daily owner-email cap.
- Owner email is fully best-effort: an unconfigured transport,
  unconfirmed/unsubscribed/suspended owner, or send failure never fails
  the request or blocks storage.
- The dialog and SDK method are resolve-only and always settle, so a
  caller is never left hanging.

Migrations for sqlite/mysql/postgres, puter.js types, docs, backend
tests (sqlite + postgres), and a Playwright e2e spec are included.
2026-08-11 16:31:27 -07:00
Daniel Salazar afad3e37bf fix: server health tolerance for primary read (#3542) 2026-08-11 01:19:09 -07:00
Daniel Salazar a6b2161b2c fix: misc sec fixes (#3540) 2026-08-10 22:24:44 -07:00
Neal Shah 9e864e9574 PUT-1432 (#3539) 2026-08-10 23:33:20 -04:00
Daniel Salazar 79d4201f12 fix: rate limits, AI routing, and a type-check gate (#3529)
- declare rate + concurrency limits on every route and driver that lacked one
- add acquireConcurrent for websocket connections and the DAV mount
- bucket AI models by identity key only; keep resold duplicates of any vendor
- skip recently-failed provider routes; cap the fallback chain at 3 attempts
- let full-access access tokens bind a worker to an app their own user owns
- cache resolved subscriptions so tiered limits don't add a round trip
2026-08-10 19:09:47 -07:00
Daniel Salazar 0c2d7dfa34 feat: add user created timestamp on whoami (#3538) 2026-08-10 19:08:50 -07:00
Neal Shah 2a6d2fea32 allow max-tokenless models on openrouter (#3536) 2026-08-10 18:40:10 -04:00
Juan Fernando Castro 1dec5e90c4 docs: add requestAppData() to Perms sidebar (#3534) 2026-08-10 11:38:09 -07:00
Neal Shah 566298d89c make context length on togetherAI more lenient (#3533) 2026-08-10 13:14:33 -04:00
Daniel Salazar 6bb5eacf61 fix: metering kv spamc (#3532) 2026-08-10 08:36:56 -07:00
Reynaldi Chernando 390a83c860 Fix puter.js docs and types drift 10 aug (#3531) 2026-08-10 12:47:18 +07:00
Nariman Jelveh 8f23bf7b75 fix(gui): hover the add-an-app options with the accent wash, not grey
The rows already answer hover with a --select-ring border; filling them
with the grey --dashboard-hover under that accent ring read as two
systems disagreeing. The weak select tint keeps the hover in one voice
and stays under the icon chip's stronger tint in both themes.
2026-08-09 18:23:07 -07:00
Nariman Jelveh aa85dddd38 Refresh Add App AI description copy
Update the English `add_app_ai_desc` translation string in the GUI to a more direct, enthusiastic message: “Describe the app you want and AI will build it!”.
2026-08-09 18:21:31 -07:00
Nariman Jelveh 8284f35b93 fix(gui): honor godmode file_paths launches now that /apps returns booleans
The godmode gate in ExecService.launchApp checked `godmode === 1`, but
GET /apps/:name serializes the flag as a boolean, so the check always
failed and file_paths launches (e.g. Dev Center opening a worker's
source in the editor, #2218) silently dropped the file. Accept both
shapes, matching the existing guard in launch_app.js.

Also fix the two issues hiding behind that gate:
- puter.apps.get returns `uid`, not `uuid`, so the /sign call for the
  target app sent app_uid undefined — no write grant and no user-app
  token for the launched app. Use `uid` with a `uuid` fallback.
- The closeApp IPC handler had the same `godmode === 1` comparison,
  preventing godmode apps from closing other apps' windows.
2026-08-09 18:06:58 -07:00
Nariman Jelveh ccec3ce39b fix(gui): open add-an-app choices through the deep-link intro, not a + tile morph
60686643b made App Center and the AI builder morph out of the + tile the
user clicked, which answers where the window came from but not the
question these two options exist to answer: where the APP went. The grid
never showed it arriving, so the user still ended a launch with nothing
to find, no tile for minimize to fly back to, and no lesson learned.

Choosing either option now plays exactly what an /app/<name> landing
plays (beginDeepLinkLaunch): when the app already has a tile the intro
travels to it, plays the click flourish, and the window grows out of the
app's own slot; when it doesn't, the tile is INSTALLED first — the slot
opens, the progress stroke draws, the icon springs in — and only then
does the window morph out of the tile that just arrived. Same beats,
same exposure decay, same interruptibility, same duplicate-launch claim.

The app info is prefetched in parallel and handed to launch_app, the
same contract as initgui's landing path, so the intro (which may need to
draw the arriving tile from it) never costs a second round-trip.

This also reverts 60686643b's anchor plumbing (UIWindow's
dashboard_tile_el option and the dashboard_tile_in_view export): the
intro installs a real tile for the by-app-name morph to find, so no
stand-in anchor is needed.
2026-08-09 18:02:39 -07:00
Nariman Jelveh 60686643b3 fix(gui): launch from the add-an-app modal with the tile morph, not a fade
Choosing App Center or the AI builder opened the window with the plain
fade while every other launch in the Apps tab grows out of the icon that
was clicked — the one launch that most needs to say where it came from
read as the app appearing from nowhere.

The morph resolved its anchor by app name, which cannot work here: the
whole point of these two options is installing an app you do not have,
and an app with no tile has no name to find. So the launcher may name its
anchor element (dashboard_tile_el), and TabApps resolves it once before
launching so the click flourish and the window's half can never disagree
about which tile the launch came from: the app's own tile when it already
has one on screen — minimize flies the window back there, so that is
where opening should come from — and otherwise the + tile itself.

dashboard_tile_in_view's visibility test is now a named helper it shares
with the new path, so an anchor on an offscreen pager page is still
rejected and still falls through to the fade.
2026-08-09 17:49:50 -07:00
Nariman Jelveh 61bfcbdc83 feat(gui): add an add-an-app tile at the end of the Apps grid
The Apps grid showed what you have and offered no way to get more. A
dashed plus tile now rides at the tail of every page of it: after every
app, on the last page, and nowhere else. It wears the same tile skeleton
as an app so hover, focus, and arrow keys treat it identically, but
carries no app name or group id — which is what keeps it out of the
saved order, the running dots, and the deep-link tile lookups. It cannot
be picked up, is never a drop target (so no app can land past it), has
no context menu, and sits still while the apps jiggle in reorder mode.
Search results leave it out: a query asks which apps you have.

Clicking it asks how you would like to get an app: find one in the App
Center, have one built by the AI builder, or request one that does not
exist yet. The request is a form in the same card rather than the stock
Contact Us window — a titled pane one step deeper, whose Back arrow
returns to the options with the draft intact. It posts the same
/contactUs request that window did; a failure is said inside the form,
where the request is still there to retry, rather than in an alert
window the next click on the dashboard would bury.
2026-08-09 17:37:09 -07:00
Nariman Jelveh c8985b621d Add copy-link support for app tiles
Extract app tile URL generation into a shared helper and use it for a new dashboard context menu action that copies the tile's destination. This keeps copied links aligned with the existing open-in-new-tab behavior and adds regression coverage for app, external, encoded, and folder tiles.
2026-08-09 16:54:18 -07:00
Nariman Jelveh 42299a64ed Update recommended apps list
Add 'blockarena' and remove 'pretty-tiles', 'galaxy-troops', and 'blend-fruits' from the recommended apps list.
2026-08-09 11:20:40 -07:00
jelveh 344db5cb0b Remove vault from recommended apps
Drop `vault` from the recommended apps list so it no longer appears in app recommendations.
2026-08-08 11:54:56 -07:00
jelveh 66824f607d feat(gui): install an app onto the grid when its landing is what installs it
A /app/<name> landing for an app the dashboard didn't list opened with
a plain fade and left the app NOWHERE: the grid had been fetched before
the launch granted the permission that installedApps reports, so Back
found no tile to minimize into and the user landed on a dashboard
without the app they were just inside. The round trip that teaches
"windows are inflated tiles" broke on exactly the landing where a new
user is forming that model.

Now the landing SAYS what it does. The app joins the grid at the tail
(where new apps land) with a tile drawn from the landing's own app-info
prefetch -- only a confirmed-real app ever materializes, with its real
icon -- held invisible until the intro has revealed the grid and
travelled to the tile's page, and then it is INSTALLED, in the grammar
every app store taught: the slot opens (the folder well's own surface)
with the icon dim inside it, a progress stroke draws clockwise around
the slot, and on completion the icon springs to full color and size as
the label names it. Only then does the flourish play and the window
grow out of the tile. The arrival is the one beat that never decays:
per-app news delivered at most once per app, not a repeated lesson.
Minimize now has a target, and the app is simply there afterward.

The regression surface is guarded on every edge: the parked-invisible
state is re-applied by renderApps itself so mid-intro re-renders can't
strip it, and settleDeepLinkLaunch -- which the landing's finally
always reaches -- force-reveals it, so no path leaves an invisible
tile. A _pendingInstalls overlay (the mirror of _removedLocal) keeps a
refresh fetched before the grant lands from evicting the tile, and
retires itself once installedApps confirms the app or the user
uninstalls it again. The order is never saved on the splice, so the
default-position append can't freeze a custom order the user never
made. Failed or absent app info adds nothing; installed-app landings
are untouched; animations-off and reduced-motion users skip the
choreography but still get the tile, silently. The progress ring is a
transient element (the tile's ::before is the well, its ::after the
running dot) drawn as an SVG dashoffset sweep -- no Houdini dependency
-- and the app-info prefetch now asks for the 128px icons every other
tile already uses.
2026-08-08 11:25:48 -07:00
jelveh 66dedf3082 feat(gui): let the app drawer's intro flash retire once learned
The control drawer on headless dashboard apps opened itself on EVERY
window open, forever -- 2.6 seconds of tray over the app's own top
edge to teach a lesson (the controls live in this tongue) that is
learned in a few exposures. Same doctrine as the dashboard's
deep-link intro: pedagogy that knows when to step aside.

A per-account kv counter records deliveries, once per window: a flash
played while the user could see it, or the user opening the drawer
themselves (hover, tap, focus) -- the stronger proof, since the
drawer is a headless app's only chrome and must never decay out of a
user's awareness. After three, new windows keep the bare tongue and
the flash stops; hover, tap, and focus expansion never decay.

The attach is synchronous and the stored count is not: the first
drawer of a session races the read against a short cap and defaults
to teaching -- a hung read must not hold the intro hostage, and every
failure mode (unparseable value, failed read, timed-out read, failed
write) errs toward teaching once more, never toward never teaching.
The resolved count is cached and kept current in memory, so later
windows decide synchronously and a session's own not-yet-written
increments still count. kv.incr keeps two devices from losing an
increment, and the pre-read caps the key once the lesson is learned.

If the user beats a still-deciding flash to the drawer, the flash is
skipped rather than played over their hand -- its auto-collapse would
shut a drawer they are actively using. A flash in a hidden tab still
defers to first reveal (it rides requestAnimationFrame, as before)
and an unseen one is never counted.
2026-08-08 10:25:26 -07:00
Daniel Salazar 95da802b13 fix: metering hardening (#3527) 2026-08-08 06:10:01 -07:00
Daniel Salazar 0491d5fea7 fix: mcp better uploads (#3528) 2026-08-08 06:09:49 -07:00
Juan Fernando CastroandDaniel Salazar d202be10a9 feat: let apps use another app's data with user consent (#3516)
* feat(perms): add cross-app app-data permission vocabulary

* feat(perms): sweep app grants by permission prefix

* feat(perms): resolve and withdraw cross-app data grants

* feat(kv): support an authorized namespace override and per-key privacy

* feat(kv): gate cross-app KV access behind app-data grants

* feat(fs): allow cross-app AppData access and require a scope to delete

* feat(auth): accept permission lists and gate app-data grants

* feat(perms): add requestAppData to the puter.js SDK

* feat(gui): carry permission lists through the IPC and popup transports

* feat(gui): describe cross-app data requests in the consent dialog

* docs: document requestAppData and per-entry KV privacy

* perf(perms): sweep cross-app grants only for origin-bootstrapped apps

* fix(gui): stop double-encoding cross-app consent text

* fix(perms): close three gaps in cross-app grant enforcement

* fix(kv): meter and batch the per-entry privacy probe

* fix(perms): resolve app identifiers and scopes more strictly in the SDK

* test(perms): cover the cross-app consent flow end to end

* fix: small missing token resolution for app

also adds the same exclusion for the batchPut api, small change

* fix: make resolved actor optional

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-08-08 04:04:07 -07:00
jelveh 2c17476c07 fix(gui): folder pages, live drag-out, and one merge pulse in My Apps
Three fixes to the My Apps folders:

A folder big enough to need it now paginates like the grid outside
(iOS folders page too) instead of growing a scrollbar: pages of
cols x 3 tiles in a scroll-snap scroller with dots below, wheel and
swipe to flip, and a drag held at the card's edge flips folder pages
so reordering works across them. The card's height stays constant
across pages, and a deep-link landing on an app that lives on a later
folder page travels there before the launch morph.

Dragging an app out of the open folder no longer drops it blind:
after a beat held outside the card, the app leaves the folder, the
card closes behind it, and the SAME drag carries on over the grid --
placeholder shuffle, edge page flips, even a drop into another
folder -- so the app lands where the user watched themselves put it.
A quick drop during the beat still lands beside the folder, and
swinging back inside within the beat cancels cleanly.

The folder well's fill played twice on nearly every merge: the rest
countdown anchored at first contact with the target, so the hand
decelerating INTO the tile always read as movement when the dwell
elapsed, and the deliberate refill-from-empty replayed the fill. The
countdown now re-anchors on the move events themselves -- rest is
only visible at its edges -- so the fill that completes is the one
started by the last movement, played exactly once.
2026-08-08 01:28:09 -07:00
Nariman Jelveh 06d94534e8 feat(gui): iOS-style app folders in the My Apps grid (#3525)
* feat(gui): iOS-style app folders in the My Apps grid

Drag one app onto another and let it settle: a well opens under the
target and the drop makes a folder of the two. Dropping onto an existing
folder joins it. A folder opens by growing out of its own icon into a
card over a blurred grid, where its apps can be launched, rearranged,
renamed, or carried back out.

Hovering a tile mid-drag means two things — "push over, I'm passing
through" and "swallow me" — and the tile is barely bigger than its icon,
so pixels can't separate them; motion does. The shuffle is held while
the folder offer stands and fires when the icon leaves the tile, or at
the drop, so a quick drop onto a neighbour still reorders exactly as it
did. The offer itself re-arms rather than cancelling on movement: the
last events of a drag are the ones carrying the icon onto the target and
nothing is dispatched while it rests, so a cancel-on-movement dwell
could never fire at all.

Folders are stored in their own kv key; the grid's ORDER stays entirely
in the existing saved app order, with a folder occupying its first
member's slot and its members contiguous. Every saved order therefore
stays valid with no migration, and an app whose installedApps page
failed to load keeps both its folder and its position. Folders never
nest, one that drops below two apps dissolves, and a corrupt kv value
degrades to "no folders" rather than a broken tab.

Elsewhere:
- Search looks THROUGH folders — a match the user then has to hunt for
  inside one is not an answer.
- Minimize morphs into the FOLDER when an app lives in one, and a
  /app/<name> landing opens the folder so the launch grows out of the
  icon where the app actually is.
- The uninstall FLIP keyed surviving tiles by app name, which a folder
  tile doesn't have; it now keys by identity.

New pure model in appGroups.js with tests; verified end to end in the
running dashboard (create, join, open, rename, reorder, eject, ungroup,
launch-from-folder) alongside plain reorder, search, and uninstall.

* fix(gui): keep the folder name field from inheriting input[type=text] sizing

style.css styles every input[type=text] with `width: 100%` and grows it to
`padding: 7px; border: 2px` on focus. `input.myapps-group-name` matches at the
same specificity, so it only wins the properties it actually declares — width
was never one of them, and the focus rule declared neither padding nor border
width. The name field therefore spanned the entire folder card (so its hover
and focus chip read as a full-width bar rather than the name) and grew 8px
taller the moment it was clicked, shoving the folder's app grid down.

Spell the three out, in both the resting and the focus rule — the same trap
.myapps-search already documents next door.

* fix(gui): size folder icon ghosts to the icon they stand on

Border-box only reaches a folder's icon through `.dashboard * { box-sizing }`,
and every ghost cloned from one is appended to <body>, outside that rule: the
drag ghost, the click-time launch flourish, and the open/minimize morph ghosts
all fall back to content-box, where .myapps-group-icon's 5px padding is added
to the 56px slot. Each ghost rendered 66px square and 5px off, so it visibly
popped at exactly the moment it was supposed to sit flush on the real icon.

State box-sizing on the rule itself so a clone carries it wherever it lands.

* fix(gui): stop a closing folder from swallowing the next click

_closeGroup drops the open class and leaves the overlay in place for
GROUP_PANEL_CLOSE_MS so the card can recede into its tile. The scrim is
`position: fixed; inset: 0` and still hit-testable for that whole quarter
second, so a click on the grid during it landed on the outgoing overlay — whose
handler only re-runs _closeGroup, now a no-op. Shutting a folder and reaching
straight for an app did nothing.

Take the outgoing overlay out of hit-testing; it has no interactive job left.

* fix(gui): keep a folder name typed right up to the moment it closes

The name box commits on blur, and every exit that goes through a pointer blurs
it while the folder is still open — so clicking outside, or launching an app
from inside, keeps what was typed. Escape does not: _closeGroup clears
_openGroupId first and only then moves focus to the tile below (or removes the
card outright), so the blur arrives with no open folder to rename and
_renameGroup drops it. A brand-new folder opens with its name selected for
exactly this edit, so "type Games, press Escape" — the obvious way to dismiss
a dialog — was the path most likely to lose it.

Commit the pending name on the way out, before the folder id is gone.

* fix(gui): close an open folder when the Apps tab is re-entered

The dashboard hides an inactive section and calls onActivate on the way back
in; there is no deactivate hook, so a folder left open survives the round trip
and greets the user still open over a grid they walked away from. Worse,
onActivate's focusSearch then lands the caret in the search box behind the
folder's scrim, and typing filters the grid the card is covering — a modal with
the keyboard pointed outside it.

Shut the folder as the tab comes back: returning to the tab is returning to
the grid.

* fix(gui): move focus into a folder when it opens

Opening a folder called .focus({ preventScroll: true }) on a jQuery
object. jQuery's .focus() shorthand reads a lone non-function argument
as event DATA and binds a handler with it, so it never moved focus:
the folder opened modal over the grid with focus still on the tile
behind its scrim, where Tab walked away through the inert grid instead
of cycling inside the dialog — and the object it bound as a handler
threw a TypeError on that tile's every subsequent focus.

Focus the DOM node instead, as every other focus call in this file
already does.

* fix(gui): stop renaming a folder from swallowing the click that commits it

The folder's name box commits on blur, and blur fires on the PRESS —
before the click that press belongs to. Committing re-rendered, and the
re-render replaced every tile in the open folder, so by the time the
click was dispatched the tile under the pointer was detached and the
delegated handler never saw it. Typing a name and then tapping an app
in the folder — the path a brand-new folder puts the user on — renamed
the folder and did nothing else; the app only opened on a second click.

Rebuild the folder's contents only when they actually differ from what
is on screen. A rename doesn't change them, so nothing is detached, and
a background refresh no longer throws away hover/focus either. Tiles
that survive get their drag resting-rects cleared, since the card can
have moved under them since the rects were taken.

* fix(gui): keep Enter in a folder's name box from leaving the folder

Committing the name with Enter blurred the box, which left focus on
<body> — outside a dialog that is marked aria-modal and that traps Tab
on its own subtree. The next Tab therefore walked off through the inert
grid the folder is covering, exactly what the trap exists to prevent.

Step out onto the folder's first app instead; the same blur still
commits the name. The keystroke is stopped at the box because the
grid's document-level key handler reads Enter on a focused tile as
"launch it", and would otherwise have taken the focus move as its cue
to open an app the user never asked for.

* fix(gui): stop an open folder clipping its own uninstall badges

The folder's grid scrolls, so it clips anything outside its padding box
— and reorder mode's uninstall badge deliberately overhangs the top-left
corner of every tile. The top row's badges therefore rendered as flat
tabs rather than circles, on the one surface where they are a touch
user's only way to uninstall an app they have filed away.

Give the scroller 9px of top padding for the overhang to sit in and take
it straight back off as margin, so the card and everything in it stays
exactly where it was.

* fix(gui): hold page edge-flips while a folder merge is being offered

A tile in the pager's last column sits inside the 60px edge-flip zone, so
resting a dragged icon on it — the folder-making gesture — armed the edge
dwell alongside the merge dwell, and the page flipped out from under the
very folder the user was watching form (the merge target scrolls away
mid-offer, and the drop then resolves against its stale resting rect).
Worst on phones, where 72px tiles overlap the zone across the whole last
column.

A live merge offer now holds the edge flip: entering the zone arms no
dwell while an offer stands, and an offer that arrives during a running
dwell is re-checked at the flip (a resting pointer fires no event that
could clear the timer). Carrying the icon off the tile withdraws the offer
and the hold with it, so deliberate flips — resting in the bare edge
gutter — behave as before.

* fix(gui): make Escape cancel a folder rename instead of saving it

Escape pressed mid-edit fell through to the folder's close handler, and
closing commits whatever the name box holds — so the one key every inline
rename uses for "never mind" stored the abandoned half-typed name. Escape
in the box now puts the stored name back and steps out to the folder's
tiles, exactly the cancel Finder/Explorer taught; with nothing left to
cancel (name untouched) it falls through and closes the folder as before,
so a second press still exits.

* fix(gui): refill the folder well when the merge countdown restarts

The merge dwell pins its stillness anchor at first contact with the tile,
and a hand decelerating INTO a target routinely covers more than the
7px allowance between that moment and the first tick — so the countdown
quietly restarts. The well's fill, tuned to the same 460ms, had already
completed by then and just sat there half-open: a drop it seemed to
promise a folder for actually reordered. Restart the fill with the
countdown, so what the well shows is always the countdown that is
actually running.

* fix(gui): keep keyboard focus inside an open folder across its edits

The folder card is a modal dialog, but three of its flows stranded focus
on <body>, where Tab walks the inert grid behind the scrim:

- Remove from Folder rebuilds the card's grid after the context menu has
  dropped focus, so nothing inside the dialog holds it. The rebuild now
  hands focus back to the same app's tile (or the first) whenever it finds
  focus on <body> — never stealing from an uninstall modal, which holds it
  legitimately.
- The same eject dissolving the folder re-renders the grid right after
  _closeGroup's focus hand-back, replacing the very tile it chose. The
  ejected app's own tile — where the user is looking — takes focus instead.
- A click on the card's empty space focused nothing at all. The card now
  carries tabindex=-1 so such clicks land on it (no visible ring; a focus
  ring around the whole card would misread plumbing as selection), and the
  Tab trap wraps from there instead of stepping off through the scrim.

* fix(gui): size the folder name box to the name it holds

The box sat at the browser's default ~20ch regardless of content: a short
name floated in a hover pill far wider than the word, and a 40-character
one clipped while the card had room to spare. field-sizing: content hugs
the text between a 90px floor and the card's width, iOS-style; engines
without the property keep the default box exactly as before.
2026-08-08 00:28:03 -07:00
Neal Shah aea828ff12 add worker create event and new query methods (#3524) 2026-08-07 17:51:37 -04:00
Daniel Salazar c24f0fe08b fix: mcp fs copy, move, rename (#3517)
* fix: mcp fs copy, move, rename

* feat: more mcp tools for kv
2026-08-07 17:14:40 -04:00
jelveh 085ac116c7 fix(gui): keep app landings through signup, and unstack the session picker
Signing up from a direct /app/<name> landing redirected to '/' on
success, silently dropping the app the URL asked for — the deep link's
launch (and its dashboard intro) never happened. Signup now mirrors
login's pathname-preserving redirect, but only for app-landing routes
(/app/<name>, /desktop/app/<name>): every other route keeps the
historical '/', notably /action/signup, where returning to the same
path would just show the signup form again. The query string stays
dropped, matching login's credential-leak hygiene. This covers all
three signup entry points reachable from a landing: the login cover's
"Sign up", the session picker's "Create Account", and the
must_login_or_signup fallback.

The session picker (UIWindowSessionList) also left itself on top of
the cover windows its two links open, hiding their username fields:
"Create Account" tried to close the picker via the LOGIN window's
c2a selector (which matches nothing in the picker), and "Log Into
Another Account" never closed it at all. Both now close the picker —
in the reload flows only: the no-reload (popup) flows keep it open,
where it doubles as the fallback UI when the login/signup window is
abandoned mid-flow.

Picking an account was already correct (location.reload() keeps the
landing URL); with these fixes all three picker paths, plain login,
and signup all return to the app landing, where the boot replays the
launch and its intro.
2026-08-07 11:51:44 -07:00
jelveh 5538a5d24b feat(gui): let the deep-link intro step aside for users who know it
The intro exists to teach the dashboard's spatial model; once learned it
would only tax every bookmarked landing. Two mechanisms remove that tax:

- Interruptible: any real user input (pointer/key/wheel, isTrusted only)
  during the intro skips the remaining choreography and launches at
  once — it wakes the in-flight beat sleep, so the skip is immediate.
  Input never cancels the launch itself and is never swallowed;
  whatever it was doing (typing a search, clicking a tile) still
  happens.

- Exposure decay: after 3 delivered — or deliberately skipped — intros
  the beats collapse and the sequence plays in one breath, exactly like
  a warm tile click. Counted per account in kv (kv.incr, atomic across
  devices; capped at the threshold so the key stops changing) rather
  than per device: the lesson lives in the user's head and the account
  follows them, while localStorage would also bleed between accounts on
  a shared browser. Only real exposures count: a delivered flourish or
  an active skip with the grid on screen — hidden tabs, timeouts, and
  no-tile landings teach nothing and don't count. The animated page
  flip is exempt from decay: it isn't a repeated lesson, it's live
  wayfinding to where this app lives, and its settle is needed anyway
  to put the tile in view for the morph. Every failure mode (slow or
  failed kv read) errs toward teaching once more, never toward never
  teaching.

Also bounds the intro's wait on the app-list load by the same deadline
as the tile wait — fetch has no timeout of its own, so a stalled
installedApps request used to hold the deep-link launch hostage
indefinitely.
2026-08-07 11:23:47 -07:00
jelveh 14c14fb5c5 feat(gui): animate the pager flip when a deep-linked app lives on a later page
The deep-link intro used to flip to the tile's page instantly behind the
grid's load-fade, so landings on off-page apps woke up on page N with
nothing but the pager dots hinting a move happened. Now the grid always
reveals on its first page, holds the grid beat, and then visibly travels
to the tile's page before the tile's flourish — the journey shows where
the app lives, which is also where minimize will put it back.

Smooth scrollTo has no reliable completion event across engines, so the
travel uses a settle allowance (DEEP_LINK_INTRO_FLIP_SETTLE_MS, same
pattern as the drag code's DRAG_FLIP_SETTLE_MS): the scroll's ~450ms
plus a rest so the landing reads before the tile pops. First-page apps
skip the flip entirely and are unaffected; the hidden-page bail-out is
re-checked after the flip settles.
2026-08-07 11:12:03 -07:00
jelveh 41fc8d890b feat(gui): play the tile click→morph→open intro on /app/<name> landings
A direct landing on /app/<name> now plays the same sequence a real
Apps-tab tile click does — grid appears, a beat, the tile's icon ghost
pops out of its slot, a beat, and the window morphs out of the tile —
so the landing shows the user what is being opened and where minimize
puts it back.

TabApps.beginDeepLinkLaunch waits (3s cap) for the tile to be genuinely
visible (list loaded, render done, pager flipped to the tile's page,
load-fade revealed, icon painted), paces the beats, and claims the app
in _launchingApps so a click mid-intro can't spawn a duplicate. The
launch's app-info fetch is prefetched in parallel so the intro never
delays the app's own round-trip. No tile, animations off, reduced
motion, or a background tab (hidden pages throttle timers and defer
rendering) all skip straight to the immediate plain-fade launch.

Also fixes the tab title sticking as the app's name after closing a
deep-linked app: the landing's replaceState('/') committed the
dashboard's history entry while the page still carried the server's
app-name title, and Chrome shows an entry's stored title when close/
Back traverses onto it — document.title is now reset before the
replaceState so the entry is stamped with the dashboard's own title.
2026-08-07 10:34:21 -07:00
Reynaldi Chernando 45bdc46df0 Remove deprecated models 7 aug (#3522) 2026-08-07 23:20:50 +07:00
jelveh 82ca390bf4 fix(gui): create-in-folder via sidebar right-click renders in target dir
Right-clicking a sidebar/breadcrumb folder in the dashboard Files tab and
choosing New > Folder (or any file type) created the item in the right-
clicked folder correctly, but the UI inserted the new row and started the
inline rename in whatever directory was currently open — so the item
appeared to be created in the wrong place until a refresh.

Now, when the creation target isn't the directory on screen, navigate to
the target and run the select + rename flow there; same-directory
creation keeps the incremental insert.
2026-08-07 08:49:13 -07:00
Neal Shah edfd67a83c fix PUT-1444 (#3515) 2026-08-06 17:11:04 -04:00
404oops ff16ec8900 Add Neuralwatt support (#3509) 2026-08-06 17:01:33 -04:00
Daniel Salazar c7edfc0c74 perf: improve metering perf + add extra compat for monthly charges (#3513)
closes PUT-1445 and PUT-1446
2026-08-06 10:26:47 -07:00
velzie 09f30d693c Peer: allow anontoken in place of authentication (#3501) 2026-08-06 02:03:53 -04:00
Reynaldi Chernando a9ec15d121 Update NPM readme (#3510) 2026-08-05 17:07:02 -07:00
Daniel Salazar eb53c842dd fix: oidc issues with cache (#3512)
closes #3497
closes #3502
2026-08-05 17:05:31 -07:00
Daniel Salazar 1be5ce45f7 feat: support dekstop app linking again (#3511) 2026-08-05 14:16:02 -07:00
Neal Shah 294055e055 PUT-1436, PUT-1435, PUT-1371 (#3507) 2026-08-05 17:11:50 -04:00
jelveh 036008d8a7 fix: keep uninstalled recommended apps from resurrecting in the Apps tab
Uninstall only revokes permissions, but the recommended launch list is a
global hardcoded set that knows nothing about per-user revokes — so an
uninstalled recommended app's tile came back on every reload. Persist
uninstalled app names in kv (dashboard_removed_apps) and filter only the
recommended merge against them; installedApps is never filtered, so a
genuinely (re)installed app always shows.
2026-08-05 11:01:14 -07:00
Daniel Salazar 6df292f495 fix: PUT-1429 (#3508) 2026-08-05 01:17:34 -07:00
Daniel Salazar 40c8d09f05 fix: PUT-1401 (#3504) 2026-08-04 16:37:33 -07:00
meridah7 4a211b9092 docs: document the provider option and refresh the vendor list in chat() (#3505) 2026-08-04 16:22:03 -07:00
meridah7 24fc0b0e5f fix(ai-chat): infron default model is dropped by the aggregator alias rule (#3506)
* fix(ai-chat): use a resolvable default model for the infron provider

* test(ai-chat): assert the infron default model is in the served catalog
2026-08-04 16:21:55 -07:00
Nariman Jelveh f58772ec33 feat: swap browser favicon to the focused app's icon in dashboard 2026-08-04 14:16:32 -07:00
Neal Shah eb830799c0 add limit bypass (#3500)
* add limit bypass

* update test
2026-08-03 19:43:59 -04:00
jelveh 8df72ef343 fix: dashboard app drawer icon popping in late
The drawer resolved its own icon URL (a differently-sized variant under a
different URL than the dashboard tiles use), so its <img> fetched cold and
the icon popped in mid-intro. Reuse the bitmap an Apps-tab tile or Home-tab
recent already decoded — instant from the image cache; when nothing is
rendered yet (deep links), fade the icon in on load instead of popping, and
fall back to the generic app icon on a failed fetch.
2026-08-03 11:50:43 -07:00
jelveh 63b2fc34d9 feat: highlight pasted items in dashboard files tab
Pasted (copied or cut) items now land selected, the same treatment
uploads get. copy_clipboard_items resolves with the created items'
paths instead of firing and forgetting, moveClipboardItems returns
each move response's authoritative final path (with Keep Both the
landed name differs from the source's), and both dashboard paste
entry points refresh with strong consistency and hand the new paths
to selectUploadedRows. Pastes into a folder that isn't rendered match
no rows and the highlight is a no-op.
2026-08-03 06:52:40 -07:00
Nariman Jelveh 33d7324a24 feat: 'Keep Both' option in name-conflict dialogs (#3496)
Pasting or moving onto an existing name only offered Replace or
Cancel. Add a macOS-style middle ground: Keep Both retries the
operation with dedupe_name, landing the item as "name (1).ext" and
leaving the existing item untouched.

- puter-js move.js now forwards dedupeName to the wire (copy already
  did; move dropped it).
- All four conflict dialogs offer the new button: copy_clipboard_items,
  copy_items and move_items in helpers.js, and the dashboard's
  moveClipboardItems. Multi-item selections show
  Replace / Replace all / Keep Both / Skip.
- New keep_both translation key (other locales fall back to English).
2026-08-03 06:39:18 -07:00
Nariman Jelveh 62be61642d fix: phantom name conflict when pasting onto a just-renamed path (#3495)
Renaming (or moving) an entry left the OLD path's cache key serving
the pre-update entry for the full 60s TTL: updateEntry invalidated
only keys derived from the updated row, whose path is already the new
one. Pasting an item under the freed name then hit the stale cache in
the collision check and reported a conflict for a file that no longer
exists — and accepting the Replace it offered deleted the renamed file,
since the stale entry carries its uuid.

Read the pre-update entry when the patch changes the path and
invalidate its keys alongside the new ones. The move flow was shielded
by the outer.gui.item.moved cache-invalidation handler at the
controller layer; rename had no such band-aid, and fixing the store
covers every caller regardless of which events fire.
2026-08-02 23:53:59 -07:00
Nariman Jelveh 8cebbcf3be fix: thumbnails breaking after copy — duplicate the S3 object per copy (#3494)
A copied fsentry kept its source's thumbnail pointer verbatim, so both
rows shared one S3 thumbnail object. fs.remove.node deletes the
pointed-to object, so the first removal among the sharers — e.g. the
remove performed by a replace-on-copy — broke every other sharer's
thumbnail. FSService already emits fs.copy.node for exactly this
reason (its doc comment describes the duplication), but the thumbnails
extension never subscribed to it.

Add the missing handler: S3-copy the thumbnail to a freshly minted key
and repoint the copied row, under the same only-keys-we-minted trust
rule as the read/remove paths. If the shared object is already gone
(pre-fix damage), null the pointer instead of leaving the row
advertising a thumbnail it doesn't have.
2026-08-02 23:22:24 -07:00
Nariman Jelveh 5c2a423658 fix: progress window covering paste conflict dialog; ghost row after Replace (#3493)
* fix: progress window covering paste conflict dialog; ghost row after Replace

Pasting a copied file onto a name collision buried the Replace/Cancel
dialog under a stuck 'Preparing...' progress window, and answering
Replace left a stale duplicate row in every client.

- helpers.js: copy_clipboard_items armed its delayed progress window
  with a 0ms timer (its siblings use 2s), so the window opened
  instantly over the dialog. Use 2s, and in all three of
  copy_clipboard_items / copy_items / move_items pause the timer while
  a conflict dialog (or the own-location / trash-deny alerts) is
  waiting for input, re-arming it after. A window that opens
  mid-operation now shows the current file instead of a stuck
  'Preparing...', and the trash-deny bail no longer leaks a timer that
  opened an orphan window after the operation ended.
- LegacyFSController: /copy and /move dropped the legacy 'overwritten'
  response field and never emitted item.removed for the entry an
  overwrite deleted, so clients kept a ghost row until re-listing.
  Resolve the entry before the operation, return it, and emit
  item.removed on success.
- helpers.js: copy_items read resp[0].overwritten but removed
  resp.overwritten (always undefined), and the data-uid cleanup
  selectors were unquoted — invalid CSS when a UUID starts with a
  digit. Fixed all three sites.

* test: pin the v1 overwrite/collision wire contract for move and copy

The collision tests asserted only statusCode 409, which is how the
item_with_same_name_exists code regressed to 'conflict' unnoticed and
broke every replace/skip prompt in the GUI. Assert the legacy code and
entry_name explicitly, and add controller tests for the overwrite path:
the replaced entry must ride along as 'overwritten' in the /copy and
/move responses and be announced via outer.gui.item.removed so clients
drop its row.
2026-08-02 22:17:10 -07:00
Nariman Jelveh 1a9f025db6 fix: surface name collisions when pasting instead of failing silently (#3492)
Cut+paste onto a folder containing an item with the same name failed
with no feedback: the dashboard's moveClipboardItems swallowed the
error into console.error and cleared the clipboard, and the rewritten
backend broke the v1 wire contract the GUI's conflict prompts key on.

- FSService: move() and copy() collisions again return
  item_with_same_name_exists + entry_name (the contract the write path
  already preserves) instead of a generic 'conflict' code. This also
  restores the desktop's existing Replace/Skip dialogs, which were
  silently broken against the new backend.
- Dashboard: moveClipboardItems now mirrors the desktop's move_items
  conflict flow (Replace / Replace all / Skip / Cancel, retry with
  overwrite) and surfaces other move errors in an alert.
- keyboard.js: the desktop's global Ctrl+V handler threw an uncaught
  TypeError on every paste in dashboard mode (its file container has
  no data-path); guard it — which also prevents a double-paste if
  dashboard containers ever gain one.
2026-08-02 21:33:01 -07:00
jelveh a4667073f5 Update RecommendedAppsService.ts 2026-08-02 20:55:33 -07:00
jelveh 23d251e482 fix: dashboard files list blanking during same-directory refresh
Re-rendering the directory already on screen (after an upload, sort
change, undo, etc.) used to clear the list and show a spinner before
the readdir round-trip, blanking the pane for the whole fetch. Keep
the current rows visible until the fresh listing arrives, then swap
the DOM in one pass and restore the scroll position. Navigation to a
different directory still clears immediately.
2026-08-02 20:30:20 -07:00
jelveh 0c9dfb03e2 Show action bar only in mobile select mode 2026-08-02 19:49:14 -07:00
Nariman Jelveh 6dc928e0c3 fix: file picker dialogs going behind parent window when dragged by titlebar 2026-08-02 17:38:25 -07:00
Daniel Salazar 84582ee33d fix: alarm channels (#3491) 2026-08-01 15:55:06 -07:00
Daniel Salazar 116d6e6663 tests: big test push for better coverage (#3490) 2026-08-01 14:29:59 -07:00
Daniel Salazar 7d0d44aef9 feat: slack alarms (#3489) 2026-08-01 13:12:41 -07:00
Daniel Salazar 3ac6c8532e fix: open ai cost (#3488) 2026-08-01 12:34:36 -07:00
Nariman Jelveh f6a50c4fb4 Update recommended apps list order
Reorders and updates the recommended apps list: removes 'butler', 'code', and 'traffic-tap-puzzle'; adds 'contacts', 'diagram', and 'basketball-tap' (relocated from end); adjusts overall ordering of several apps.
2026-08-01 12:26:44 -07:00
Daniel Salazar c8113d7514 fix: auth message popups (#3487)
* fix: auth message popups

* remove v1 auth
2026-08-01 10:45:16 -07:00
jelveh 6159bc9ae1 Polish dashboard settings button styling
Update `.dashboard-settings-card .button` to use dashboard theme tokens for text, background, and borders, and add clearer hover/focus/active states with subtle elevation and transitions. Also override disabled styles in this context so dark mode no longer shows hard-coded light greys from the base `.button:disabled` rule.
2026-08-01 00:27:27 -07:00
Daniel Salazar 3a11d0d1f1 feat: sandboxed app workers (#3486) 2026-08-01 00:02:06 -07:00
jelveh 535cf5753e Merge branch 'main' of https://github.com/HeyPuter/puter 2026-07-31 14:14:11 -07:00
jelveh cee4a6ccce Improve app reorder mode on touch devices 2026-07-31 14:14:04 -07:00
Daniel Salazar 1f1f95c2f8 fix: auth me for local dev (#3484) 2026-07-31 14:04:59 -07:00
jelveh f600e857b0 GUI: touch reorder mode for the dashboard Apps tab
Long-press-to-drag can't be made reliable on touch: touch-action is
consulted at gesture start, so the pager's pan-x claims the finger
before a drag can begin (worst on iOS). Replace it with an explicit
edit mode - a cog button (touch-primary devices only) enters it, tiles
jiggle and drag on first movement, iOS-style x badges uninstall, and
Done exits. Drops still persist immediately, same as desktop.

Also raise the drag ghost above the window z-index bands; it was
rendering invisibly behind the fullpage dashboard window.
2026-07-31 13:46:17 -07:00
Neal ShahandDaniel Salazar 08d1708378 change cors auth path (#3464)
* change cors auth path

* undo oidc ref changes

* scary OIDC state changes

* fix: bad cors signin

* puterjs changes

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-07-31 01:55:01 -04:00
Nariman Jelveh f1bc065730 GUI: fix dev-server port-retry crash under Express 5 (#3482)
Express 5's app.listen wraps the listen callback in once() and also
invokes it on 'error', with the error as the first argument — at which
point server.address() is null, so the startup log threw a TypeError
and killed the process before the EADDRINUSE handler could try the
next port. Bail out of the callback when it receives an error and let
the 'error' listener own the retry.

Verified: with 4000 occupied the server now logs the retry and comes
up on 4001; with the port free it binds 4000 as before.
2026-07-30 18:11:12 -07:00
Nariman Jelveh b55782f066 GUI: add npm start --server=<domain> to run the GUI against a remote backend (#3481)
- npm start --server=puter.com (or -- --server=...) skips the local backend
  and serves the bundled GUI locally, pointed at the remote server's API.
  Bare domains resolve to https://api.<domain>; full origins are used
  verbatim. gui_origin points at the remote origin so /whoarewe, login,
  anti-csrf, socket.io, and builtin apps hit the real backend (CORS-open).
- --extensions=<dir>[;<dir>...] bundles out-of-tree GUI extension
  directories (sugar for PUTER_GUI_EXTENSION_PATHS). Their imports resolve
  as if the files lived in src/gui/src/extensions, with the extension's own
  files taking precedence, and bare imports fall back to the repo-root
  node_modules.
- Fix the bit-rotted dev-server: Express 5 wildcard routes, pass gui()
  params (previously called with none), inject the service_script shim,
  load bundle.min.js + bundle.min.css in prod mode, serve /sdk, and
  properly await the webpack build (it previously resolved immediately).
2026-07-30 16:29:54 -07:00
jelvehandClaude Fable 5 2c9ae3489e Suggested apps: rank registered apps above the editor fallback
For extensions with no intentional built-in mapping (doc, docx, and
every other unmapped type), suggestionsForExtension fell back to
['editor'], and #resolveForExtension always placed built-ins ahead of
apps from app_filetype_association. Since suggested[0] drives the GUI's
double-click open path and /open_item, a .docx defaulted to opening as
plain text in editor even when a word processor explicitly registered
the extension.

Tag the unknown-extension result as a fallback and order third-party
filetype-association apps ahead of it. Intentional mappings (code, txt,
md, images, pdf, media) keep built-ins in the head slot as before, and
the editor guess still appears as a last-resort option.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-30 13:51:01 -07:00
Daniel Salazar b28b41c90b otel: better telemetry numbers (#3480) 2026-07-30 13:23:16 -07:00
Nariman Jelveh f34c4dc335 Apps: normalize filetype associations to bare lowercase extensions (#3479)
* Apps: normalize filetype associations to bare lowercase extensions

Suggested-apps lookups match app_filetype_association rows against the
bare lowercase extension ('docx'), but writes stored whatever the
developer typed. Rows like '.docx' never matched, so those apps
silently dropped out of Open With suggestions.

AppStore now canonicalizes on write (trim, lowercase, strip leading
dots, dedupe, drop empties) and tolerates the dotted legacy form on
read: getAppsByFiletype normalizes the requested extension, matches
both 'docx' and '.docx', and dedupes apps associated under both forms.
Cache invalidation keys are normalized the same way. Existing dotted
rows work without a data migration.

* Update apps tests for extension canonicalization

Adjust apps API tests to match current normalization behavior for `filetypeAssociations`: extension values are stored as lowercase bare extensions (e.g. `.txt` -> `txt`), while MIME types remain unchanged. Added inline comments in both test suites to document this expected remap.
2026-07-30 13:13:14 -07:00
Daniel Salazar 5a157197b6 fix: PUT-1398 (#3478) 2026-07-30 01:42:40 -07:00
Daniel Salazar 8a711e0254 driver controller change (#3477)
* fix: launch app

* driver controller change
2026-07-30 00:41:29 -07:00
jelveh a22321daa2 Dashboard: closing a launched app goes home, not into its launcher
An app launched by another app minimizes its launcher (iOS-style
takeover). Closing the child then popped onto the launcher's history
entry, and the popstate handler restored it — so quitting an app threw
the user into a different full-screen app they had not asked for, zoom
animation and all. Closing an app should go home.

Mark the launcher when it is minimized FOR a child
(data-minimized_for_child). If it still carries that mark when the child
closes, the close's single history hop lands on the dashboard instead:
the entry is rewritten to the dashboard's route and the launcher is left
minimized, still running behind its tile's running dot. No extra history
traversal — the joint session history an app's iframe shares is exactly
what pop_dashboard_app_url's watchdog exists to survive, so the hop
count is unchanged.

Back is untouched and still returns to the launcher — that is the
navigation gesture, whereas close dismisses. Any restore (Back, tile
click, Forward) clears the mark, so quitting a child after the user has
brought the launcher back leaves the launcher alone.
2026-07-29 22:04:10 -07:00
jelveh fbf1bd18b1 Dashboard: an app launched by another app takes over the tab
In dashboard mode apps are full-tab experiences, but an app launched by
another app (puter.ui.launchApp) opened as a floating titlebar window
over its full-tab parent: the parent stayed on screen around its edges,
couldn't be raised (focusWindow never raises stay_on_top windows), and a
child with no Apps-tab tile had no switcher to come back to once
minimized.

Give child launches the same treatment as tile launches — maximized, so
they get the headless chrome and control drawer — and minimize the
parent behind them, iOS-style. State only: the parent's /app/<name>
history entry already sits beneath the child's, so Back from the child
(and the child's close, which consumes its own entry) lands on the
parent's entry and the existing popstate handler restores it. The parent
keeps running while hidden, so parent/child IPC is unaffected.

Explorer keeps its windowed form, background apps don't minimize their
parent, and an explicit `maximized` option still wins. Desktop mode is
untouched: both changes are gated on is_dashboard_mode, so an app
launching an app there still gets a floating child over a visible
parent.
2026-07-29 21:36:47 -07:00
jelveh dc371fc4eb Windows: keep dialogs and their apps in the stay-on-top z band
focusWindow re-raised a focused window's parent (and children) with the
bare z counter. For a file dialog owned by a fullpage/dashboard app that
DEMOTED the app out of the 99999999+ stay-on-top band it was created in,
burying app and dialog under every other open app window — opening a
file picker from an app stacked over another (e.g. an app launched from
Dev Center) made both vanish beneath the window below. The dialog itself
had the same flaw: raised into the plain counter band, under every app.

Raise each window within its own stacking band instead: stay-on-top
windows (and windows hanging off one through parent_uuid, walked up the
chain) get 99999999 + counter; everything else keeps the bare counter —
so desktop stacking arithmetic is unchanged. Same demotion family as the
showWindow restore fix in #3427.
2026-07-29 21:11:21 -07:00
jelveh 97e4e78df4 Dashboard files: drop the root crumb from the breadcrumb
Every path rendered as "> Puter > user > ...", but the root crumb is
noise on anything beneath it. Render it only at the root itself, where
it's all there is to show (the Up button can still reach /, and an
empty breadcrumb bar there would strand the user). Each crumb keeps
its leading caret, including the first.
2026-07-29 20:12:22 -07:00
jelveh 007649296b Keyboard: stop swallowing Enter when nothing claims it
The global Enter handler preventDefault'ed unconditionally, then
returned without acting unless a launch-menu item, context-menu item,
or selected item existed. That suppressed the browser's native Enter
activation on whatever was focused — a focused link or button (e.g. a
dashboard sidebar item, a dialog button) did nothing on Enter.

Move preventDefault/stopPropagation into the branches that actually
handle the key; otherwise fall through so native activation runs. The
handled paths are unchanged: Enter still opens the selected item,
launch-menu entry, and context-menu entry.
2026-07-29 19:40:16 -07:00
jelveh 9392aed402 Dashboard: restyle keyboard focus ring on sidebar controls
Sidebar items (and the user/collapse/close buttons) had no
:focus-visible style, so keyboard focus painted the UA's default blue
ring — hard-edged and bleeding outside the item. Use the dashboard's
own selection ring instead, tucked inside the control so it hugs the
rounded corners. :focus-visible only, so mouse clicks never paint it.
2026-07-29 19:40:06 -07:00
jelveh 32e417fe79 Dashboard files: clicking an open file returns to its window
In dashboard mode there is no taskbar, so a minimized app window's only
switcher was its Apps-tab tile — invisible from the Files tab. Opening a
file, minimizing, and clicking the file again launched a second instance
of the app, stranding the first (and any unsaved edits) somewhere
unreachable.

Make the file row itself the switcher:

- launch_app stamps the opened file's uid on the window (data-file_uid;
  the signature's uid wins so shortcuts resolve to their target)
- open_item (dashboard mode) restores/focuses an existing window that
  has the file open instead of launching a duplicate, mirroring the
  Apps-tab tile's single-instance behavior — keyed by file, not app, so
  opening a different file still gets its own instance
- re-clicks while a launch's fetches are still in flight are swallowed
  (same idea as TabApps._launchingApps, keyed by file uid, TTL'd so a
  failed launch can't swallow clicks forever)
- dashboard file opens now default to maximized, so they get the same
  headless full-tab chrome + control drawer as tile launches instead of
  a floating titlebar window
- rows show a dot while their file is open in a (possibly minimized)
  window — under the name in grid view, inline after it in list view —
  driven by the existing dashboard-app-windows-changed event

Desktop mode is untouched: the reuse branch and maximized default are
gated on is_dashboard_mode, and opening the same file twice there still
creates two windows as before.
2026-07-29 19:03:33 -07:00
Daniel Salazar 0e1f617401 fix: errors for bad keys (#3475) 2026-07-29 15:58:53 -07:00
Felix-Ayush bd6efaaa4e fix(puter-js): ignore peer offers for unknown connections (#3473)
createAnswer ran outside the connection null-check, so stale or unknown
offer ids threw on the signalling socket. Guard the whole offer path and
return the #message promise from onmessage so callers can observe errors.
2026-07-29 15:58:39 -07:00
Nariman Jelveh 2b6dc528ce Merge branch 'main' of https://github.com/HeyPuter/puter 2026-07-29 15:48:21 -07:00
Nariman Jelveh 9368ad57b6 Preserve UI state across tab visibility refresh 2026-07-29 15:47:59 -07:00
Felix-Ayush e97d338f3a fix(ai-chat): preserve max_tokens 0 for Alibaba, DeepSeek, and Claude (#3472)
These providers used `max_tokens || default`, so an explicit zero was
replaced by the fallback. Use nullish coalescing so 0 is forwarded.
2026-07-29 18:38:31 -04:00
Nariman Jelveh a87a517b82 Anchor file row menu to button edge
File row overflow menus in the dashboard now open anchored to the ⋯ button (below it with right-edge alignment) instead of pointer position, improving placement consistency. UIContextMenu now supports `position.right` for right-edge pinning, and the ⋯ trigger keeps an active visual state while its menu is open so hover styling doesn’t drop when the cursor moves onto the menu.
2026-07-29 15:25:51 -07:00
Nariman Jelveh dbb52171c8 Refine dashboard file row hover styling 2026-07-29 15:05:21 -07:00
Daniel Salazar 6c020318f9 perf: traces and perf improvements (#3471) 2026-07-29 14:35:15 -07:00
Nariman Jelveh e16994db3b Fix I-beam cursor on dashboard file rows in Safari
Safari ignores user-select: none for cursor styling, showing a text I-beam over file rows. Setting cursor: default explicitly fixes this and is inherited by child elements.
2026-07-29 14:18:50 -07:00
Daniel Salazar 8491a0b55d fix: limit api pointing to apps only (#3470) 2026-07-29 14:03:27 -07:00
Felix-Ayush 710e4da4c0 fix(puter-js): size img2txt data URIs by decoded bytes (#3457)
Match speech2txt/speech2speech and reject on decoded payload size, not
data-URI string length. Base64 expands ~4/3, so string-length checks
falsely rejected valid images under the 10MB limit.
2026-07-29 12:21:21 -07:00
Felix-Ayush fd1f4e2ac2 fix(puter-js): forward temperature 0 and max_tokens 0 in ai.chat (#3455)
Truthy checks dropped legitimate zero values, so deterministic sampling
never reached the driver. Match the stream option and use !== undefined.
2026-07-29 12:21:15 -07:00
Kobi Hikri 70bf03a6e7 ci: attach provenance and SBOM attestations to the published image (#3463) 2026-07-29 12:21:02 -07:00
Felix-Ayush 6f90d7a199 fix(ai-chat): forward temperature 0 and max_tokens 0 to providers (#3456)
Completions-family providers used truthy spreads, so temperature: 0 and
max_tokens: 0 were stripped before the SDK call. Align with Responses
providers that already use !== undefined.
2026-07-29 11:49:36 -07:00
Daniel Salazar f0b1947c54 feat: make uninstall also revoke + dashboard css changes (#3469)
* feat: make uninstall also revoke

* fix: wording for free trial
2026-07-29 10:40:43 -07:00
Daniel Salazar 92193ff11f chore: kill collector (#3466) 2026-07-28 19:02:49 -07:00
Daniel Salazar f9396b7283 fix: thumbnail res (#3465) 2026-07-28 17:31:01 -07:00
Sauhbagya PrasadandDaniel Salazar 7cf3e03433 Replaced Nginx with Caddy (#3378)
* Replace nginx with Caddy

* updated caddy location

* seprate logic for local and with domain caddy setup

* updated install

* updated install scripts

new ps1 script with caddy, and removed instances of nginx

* fix Caddy Host routing, restore TLS/SELinux/healthcheck, update docs

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-07-28 14:52:13 -07:00
Neal Shah 0a61b4e78b add fixes 1 (#3458)
* add fixes 1

* change socketio test

* clarify that success/error callbacks are legacy

* fix wisp issues
2026-07-28 17:34:12 -04:00
Reynaldi Chernando 1db4324c02 remove deprecated ai models (#3444)
* remove deprecated ai models

* test

* fix tests
2026-07-28 12:41:04 -07:00
Daniel Salazar 69529ee0db perf: try to improve app opens speed (#3462)
* perf: try to improve app opens speed

* make /rao non blocking to allow faster conn swap
2026-07-28 11:12:44 -07:00
877 changed files with 87936 additions and 12641 deletions

No files matched your search

+2 -2
View File
@@ -2,9 +2,9 @@
# docker-compose.full.yml up -d`. None of the defaults below are safe for
# anything beyond a local laptop test.
# ── Public-facing ports (nginx) ---------------------------------------
# ── Public-facing ports (Caddy) ---------------------------------------
HTTP_PORT=80
# HTTPS_PORT=443 # uncomment after you enable TLS in nginx/nginx.conf
# HTTPS_PORT=443 # uncomment after you enable TLS in caddy/Caddyfile
# ── MariaDB ------------------------------------------------------------
MARIADB_ROOT_PASSWORD=replace-with-strong-password
+32
View File
@@ -13,6 +13,9 @@ on:
- 'tools/**'
- 'package.json'
- 'package-lock.json'
# The type-check job below reads these.
- 'tsconfig.json'
- 'tsconfig.build.json'
- '.github/workflows/backend-tests.yaml'
permissions:
@@ -20,6 +23,35 @@ permissions:
pull-requests: write
jobs:
# `tsconfig.build.json` builds with `noCheck: true`, so nothing else in CI
# runs the type checker and a missing export compiles to `undefined`,
# surfacing only when the call is finally reached at runtime. This diffs
# against tools/typecheck-baseline.json and fails only on *new* errors.
#
# The consuming repo runs an equivalent gate, but only once someone bumps the
# submodule pointer — too late to keep the error off this repo's default
# branch. Hence a gate here, on this repo's own pull requests.
#
# Its own job rather than a step in `test`: that one runs a base/PR matrix for
# coverage comparison, and the check only needs the PR ref, once.
typecheck:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '24'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Type check (new errors only)
run: npm run typecheck
test:
runs-on: ubuntu-latest
strategy:
+2
View File
@@ -87,6 +87,8 @@ jobs:
platforms: linux/amd64,linux/arm64
context: .
push: true
provenance: mode=max
sbom: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
+27
View File
@@ -23,6 +23,33 @@ permissions:
pull-requests: write
jobs:
# The JSDoc in src/puter-js/src is the source of truth for the SDK's public
# types. The declarations shipped to npm are generated from it at build time
# and never committed, so what needs guarding is the JSDoc itself: this
# generates the declarations and type-checks the published surface without
# skipLibCheck, which is how broken re-exports used to go unnoticed.
#
# Its own job rather than a step in `test`: that one builds bundles and
# installs a browser, and this needs neither.
types:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '24'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Check the puter.js JSDoc produces declarations that type-check
run: npm run check:puterjs:types
test:
runs-on: ubuntu-latest
+9 -9
View File
@@ -10,6 +10,7 @@ Use these as the source of truth before exploring further:
- [doc/architecture.md](doc/architecture.md) — backend layered stack (controllers → drivers → services → stores → clients), `PuterServer` wiring, `Context` (ALS), and extensions.
- [doc/contributing-apis.md](doc/contributing-apis.md) — adding and maintaining public APIs end to end (backend surface → puter.js → types → docs → tests). Follow it for any API work.
- [doc/pagination.md](doc/pagination.md) — the one pagination convention for list APIs (limit/cursor/offset/includeTotal, envelope shape, cursor semantics).
- [doc/alarms.md](doc/alarms.md) — raising alarms and picking a severity (what pages, what only gets recorded), plus the config that routes them.
- [doc/self-hosting.md](doc/self-hosting.md) — running Puter outside hosted infra.
- [CONTRIBUTING.md](CONTRIBUTING.md) — testing, security, AI-assisted code, PR conventions, Boy Scout Rule.
- [SECURITY.md](SECURITY.md) — how to report vulnerabilities (do not file them publicly).
@@ -97,24 +98,23 @@ Follow the same layered structure inside an extension — unless it only needs a
[src/puter-js/](src/puter-js/) is the public SDK. It ships live from `https://js.puter.com/v2/` with no version pinning — every existing app picks up changes immediately. Treat every observable behavior (signatures, response fields, error codes) as something a production app depends on.
Layout: SDK modules in [src/puter-js/src/modules/](src/puter-js/src/modules/) (one file or directory per area — `FileSystem/`, `KV.js`, `ai/`, …), shared plumbing in [src/puter-js/src/lib/](src/puter-js/src/lib/), hand-maintained type declarations in [src/puter-js/types/](src/puter-js/types/), API tests in [src/puter-js/tests/api/](src/puter-js/tests/api/), UI e2e tests in [src/puter-js/tests/e2e/](src/puter-js/tests/e2e/), developer docs in [src/docs/](src/docs/).
Layout: SDK modules in [src/puter-js/src/modules/](src/puter-js/src/modules/) (one file or directory per area — `FileSystem/`, `kv/`, `ai/`, …), shared plumbing in [src/puter-js/src/lib/](src/puter-js/src/lib/), generated (gitignored) type declarations in `src/puter-js/types/`, API tests in [src/puter-js/tests/api/](src/puter-js/tests/api/), UI e2e tests in [src/puter-js/tests/e2e/](src/puter-js/tests/e2e/), developer docs in [src/docs/](src/docs/).
Language & typing: puter.js source is plain JavaScript — never TypeScript files — typed via JSDoc. Reference the hand-maintained declarations in [src/puter-js/types/](src/puter-js/types/) with `import(...)` types rather than re-declaring shapes:
Language & typing: puter.js source is plain JavaScript — never TypeScript files — typed via JSDoc. **The JSDoc is the source of truth for the SDK's types.** `src/puter-js/types/` is `tsc --emitDeclarationOnly` output: the SDK build (`npm run build` in `src/puter-js`) generates it, the npm tarball ships it so TypeScript consumers get declarations, and git ignores it — it is never committed and never edited by hand. `npm run check:puterjs:types` generates it in CI and type-checks the result without `skipLibCheck`. The one hand-written declaration file is [src/puter-js/index.d.ts](src/puter-js/index.d.ts), which decides what is public and re-exports the generated names.
Declare a shape where it belongs, and reference it with an `import(...)` type from elsewhere. A shape more than one file needs lives in the module's `types.js`; a shape shared across modules lives in [src/puter-js/src/lib/types.js](src/puter-js/src/lib/types.js); a shape with one consumer can stay next to it:
```js
/** @typedef {import('../../types/modules/ai').ChatOptions} ChatOptions */
/** @type {ChatOptions} */
const options = { model: 'gpt-5-nano' };
/** @typedef {import('./types.js').KVOptConfig} KVOptConfig */
```
When a shape only exists locally, declare it as an inline object literal — not the `@typedef {Object}` + `@property` list form — and use `unknown` over `*`:
Use `@typedef {Object}` + `@property` for any shape whose fields need documenting — it is the only JSDoc form that carries a doc comment per field into the generated declaration. Keep the inline object-literal form for small internal shapes with nothing to say about each field, and prefer `unknown` over `*`:
```js
/** @typedef {{ key: string, value: unknown }} KVEntry */
```
Public (exposed) methods must carry JSDoc types — parameters and return value, matching the `.d.ts` declarations exactly. Unexposed/private helpers are typed at the contributor's discretion: annotate where it helps the next reader, and either way keep them clean.
Public (exposed) methods must carry JSDoc types — parameters and return value — with one `@overload` block per accepted call form, since those overloads *are* the published signature. Unexposed/private helpers are typed at the contributor's discretion: annotate where it helps the next reader, and either way keep them clean. Members tagged `@internal` are stripped from the generated declarations, so use that tag rather than `@private` to keep something off the public surface.
Typing in JS files is encouraged: annotate with JSDoc `@type`/`@param`/`@returns` using the TypeScript type system, and define shared shapes with `@typedef`. API types must not be `unknown` or untyped `...args` — spell out the real parameter and return shapes; the only exception is values passed through transparently to an upstream layer that owns their type. For example:
@@ -132,7 +132,7 @@ Every SDK change carries all five of the following — a puter.js PR missing one
1. **Backward compatibility.** Mandatory unless a maintainer explicitly signs off on a break. Existing call signatures keep working (including both positional and options-object forms where a method supports them); new parameters are optional with defaults that preserve old behavior; never rename or repurpose existing params, response fields, or error codes. New parameter names are `camelCase` (existing `snake_case` stays for compatibility). Say in the PR how existing callers are unaffected.
2. **Tests.** Add or extend a suite in [tests/api/suites/](src/puter-js/tests/api/suites/) (`<area>.suite.ts`; register new suites in `suites/index.ts` — no globbing). One suite runs unchanged on node, browser, and workerd via `npm run test:puterjs`; never write a per-platform test. The runners execute the **built** bundle — run `npm run build:workerLib` after SDK changes or the suite silently tests stale code. For `puter.ui.*` methods rendered by the desktop, use the Playwright e2e harness instead — see [src/puter-js/TESTING.md](src/puter-js/TESTING.md).
3. **Docs.** New or changed APIs update [src/docs/src/](src/docs/src/): the method page (`<Area>/<method>.md`, with frontmatter and a runnable example) and the area overview when the surface changes. Docs are the contract users code against — signatures, defaults, and return shapes must match the implementation exactly.
4. **Types.** Update [src/puter-js/types/modules/](src/puter-js/types/modules/)`<module>.d.ts` and re-export new types through `index.d.ts` / `types/puter.d.ts`. Declarations must match runtime behavior exactly — a wrong type is worse than a missing one.
4. **Types.** Type the change in JSDoc on the implementation, then run `npm run check:puterjs:types` to confirm it still produces declarations that type-check. Name any new type in [src/puter-js/index.d.ts](src/puter-js/index.d.ts) if consumers should be able to import it. Declarations must match runtime behavior exactly — a wrong type is worse than a missing one.
5. **Error handling.** Reject/throw `{ message, code }` objects with stable `snake_case` codes, matching the existing modules (see `KV.js`). Validate cheap preconditions client-side before making the network call; pass backend errors through unchanged rather than swallowing or re-wrapping them. Error codes are API surface — changing one is a breaking change.
[doc/contributing-apis.md](doc/contributing-apis.md) walks the full lifecycle of adding an API across backend + SDK.
+73
View File
@@ -0,0 +1,73 @@
# Reverse proxy in front of Puter — mirrors what the prod ALB does:
# accepts every Host header, forwards to the Puter container, and lets
# the Puter app handle subdomain-based routing internally (api.*,
# site.*, app.*, dev.*, plus the per-user subdomains under those).
#
# To enable TLS:
# 1. Drop a wildcard fullchain.pem + privkey.pem into ./puter/tls/
# (see "Step 3 — TLS" in doc/self-hosting.md).
# 2. Uncomment the `:443` block at the bottom of this file, and swap the
# `:80` block for the redirect shown alongside it.
# 3. Uncomment the `443:443` port mapping under the `caddy` service in
# docker-compose.yml, and HTTPS_PORT in .env.
# 4. Set `"protocol": "https"` and `"pub_port": 443` in config.json.
{
# Certs are supplied by the operator, not issued by Caddy. Puter serves
# per-user sites and apps on dynamic subdomains (<name>.site.<domain>,
# <name>.app.<domain>) — only a DNS-01 wildcard cert covers those, and
# DNS-01 needs a provider plugin that isn't in the stock caddy image.
# `off` stops Caddy attempting ACME on boot (it would fail and leave the
# site unreachable) and stops it inventing its own http→https redirects.
auto_https off
}
# Shared handling, imported by the HTTP and HTTPS site blocks below so the
# two can't drift apart.
(puter_routes) {
# Rough size cap that mirrors prod ALB defaults; tune for your uploads.
# Puter chunks large uploads, so 1 GiB per request is plenty.
request_body {
max_size 1024MiB
}
# RustFS — see the `s3` service in docker-compose.yml. Browsers PUT/GET
# here for presigned-URL uploads / downloads. Matched on the `s3.`
# subdomain of whatever the install domain is, so signature verification
# works (Caddy preserves the original Host end-to-end) and HTTPS stays
# clean — no mixed content from a port-9000 host publish.
@s3 header_regexp Host ^s3\.
handle @s3 {
reverse_proxy s3:9000
}
# Everything else, on every Host, goes to Puter — which routes on that
# Host internally. Caddy forwards it unchanged and adds X-Forwarded-For
# / -Proto / -Host, which is what `trust_proxy` in config.json counts.
handle {
reverse_proxy puter:4100 {
# Stream responses through unbuffered — Puter uses SSE and
# socket.io. WebSocket upgrades need no config of their own;
# Caddy proxies them by default.
flush_interval -1
}
}
}
# ── HTTP (port 80) ─────────────────────────────────────────────────────
# A site address with no hostname is the catch-all: it answers for every
# Host, which is what Puter's subdomain routing requires.
:80 {
import puter_routes
}
# ── HTTPS (port 443) — uncomment after dropping certs in ./puter/tls/ ──
# Replace the `:80` block above with a redirect to force HTTPS everywhere:
#
# :80 {
# redir https://{host}{uri} permanent
# }
#
# :443 {
# tls /etc/caddy/tls/fullchain.pem /etc/caddy/tls/privkey.pem
# import puter_routes
# }
-2
View File
@@ -5,9 +5,7 @@
"protocol": "http",
"domain": "puter.localhost",
"cookie_name": "puter_auth_token",
"jwt_secret": "dev-jwt-secret-change-me",
"jwt_secret_v2": "dev-jwt-secret-v2-change-me",
"allow_v1_tokens": true,
"url_signature_secret": "dev-url-signature-secret-change-me",
"allow_all_host_values": true,
"allow_no_host_header": true,
+74 -6
View File
@@ -73,11 +73,7 @@
// ── Auth / session ──────────────────────────────────────────────────
// ALWAYS replace these for any public install — `openssl rand -hex 64`.
// `jwt_secret` is legacy verify-only; new tokens sign with `jwt_secret_v2`.
"jwt_secret": "change-me",
"jwt_secret_v2": "change-me",
// Set false to retire v1 tokens entirely (ROLLOUT-1).
"allow_v1_tokens": true,
"url_signature_secret": "change-me",
"cookie_name": "puter_auth_token",
"min_pass_length": 6,
@@ -255,6 +251,60 @@
}
},
// ── Alarms / alerting ───────────────────────────────────────────────
// Where system alarms go. Severity is the routing decision — each
// transport takes everything at or above its own `minSeverity`:
//
// critical — an unhandled server error; pages on-call.
// error — pages as well; prefer critical or warning.
// warning — look at it today; no page.
// info — a record in the chat channel only.
//
// Both transports are off unless enabled, so a self-hosted node just
// logs its alarms to the console.
"pager": {
// Severity for call sites that don't pick one. Default "critical".
"defaultSeverity": "critical",
// Retier or silence an alarm without a deploy. Keys are alarm ids,
// or a prefix ending in `*`; the exact id wins over a pattern, and
// the longest matching pattern wins among patterns. Values are a
// severity or "mute". This is applied last, so it overrides both the
// call site and any known-error rule.
"severityOverrides": {
// "cronMonitor:*": "info",
// "http_500:GET:/some/flapping/route:*": "mute"
},
"pagerduty": {
"enabled": false,
"routingKey": "",
// Lowest severity that reaches PagerDuty. Default "warning",
// which keeps `info` out of the paging system entirely.
"minSeverity": "warning"
},
// Slack incoming webhook — the low-noise destination for everything
// that shouldn't page.
"slack": {
"enabled": false,
"webhookUrl": "",
// Optional; defaults to the channel the webhook was created for.
"channel": "#alerts",
"username": "puter-alarms",
// Severity window posted to Slack. The ceiling defaults to
// "info" when PagerDuty is configured — what pages belongs in
// the pager, not in chat — and to "critical" when Slack is the
// only transport.
"minSeverity": "info",
"maxSeverity": "info",
// Don't repost the same alarm id within this window. The first
// occurrence always posts; the next post that gets through
// reports how many occurrences piled up. 0 disables throttling.
"repeatThrottleMs": 900000
}
},
// ── Rate limiting ───────────────────────────────────────────────────
// `memory` for single-node, `redis` for multi-node (default), `kv` for
// dynamo-backed counters.
@@ -387,6 +437,24 @@
],
// ── Metering ────────────────────────────────────────────────────────
// When true, all metering checks pass — no per-actor limits enforced.
"unlimitedMetering": false
// When true, every account resolves to an unlimited policy: usage is still
// recorded, but nothing is ever refused for lack of budget. This is the
// setting for a deployment with no way to buy more — without it, accounts
// are held to the free monthly allowance and start getting 402s from the AI
// surfaces, file transfers and KV once they pass it.
"unlimitedMetering": false,
// Whether an account that has spent its whole allowance is refused the
// operations that spend it — file transfers, KV calls. Recording is
// unaffected either way. `workers` extends the same refusal to
// worker-driven calls, which are exempt by default because a deployed
// worker has nowhere to surface a payment prompt.
// "meteringEnforcement": { "enabled": true, "workers": false },
// Fleet-wide spend rate, in micro-cents per minute, past which metering
// raises the `metering:excessiveGlobalUsageRate` alarm. Omit it (the
// default) to leave the check off: the only useful value is a multiple of
// what this deployment's normal traffic costs, so it has to be measured
// rather than guessed, and a stale number here alarms on healthy growth.
// "maxGlobalUsagePerMinute": 200000000
}
+119
View File
@@ -0,0 +1,119 @@
# Alarms
`clients.alarm` is the one way code reports that something is wrong. It
de-dupes by alarm id, counts occurrences, and routes each alarm to alert
transports by **severity**.
```ts
this.clients.alarm.create(
`metering_write_failed:${userUuid}`, // de-dupe key
'Metering write failed', // what a human reads
{ userUuid, appId, error }, // context fields
'info', // severity
);
```
## Severity is the routing decision
| Severity | Meaning | Goes to |
| ---------- | ------------------------------------------------------ | ----------- |
| `critical` | An unhandled server error. Someone gets woken up. | Pager |
| `error` | Same urgency as critical; prefer one of the other two. | Pager |
| `warning` | Worth a look today. Nobody is paged. | Pager (low) |
| `info` | A record of something expected-but-notable. | Chat |
Each transport declares the severity window it accepts, so the value a call
site passes is what decides where the alarm lands. The two windows don't
overlap by default: anything that pages lives in the paging system, and chat
is the record of what didn't. The bar for `critical` is
deliberately high: an unhandled 5xx out of the HTTP error handler is the main
thing that still pages. Anything a human can look at tomorrow is `warning`,
and anything that's just worth recording is `info`.
Omitting the severity takes `pager.defaultSeverity` (itself `critical`), so
pass one explicitly unless you really mean "page someone".
### Choosing one
- Did the server fail to do its job in a way nobody expected? → `critical`
- Is a background job, rate, or dependency degraded? → `warning`
- Is this a user doing something notable (tripping an abuse heuristic,
overspending)? → `info`
- Did one of *our own* limits reject a caller — a rate limit, a concurrency
cap, a quota? → don't alarm at all. The limit doing its job is not an
event; the 429 is the whole signal, and alarming on it only produces noise
proportional to traffic. An *upstream provider* rate-limiting us is the
opposite case and still alarms (`upstream_rate_limited`, `info`) — that one
is not something we chose.
An extension whose signals are all one tier can default its own local
`raiseAlarm` helper to that tier instead of repeating it at every call site —
see [extensions/cronMonitor](../../../extensions/cronMonitor/index.js).
## One incident per occurrence, unless you say otherwise
Alarms always de-dupe *in process* — repeats of an id bump its occurrence
count rather than creating a second alarm. What that means for the pager is a
separate decision, and by default every occurrence opens its own PagerDuty
incident: two failed scans an hour apart are two things that happened, and
closing one shouldn't hide the other.
Pass `{ dedup: true }` as a fifth argument when repeats of the id really are
one recurring fault, and they collapse onto a single incident carrying the
occurrence count:
```ts
this.clients.alarm.create(alarmId, message, fields, 'critical', {
dedup: true,
});
```
The HTTP error handler uses it: its id is route + error signature, so a hot
loop of the same crash is one incident with N occurrences instead of N pages.
Reach for it anywhere else only when the id is that specific — otherwise a
per-request alarm can flood the pager.
## Configuration
Everything lives under `pager` in config (see
[config.template.jsonc](../config.template.jsonc) for the annotated version).
Both transports are off unless enabled, so a self-hosted node just logs
alarms to the console.
```jsonc
"pager": {
"defaultSeverity": "critical",
"severityOverrides": { "cronMonitor:*": "info" },
"pagerduty": { "enabled": true, "routingKey": "…", "minSeverity": "warning" },
"slack": {
"enabled": true,
"webhookUrl": "…",
"channel": "#alerts",
"minSeverity": "info",
"maxSeverity": "info",
"repeatThrottleMs": 900000,
},
}
```
Slack's `maxSeverity` defaults to `info` whenever PagerDuty is configured, and
to `critical` when it isn't — a node with only a webhook still sees
everything. Raise it to have chat mirror the paging tiers as well.
### Retiering without a deploy
`severityOverrides` is the escape hatch for an alarm that turns out to be
noisier or more serious than its call site assumed. Keys are alarm ids or a
prefix ending in `*`; the exact id beats a pattern, and the longest matching
prefix wins among patterns. Values are a severity, or `mute` to drop the
alarm before any transport sees it.
It is applied *after* the call site's severity and any known-error rule, so
config always has the last word.
### Repeat throttling
The chat transport won't repost the same alarm id within
`repeatThrottleMs` (default 15 minutes). The first occurrence always posts,
and the next one that gets through reports how many piled up in between —
so a hot loop reads as one message with a count, not a wall of them.
+5 -1
View File
@@ -54,6 +54,7 @@ Both are supported ways to define an API. **Prefer a controller when you need fi
- **Controller routes** (and extension routes — same options) take [`RouteOptions`](../src/backend/core/http/types.ts): auth gates (`requireAuth`, `requireUserActor`, `noUserSession`, `adminOnly`, `allowedAppIds`, and the access-token controls), `subdomain` routing, per-route `rateLimit`, body parsers, and arbitrary extra `middleware`. The auth flavors are subtle and default-deny — read the JSDoc on each field before picking.
- **Driver methods** get their policies from the `@Driver` options: per-method `rateLimit` (limit/window/backend), `concurrent` in-flight caps (optionally `bySubscription`), and `noUserSession`. The `/drivers/call` surface enforces them.
- **An endpoint that spends metered resources** on the caller's behalf — moving file content, making object-store requests, anything else the account is billed for — also declares `requireCredits: true`, which turns an account with nothing left of its budget away with a 402 before the handler runs. Endpoints that only describe or delete things deliberately don't: an account that has run out still has to be able to see what it has, clear it, and reach its billing pages. Drivers have no route options to declare this on, so they call `assertActorHasCredits` themselves ([src/backend/services/metering/enforcement.ts](../src/backend/services/metering/enforcement.ts)) — see `KVStoreDriver`, which does it once for every method.
### 3. puter.js
@@ -62,7 +63,10 @@ Both are supported ways to define an API. **Prefer a controller when you need fi
### 4. Types
- Add or extend the declaration in [src/puter-js/types/modules/](../src/puter-js/types/modules/)`<module>.d.ts` and re-export new types through `index.d.ts`. Declarations must match the runtime exactly — optionality, defaults, and return types included.
- Type the method where you wrote it, in JSDoc: `@param`/`@returns` on the implementation, one `@overload` block per accepted call form, and `@typedef {Object}` + `@property` for any new shape. The JSDoc is the source of truth — declarations are generated from it, so there is nothing to keep in sync by hand.
- Put a shape more than one file needs in the module's `types.js` (e.g. [src/puter-js/src/modules/kv/types.js](../src/puter-js/src/modules/kv/types.js)); anything shared across modules goes in [src/puter-js/src/lib/types.js](../src/puter-js/src/lib/types.js). A shape with one consumer can stay next to it.
- Run `npm run check:puterjs:types` — it generates the declarations and type-checks the published surface without `skipLibCheck`. **Never edit anything under `src/puter-js/types/`**: it is gitignored build output, produced by the SDK build and shipped in the npm tarball, and the next build overwrites it.
- Name the new type in [src/puter-js/index.d.ts](../src/puter-js/index.d.ts) if consumers should be able to import it. That file is the one hand-written declaration in the package: it decides what is public and re-exports nothing else.
### 5. Docs
+52 -20
View File
@@ -1,6 +1,6 @@
# Self-Hosting Puter
`docker-compose.yml` brings up Puter **plus every external service it needs** — MariaDB, Valkey, DynamoDB-local, RustFS S3, nginx — wired together. Closest thing to a production deployment you can self-manage on a single host.
`docker-compose.yml` brings up Puter **plus every external service it needs** — MariaDB, Valkey, DynamoDB-local, RustFS S3, Caddy — wired together. Closest thing to a production deployment you can self-manage on a single host.
## One-line installer (recommended)
@@ -8,7 +8,7 @@
curl -fsSL https://raw.githubusercontent.com/HeyPuter/puter/main/install.sh | sh
```
Generates secrets, writes `.env` + `puter/config/config.json`, downloads `docker-compose.yml` from the OSS repo, and runs `docker compose up -d`. Re-running is safe — it won't overwrite existing config (set `PUTER_FORCE=1` to rotate). Use the manual steps below if you want to inspect or tweak each step yourself.
Generates secrets, writes `.env` + `puter/config/config.json`, downloads `docker-compose.yml` + `caddy/Caddyfile` from the OSS repo, and runs `docker compose up -d`. Re-running is safe — it won't overwrite existing config (set `PUTER_FORCE=1` to rotate). Use the manual steps below if you want to inspect or tweak each step yourself.
## Requirements
@@ -20,7 +20,7 @@ Generates secrets, writes `.env` + `puter/config/config.json`, downloads `docker
| Container | Image | Role |
| --------------- | ------------------------ | ---------------------------------------------------------- |
| `puter-nginx` | `nginx:1.27-alpine` | Reverse proxy on 80 (and 443 if TLS); forwards to Puter |
| `puter-caddy` | `caddy:2.11-alpine` | Reverse proxy on 80 (and 443 if TLS); forwards to Puter |
| `puter` | `ghcr.io/heyputer/puter` | The app |
| `puter-mariadb` | `mariadb:11` | SQL database — schema applied automatically on first boot |
| `puter-valkey` | `valkey/valkey:8-alpine` | Redis-compatible cache + rate-limiter |
@@ -47,7 +47,6 @@ State lives under `./puter/data/<service>/`.
MARIADB_ROOT_PASSWORD=$(openssl rand -hex 32)
MARIADB_PASSWORD=$(openssl rand -hex 32)
S3_SECRET_KEY=$(openssl rand -hex 32)
JWT_SECRET=$(openssl rand -hex 64)
JWT_SECRET_V2=$(openssl rand -hex 64)
URL_SIGNATURE_SECRET=$(openssl rand -hex 64)
@@ -65,7 +64,7 @@ S3_SECRET_KEY=$S3_SECRET_KEY
S3_BUCKET=puter-local
EOF
mkdir -p puter/config puter/data puter/tls
mkdir -p puter/config puter/data puter/data/caddy puter/tls
cat > puter/config/config.json <<EOF
{
"domain": "puter.localhost",
@@ -78,9 +77,7 @@ cat > puter/config/config.json <<EOF
"private_app_hosting_domain": "app.puter.localhost",
"private_app_hosting_domain_alt": "dev.puter.localhost",
"jwt_secret": "$JWT_SECRET",
"jwt_secret_v2": "$JWT_SECRET_V2",
"allow_v1_tokens": true,
"url_signature_secret": "$URL_SIGNATURE_SECRET",
"database": {
@@ -134,21 +131,21 @@ Replace `puter.localhost`, `site.puter.localhost`, `host.puter.localhost`, `dev.
Why these knobs:
- `jwt_secret` + `jwt_secret_v2` — Puter signs new auth tokens with `jwt_secret_v2` (v2 token format, `kid: 'v2'` JWT header). `jwt_secret` is verify-only and lets tokens minted before this version (cookies still in users' browsers, stored API tokens) keep working. `allow_v1_tokens: true` keeps that fallback enabled. Fresh installs can leave it as-is; future versions will flip the flag to `false` and retire `jwt_secret` entirely once legacy tokens have drained.
- `jwt_secret_v2` — the HMAC secret Puter signs and verifies auth tokens with (`kid: 'v2'` JWT header). The pre-v2 token format is retired: a token signed with the old `jwt_secret` no longer verifies, and holders are asked to sign in again. If you are upgrading from a release that had `jwt_secret`, drop it from your config — it is ignored.
- `env: "prod"` — the bundled `config.default.json` ships with `env: "dev"` (matches the source-tree `npm run start:gui` workflow, which expects webpack-dev-server emitting a CSS manifest). Self-host runs against pre-built static bundles, so `env: "prod"` makes the homepage emit the `/dist/bundle.min.css` `<link>` tag instead of waiting on a manifest that doesn't exist.
- `database.migrationPaths` — Puter applies the bundled MySQL/MariaDB schema on boot. The migration files are idempotent, so it is safe to leave this configured across restarts.
- `dynamo.bootstrapTables: true` — Puter creates its KV table on boot. **Only set against a local emulator**, never real AWS.
- `dynamo.aws` keys are dummies; DynamoDB-local doesn't validate them but the AWS SDK requires _something_. **Note:** DynamoDB uses `access_key` / `secret_key` (snake_case); S3 below uses `accessKeyId` / `secretAccessKey` (camelCase). Not interchangeable.
- `providers.ollama.enabled: false` — Puter auto-probes a local Ollama at `127.0.0.1:11434` by default; without one running you'd see `ECONNREFUSED` on every boot. To run a bundled Ollama, see [Optional: local LLM (Ollama)](#optional-local-llm-ollama) below.
- `s3.s3Config.forcePathStyle: true` — RustFS / MinIO / fauxqs need path-style URLs (`<endpoint>/<bucket>`). Real AWS S3 wants virtual-hosted (`<bucket>.<endpoint>`) — drop this flag (or set `false`) when you swap to real S3.
- `s3.s3Config.publicEndpoint` — `endpoint` (`http://s3:9000`) only resolves inside the docker network; presigned upload/download URLs handed to the browser need a host-reachable URL. nginx routes the `s3.<domain>` subdomain to RustFS internally and preserves the Host header end-to-end (required for S3 signature validation), so the browser hits the same port/protocol as the rest of the app — no separate published port, no mixed-content surprises when you turn on TLS. Switch to `https://s3.<your-domain>` once you enable TLS in Step 3. Real AWS S3 doesn't need this — its endpoint is already public; drop the field entirely.
- `trust_proxy: 1` — nginx terminates TLS and forwards `X-Forwarded-For`. Without this, `req.ip` is the docker-network address of the nginx container instead of the real client IP, which breaks rate limiting and IP-based audit logs. `1` = one trusted hop (nginx). Bump to `2` if you put Cloudflare in front of nginx; never set `true` (it trusts every hop and makes XFF forgeable).
- `s3.s3Config.publicEndpoint` — `endpoint` (`http://s3:9000`) only resolves inside the docker network; presigned upload/download URLs handed to the browser need a host-reachable URL. Caddy routes the `s3.<domain>` subdomain to RustFS internally and preserves the Host header end-to-end (required for S3 signature validation), so the browser hits the same port/protocol as the rest of the app — no separate published port, no mixed-content surprises when you turn on TLS. Switch to `https://s3.<your-domain>` once you enable TLS in Step 3. Real AWS S3 doesn't need this — its endpoint is already public; drop the field entirely.
- `trust_proxy: 1` — Caddy terminates TLS and forwards `X-Forwarded-For`. Without this, `req.ip` is the docker-network address of the Caddy container instead of the real client IP, which breaks rate limiting and IP-based audit logs. `1` = one trusted hop (Caddy). Bump to `2` if you put Cloudflare in front of Caddy; never set `true` (it trusts every hop and makes XFF forgeable).
> If you ever change `MARIADB_PASSWORD` after first boot, `.env` alone won't update MariaDB — its credentials are baked into `./puter/data/mariadb/` on first init. Either rotate the password inside MariaDB by hand or `docker compose down && rm -rf ./puter/data/mariadb` to start fresh.
## Step 2 — Point DNS at the server \[Optional\]
In your DNS provider, add records for the main domain plus the subdomains Puter and nginx route on (`api.*`, `site.*`, `app.*`, `s3.*`):
In your DNS provider, add records for the main domain plus the subdomains Puter and Caddy route on (`api.*`, `site.*`, `app.*`, `s3.*`):
```
A puter.localhost → <your server's public IP>
@@ -163,7 +160,7 @@ A dev.puter.localhost → <your server's public IP>
A *.dev.puter.localhost → <your server's public IP>
```
The wildcards are required — Puter routes via subdomains (`api.*`, `app.*`, etc.) and nginx routes browser S3 traffic via `s3.*` to RustFS.
The wildcards are required — Puter routes via subdomains (`api.*`, `app.*`, etc.) and Caddy routes browser S3 traffic via `s3.*` to RustFS.
## Step 3 — TLS (recommended for public installs) \[Optional\]
@@ -182,13 +179,15 @@ sudo certbot certonly --manual --preferred-challenges dns \
The cert needs to cover `*.puter.localhost` so that `s3.puter.localhost` (browser S3 endpoint), plus Puter's own `api.*` / `app.*` subdomains, all validate.
> **Why not Caddy's automatic HTTPS?** Caddy issues certs over ACME by itself, but only for hostnames it knows up front, and only over HTTP-01 with the stock image. Puter serves every user's site and app on a subdomain it invents at runtime (`<name>.site.<domain>`, `<name>.app.<domain>`), which needs a **wildcard** cert — and wildcards require DNS-01, which requires a DNS-provider plugin that isn't in `caddy:2.11-alpine`. So `caddy/Caddyfile` sets `auto_https off` and reads the cert you supply below. If you'd rather have Caddy manage certs, build an image with the plugin for your DNS provider (`xcaddy build --with github.com/caddy-dns/<provider>`) and swap the `tls` line for a `tls { dns <provider> … }` block.
Drop the resulting `fullchain.pem` and `privkey.pem` into `./puter/tls/`.
**Wire nginx to use them:**
**Wire Caddy to use them:**
1. Open [nginx/nginx.conf](../nginx/nginx.conf), uncomment **both** `# server { listen 443 ssl … }` blocks (one for `s3.*`, one for the catch-all).
2. (Optional) Replace the body of the port-80 blocks with `return 301 https://$host$request_uri;` to force HTTPS everywhere.
3. In [docker-compose.yml](../docker-compose.yml), uncomment the `443:443` port mapping under the `nginx` service.
1. Open [caddy/Caddyfile](../caddy/Caddyfile) and uncomment the `# :443 { … }` block at the bottom.
2. (Optional but recommended) Replace the plain `:80 { import puter_routes }` block with the `redir` version shown alongside it, to force HTTPS everywhere.
3. In [docker-compose.yml](../docker-compose.yml), uncomment the `443:443` port mapping under the `caddy` service.
4. In `.env`, uncomment `HTTPS_PORT=443`.
5. In `config.json`, switch:
```json
@@ -201,9 +200,9 @@ Drop the resulting `fullchain.pem` and `privkey.pem` into `./puter/tls/`.
## Running behind your own reverse proxy
The bundled `puter-nginx` mirrors production and is the supported default — it terminates TLS (Step 3) and forwards every Host to Puter. But plenty of self-hosters already run their own edge proxy (Caddy, Traefik, HAProxy, a cloud load balancer, another nginx). You can put Puter behind it instead; Puter doesn't terminate TLS itself in any setup, so "your proxy does TLS" is just a matter of pointing it at the Puter container and getting a handful of details right.
The bundled `puter-caddy` mirrors production and is the supported default — it terminates TLS (Step 3) and forwards every Host to Puter. But plenty of self-hosters already run their own edge proxy (Traefik, nginx, HAProxy, a cloud load balancer, another Caddy). You can put Puter behind it instead; Puter doesn't terminate TLS itself in any setup, so "your proxy does TLS" is just a matter of pointing it at the Puter container and getting a handful of details right.
You can keep the bundled nginx as the single hop your proxy talks to, or bypass it and forward straight to the Puter container on port `4100` (uncomment the `4100:4100` mapping under the `puter` service in [docker-compose.yml](../docker-compose.yml), or attach your proxy to the compose network). Either works — what matters is the rules below.
You can keep the bundled Caddy as the single hop your proxy talks to, or bypass it and forward straight to the Puter container on port `4100` (uncomment the `4100:4100` mapping under the `puter` service in [docker-compose.yml](../docker-compose.yml), or attach your proxy to the compose network). Either works — what matters is the rules below.
**The rules that actually matter** (getting any of these wrong is what causes the redirect loops and "Invalid Host header" failures people hit):
@@ -219,7 +218,7 @@ You can keep the bundled nginx as the single hop your proxy talks to, or bypass
- `X-Forwarded-For` — the real client IP (rate limiting + audit logs).
- `Upgrade` / `Connection` — passed through for WebSocket / socket.io upgrades, or the realtime connection silently fails.
5. **Set `trust_proxy` to the number of hops in front of Puter.** One external proxy talking directly to Puter → `"trust_proxy": 1` (installer: `PUTER_TRUST_PROXY=1`). A second proxy in front (e.g. Cloudflare → your proxy → Puter) → `2`. This is the count of proxies between the client and Puter, **including** the bundled nginx if you keep it. Too low and `req.ip` becomes a proxy address (breaks rate limiting); never set `true` (it trusts every hop and makes `X-Forwarded-For` forgeable).
5. **Set `trust_proxy` to the number of hops in front of Puter.** One external proxy talking directly to Puter → `"trust_proxy": 1` (installer: `PUTER_TRUST_PROXY=1`). A second proxy in front (e.g. Cloudflare → your proxy → Puter) → `2`. This is the count of proxies between the client and Puter, **including** the bundled Caddy if you keep it. Too low and `req.ip` becomes a proxy address (breaks rate limiting); never set `true` (it trusts every hop and makes `X-Forwarded-For` forgeable).
6. **Route the wildcard to Puter.** Your proxy must forward `*.<domain>` (covering `api.*`, `app.*`, and `s3.<domain>`) and `*.site.<domain>` (and any other hosting domains) to Puter — same wildcard DNS as Step 2. Puter does the per-subdomain routing internally; the proxy just needs to hand it the traffic with the Host intact.
@@ -227,6 +226,8 @@ With those in place the rest of the stack is unchanged — `docker compose up -d
## Step 4 — Bring it up
Both [docker-compose.yml](../docker-compose.yml) and [caddy/Caddyfile](../caddy/Caddyfile) need to sit next to your `.env` — clone the repo, or copy those two paths out of it. The Caddyfile is bind-mounted read-only; if it's missing, Docker creates a *directory* at that path and `puter-caddy` dies with "not a directory".
```bash
docker compose up -d
```
@@ -369,6 +370,37 @@ Default is 100 MB per user.
Set `is_storage_limited: false` for unlimited (bounded by host disk).
### Usage metering and budgets
Puter meters what an account costs to serve — bytes sent back, object-store
requests, KV capacity, AI tokens — against a monthly budget, and refuses the
operations that spend it once that budget is gone. The refusal is a `402` with
code `insufficient_funds`; reads that only describe things, and every kind of
deletion, stay available so an account can always see what it has and clear it.
On a self-hosted install this is almost certainly not what you want. There is
nowhere to buy more, so accounts are held to the free monthly allowance
(US$0.25 of measured cost — roughly 2 GiB of downloads) and start being refused
after that. Turn it off:
```json
"unlimitedMetering": true
```
Every account then resolves to an unlimited policy. Usage is still recorded, so
the dashboard still shows what is being consumed; nothing is ever refused for
lack of budget.
To keep the budgets but stop them blocking anything — recording only:
```json
"meteringEnforcement": { "enabled": false }
```
Calls driven by a deployed worker are exempt from enforcement by default,
because a worker has no prompt to show and nobody watching it fail. Set
`"meteringEnforcement": { "workers": true }` to include them.
### Captcha on signup / login
Built-in proof-of-work captcha — no external service needed.
@@ -469,7 +501,7 @@ Migrations re-apply idempotently across pulls. Volumes are preserved.
## Troubleshooting
**Site loads but I get "Bad Gateway" / nginx errors.**
**Site loads but I get a 502 / "Bad Gateway" from Caddy.**
The puter container failed to come up. `docker compose logs puter` will tell you which dependency rejected it (most often DB password mismatch between `.env` and `config.json`).
**Login screen says "admin password not set".**
+22 -12
View File
@@ -2,7 +2,7 @@
# Self-hosted Puter — full stack.
#
# Brings up Puter + every external service it needs:
# - nginx : reverse proxy (mirrors prod ALB; handles TLS + Host fan-out)
# - caddy : reverse proxy (mirrors prod ALB; handles TLS + Host fan-out)
# - valkey : redis-compatible cache / rate-limiter backend
# - mariadb : SQL database (Puter applies its schema on first boot)
# - dynamo : DynamoDB-local (KV store; Puter creates the table itself)
@@ -23,7 +23,8 @@
#
# Production:
# - Always replace the default passwords / S3 keys / Puter secrets.
# - Front Puter with TLS-terminating reverse proxy (Caddy / nginx).
# - Enable TLS on the bundled Caddy (see caddy/Caddyfile) or front the
# stack with your own TLS-terminating proxy.
# - Move state-bearing volumes to a backed-up location.
services:
@@ -118,7 +119,7 @@ services:
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY:-puter-secret-change-me}
volumes:
- ./puter/data/s3:/data:z
# Internal-only — browsers reach RustFS via nginx (`s3.<domain>`),
# Internal-only — browsers reach RustFS via Caddy (`s3.<domain>`),
# which preserves the Host header for S3 signature validation and
# rides the same TLS termination as Puter. Uncomment to also expose
# 9000 directly on the host for `aws-cli` / debugging.
@@ -249,7 +250,7 @@ services:
condition: service_started
s3-init:
condition: service_completed_successfully
# Internal-only: nginx reaches it on the compose network. Uncomment
# Internal-only: Caddy reaches it on the compose network. Uncomment
# to also expose port 4100 directly on the host (useful for debugging).
# ports:
# - "4100:4100"
@@ -270,23 +271,32 @@ services:
retries: 3
start_period: 30s
nginx:
image: nginx:1.27-alpine
container_name: puter-nginx
caddy:
image: caddy:2.11-alpine
container_name: puter-caddy
restart: unless-stopped
depends_on:
puter:
condition: service_started
ports:
- "${HTTP_PORT:-80}:80"
# Uncomment when you enable TLS in nginx/nginx.conf:
# Uncomment when you enable TLS in caddy/Caddyfile:
# - "${HTTPS_PORT:-443}:443"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro,z
# TLS certs (fullchain.pem + privkey.pem). Read-only inside.
- ./puter/tls:/etc/nginx/tls:ro,z
- ./caddy/Caddyfile:/etc/caddy/Caddyfile:ro,z
# TLS certs (wildcard fullchain.pem + privkey.pem). Read-only inside.
- ./puter/tls:/etc/caddy/tls:ro,z
# Caddy's own state — the cert store it would use if you swap this
# config over to ACME. Kept under ./puter/data with everything else
# so one directory is the whole backup.
- ./puter/data/caddy:/data:z
healthcheck:
test: ["CMD-SHELL", "wget -qO- --tries=1 --timeout=2 http://localhost/ || exit 1"]
# Hits Caddy's local-only admin API rather than proxying through to
# Puter: this reports whether the proxy itself is up and configured,
# and keeps working once the `:80` block becomes an HTTPS redirect.
# 127.0.0.1, not localhost — the admin endpoint is IPv4-only and
# busybox wget tries ::1 first.
test: ["CMD-SHELL", "wget -qO- --tries=1 --timeout=2 http://127.0.0.1:2019/config/ >/dev/null || exit 1"]
interval: 10s
timeout: 3s
retries: 5
+37 -4
View File
@@ -2,6 +2,14 @@ import { Context } from '@heyputer/backend/src/core';
import type { Actor } from '@heyputer/backend/src/core/actor';
import { HttpError } from '@heyputer/backend/src/core/http';
import { PuterDriver } from '@heyputer/backend/src/drivers/types';
import type {
DriverConcurrentConfig,
DriverRateLimitConfig,
} from '@heyputer/backend/src/drivers/meta';
import {
DEFAULT_FREE_SUBSCRIPTION,
DEFAULT_TEMP_SUBSCRIPTION,
} from '@heyputer/backend/src/services/metering/consts';
import { extension } from '@heyputer/backend/src/extensions';
// App-telemetry lets an app owner enumerate the users who have
@@ -50,17 +58,42 @@ const parseIntParam = (
* Driver exposing the `app-telemetry` interface.
*
* The `/drivers/call` permission gate checks
* `service:app-telemetry:ii:app-telemetry`, which every actor already holds
* via the blanket `service` grant (hardcoded-permissions.js +
* `service:app-telemetry:ii:app-telemetry`, which every actor already holds via
* the blanket `service` grant (hardcoded-permissions.js +
* `default_implicit_user_app_permissions`). The real authorization — "is the
* caller the app owner?" — is enforced inside `get_users` below, exactly as
* v1 did.
* caller the app owner?" — is enforced inside `get_users` below, exactly as v1
* did.
*/
export class AppTelemetryDriver extends PuterDriver {
readonly driverInterface = 'app-telemetry';
readonly driverName = 'app-telemetry';
readonly isDefault = true;
// Declaring nothing here would leave both methods on the generic
// 600/minute driver default, which does not fit a paginated scan that
// can ask for MAX_LIMIT rows at MAX_OFFSET. This is a dashboard read —
// nobody calls it in a loop.
readonly rateLimit: DriverRateLimitConfig = {
default: {
limit: 60,
window: 60_000,
bySubscription: {
[DEFAULT_FREE_SUBSCRIPTION]: 30,
[DEFAULT_TEMP_SUBSCRIPTION]: 10,
},
},
};
readonly concurrent: DriverConcurrentConfig = {
default: {
limit: 5,
bySubscription: {
[DEFAULT_FREE_SUBSCRIPTION]: 2,
[DEFAULT_TEMP_SUBSCRIPTION]: 2,
},
},
};
/** Users who have authenticated into the given app (owner-only). */
async get_users({
app_uuid,
+555
View File
@@ -0,0 +1,555 @@
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import path from 'node:path';
import {
afterAll,
afterEach,
beforeAll,
beforeEach,
describe,
expect,
it,
vi,
} from 'vitest';
import { extensionStore } from '../src/backend/extensions.ts';
import type { IConfig } from '../src/backend/types';
import './devWatcher.ts';
type Lifecycle = {
onServerStart: () => Promise<void>;
onServerShutdown: () => Promise<void>;
};
// The extension registers itself on import; that registry entry is the
// only handle on the service class.
const DevWatcherService = extensionStore.services.devWatcher as unknown as new (
config: IConfig,
clients: unknown,
stores: unknown,
services: unknown,
) => Lifecycle;
const makeService = (config: Record<string, unknown>): Lifecycle =>
new DevWatcherService(config as unknown as IConfig, {}, {}, {});
let workdir: string;
const write = (relative: string, contents: string): string => {
const abs = path.join(workdir, relative);
writeFileSync(abs, contents);
return abs;
};
/** Poll until `predicate` holds or the budget runs out. */
const waitFor = async (
predicate: () => boolean,
label: string,
timeoutMs = 15_000,
): Promise<void> => {
const start = Date.now();
while (Date.now() - start < timeoutMs) {
if (predicate()) return;
await new Promise((r) => setTimeout(r, 20));
}
throw new Error(`timed out waiting for ${label}`);
};
const logLines = (spy: { mock: { calls: unknown[][] } }): string[] =>
spy.mock.calls.map((call) => call.map(String).join(' '));
describe('devWatcher extension', () => {
let log: ReturnType<typeof vi.spyOn>;
let warn: ReturnType<typeof vi.spyOn>;
let error: ReturnType<typeof vi.spyOn>;
beforeAll(() => {
workdir = mkdtempSync(path.join(tmpdir(), 'devwatch-test-'));
});
afterAll(() => {
rmSync(workdir, { recursive: true, force: true });
delete (extensionStore.services as Record<string, unknown>).devWatcher;
});
beforeEach(() => {
log = vi.spyOn(console, 'log').mockImplementation(() => {});
warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
error = vi.spyOn(console, 'error').mockImplementation(() => {});
});
afterEach(() => {
log.mockRestore();
warn.mockRestore();
error.mockRestore();
});
const started = () =>
logLines(log).some((line) =>
line.startsWith('[devwatch] starting watchers from'),
);
// -- start gating -------------------------------------------------
it('does nothing when the server is not running in dev', async () => {
await makeService({ env: 'production' }).onServerStart();
expect(started()).toBe(false);
});
it('does nothing when devwatch is explicitly disabled', async () => {
await makeService({
env: 'dev',
devwatch: { enabled: false },
}).onServerStart();
expect(started()).toBe(false);
});
it('does nothing when the server opts out with no_devwatch', async () => {
await makeService({
env: 'dev',
no_devwatch: true,
devwatch: { enabled: true },
}).onServerStart();
expect(started()).toBe(false);
});
it('starts in dev even with no explicit devwatch block', async () => {
const service = makeService({
env: 'dev',
devwatch: {
root: workdir,
commands: [],
webpack: [],
ready_delay_ms: 0,
},
});
await service.onServerStart();
expect(started()).toBe(true);
await service.onServerShutdown();
});
it('starts outside dev when devwatch is explicitly enabled', async () => {
const service = makeService({
env: 'production',
devwatch: {
enabled: true,
root: workdir,
commands: [],
webpack: [],
ready_delay_ms: 0,
},
});
await service.onServerStart();
expect(started()).toBe(true);
await service.onServerShutdown();
});
it('ignores a devwatch config that is not an object', async () => {
// Falls back to `{}`, so the dev-env rule decides — and the
// default webpack entries would be used, so pin it off instead.
await makeService({
env: 'production',
devwatch: 'yes',
}).onServerStart();
expect(started()).toBe(false);
});
it('only starts once even if the lifecycle hook fires again', async () => {
const service = makeService({
env: 'dev',
devwatch: {
root: workdir,
commands: [],
webpack: [],
ready_delay_ms: 0,
},
});
await service.onServerStart();
await service.onServerStart();
expect(
logLines(log).filter((line) =>
line.startsWith('[devwatch] starting watchers from'),
),
).toHaveLength(1);
await service.onServerShutdown();
});
it('waits out the configured ready delay before resolving', async () => {
const service = makeService({
env: 'dev',
devwatch: {
root: workdir,
commands: [],
webpack: [],
ready_delay_ms: 60,
},
});
const start = Date.now();
await service.onServerStart();
expect(Date.now() - start).toBeGreaterThanOrEqual(50);
await service.onServerShutdown();
});
// -- child commands -----------------------------------------------
it('spawns a command, line-buffers its output and flushes the tail', async () => {
write(
'chatty.js',
[
"process.stdout.write('one\\ntwo\\n');",
"process.stderr.write('bad thing\\n');",
// No trailing newline: only the stream `end` flushes it.
"process.stdout.write('tail-no-newline');",
].join('\n'),
);
const service = makeService({
env: 'dev',
devwatch: {
root: workdir,
commands: [
{
name: 'chatty',
directory: '.',
command: 'node',
args: ['chatty.js'],
},
],
webpack: [],
ready_delay_ms: 0,
},
});
await service.onServerStart();
await waitFor(
() =>
logLines(log).some((l) =>
l.includes('[devwatch:chatty:1] tail-no-newline'),
),
'child stdout',
);
const lines = logLines(log);
expect(lines).toContain('[devwatch:chatty:1] one');
expect(lines).toContain('[devwatch:chatty:1] two');
expect(logLines(warn)).toContain('[devwatch:chatty:2] bad thing');
await waitFor(
() =>
logLines(log).some((l) =>
l.includes('[devwatch:chatty:exit] process exited'),
),
'child exit',
);
await service.onServerShutdown();
});
it('passes literal and computed env values to the child', async () => {
write(
'env.js',
'process.stdout.write(`STATIC=${process.env.STATIC_VALUE} ORIGIN=${process.env.FROM_CONFIG} MISSING=${process.env.BLOWS_UP}\\n`);',
);
const service = makeService({
env: 'dev',
origin: 'http://puter.localhost:4100',
devwatch: {
root: workdir,
commands: [
{
name: 'envtest',
directory: '.',
command: 'node',
args: ['env.js'],
env: {
STATIC_VALUE: 'literal',
FROM_CONFIG: ({
global_config,
}: {
global_config: Record<string, unknown> | null;
}) => String(global_config?.origin ?? ''),
// Reading through a null is the "config not
// loaded yet" shape the extension deliberately
// stays quiet about.
BLOWS_UP: () => {
const nothing = null as unknown as {
x: string;
};
return nothing.x;
},
NOISY: () => {
throw new Error('unexpected failure');
},
},
},
],
webpack: [],
ready_delay_ms: 0,
},
});
await service.onServerStart();
await waitFor(
() => logLines(log).some((l) => l.includes('STATIC=literal')),
'env output',
);
const line = logLines(log).find((l) => l.includes('STATIC=literal'))!;
expect(line).toContain('ORIGIN=http://puter.localhost:4100');
expect(line).toContain('MISSING=undefined');
// A null-property read is expected noise and stays silent; any
// other failure is reported.
const warnings = logLines(warn);
expect(
warnings.some((w) => w.includes('could not evaluate env function')),
).toBe(true);
expect(warnings.some((w) => w.includes('for BLOWS_UP'))).toBe(false);
expect(warnings.some((w) => w.includes('for NOISY'))).toBe(true);
await service.onServerShutdown();
});
it('kills a still-running child on shutdown', async () => {
write('forever.js', 'setInterval(() => {}, 1000);');
const service = makeService({
env: 'dev',
devwatch: {
root: workdir,
commands: [
{
name: 'forever',
directory: '.',
command: 'node',
args: ['forever.js'],
},
],
webpack: [],
ready_delay_ms: 0,
},
});
await service.onServerStart();
await service.onServerShutdown();
expect(logLines(log)).toContain('[devwatch:forever] stopping process');
await waitFor(
() =>
logLines(log).some((l) =>
l.includes('[devwatch:forever:exit] process exited'),
),
'child killed',
);
});
// -- webpack watchers ---------------------------------------------
const makeWebpackProject = (
name: string,
configFile: string,
contents: string,
packageJson?: string,
): string => {
const dir = path.join(workdir, name);
rmSync(dir, { recursive: true, force: true });
writeFileSync(
path.join(
(() => {
const { mkdirSync } =
require('node:fs') as typeof import('node:fs');
mkdirSync(path.join(dir, 'src'), { recursive: true });
return dir;
})(),
'src/entry.js',
),
"console.log('hello');\n",
);
writeFileSync(path.join(dir, configFile), contents);
if (packageJson) {
writeFileSync(path.join(dir, 'package.json'), packageJson);
}
return name;
};
it('compiles a CommonJS webpack config and reports later rebuilds', async () => {
const dir = makeWebpackProject(
'cjs-project',
'webpack.config.cjs',
`module.exports = {
mode: 'development',
entry: './src/entry.js',
output: { path: __dirname + '/out', filename: 'bundle.js' },
};`,
);
let onConfigSaw: Record<string, unknown> | undefined;
const service = makeService({
env: 'dev',
devwatch: {
root: workdir,
commands: [],
webpack: [
{
name: 'cjs',
directory: dir,
onConfig: (cfg: Record<string, unknown>) => {
onConfigSaw = cfg;
},
},
],
ready_delay_ms: 0,
},
});
await service.onServerStart();
expect(onConfigSaw).toBeDefined();
// The watcher context is anchored at <root>/<directory> so relative
// entries resolve regardless of the server's cwd.
expect(onConfigSaw!.context).toBe(path.join(workdir, dir));
// First build is silent by design; force a rebuild and assert the
// update line shows up.
await new Promise((r) => setTimeout(r, 500));
writeFileSync(
path.join(workdir, dir, 'src/entry.js'),
"console.log('hello again');\n",
);
await waitFor(
() =>
logLines(log).some((l) =>
l.includes('[devwatch] updated cjs using Webpack'),
),
'webpack rebuild',
);
await service.onServerShutdown();
});
it('resolves an ESM webpack config declared through package.json type', async () => {
const dir = makeWebpackProject(
'esm-project',
'webpack.config.js',
`export default {
mode: 'development',
entry: './src/entry.js',
output: { filename: 'bundle.js' },
};`,
JSON.stringify({ name: 'esm-project', type: 'module' }),
);
const service = makeService({
env: 'dev',
devwatch: {
root: workdir,
commands: [],
webpack: [{ directory: dir }],
ready_delay_ms: 0,
},
});
await service.onServerStart();
await service.onServerShutdown();
// Falling back to the directory as the display name is the
// documented behaviour when `name` is omitted.
expect(error).not.toHaveBeenCalled();
});
it('calls a config exported as a function and honours an explicit context', async () => {
const dir = makeWebpackProject(
'fn-project',
'webpack.config.cjs',
`module.exports = () => ({
mode: 'development',
context: 'src',
entry: './entry.js',
name: process.env.WEBPACK_MARKER,
output: { path: __dirname + '/out', filename: 'bundle.js' },
});`,
);
let seen: Record<string, unknown> | undefined;
const service = makeService({
env: 'dev',
devwatch: {
root: workdir,
commands: [],
webpack: [
{
name: 'fn',
directory: dir,
env: { WEBPACK_MARKER: 'set-during-load' },
onConfig: (cfg: Record<string, unknown>) => {
seen = cfg;
},
},
],
ready_delay_ms: 0,
},
});
await service.onServerStart();
// Relative `context` resolves against <root>/<directory>.
expect(seen!.context).toBe(path.join(workdir, dir, 'src'));
// The entry's env map is applied while the config module runs, and
// restored afterwards.
expect(seen!.name).toBe('set-during-load');
expect(process.env.WEBPACK_MARKER).toBeUndefined();
await service.onServerShutdown();
});
it('reports a failing compilation instead of crashing the server', async () => {
const dir = makeWebpackProject(
'broken-project',
'webpack.config.cjs',
`module.exports = {
mode: 'development',
entry: './src/does-not-exist.js',
output: { path: __dirname + '/out', filename: 'bundle.js' },
};`,
);
const service = makeService({
env: 'dev',
devwatch: {
root: workdir,
commands: [],
webpack: [{ name: 'broken', directory: dir }],
ready_delay_ms: 0,
},
});
await service.onServerStart();
await waitFor(
() =>
logLines(error).some((l) =>
l.includes('[devwatch] failed to update broken'),
),
'webpack failure',
);
expect(
logLines(error).some((l) =>
l.includes('[devwatch] error information: broken'),
),
).toBe(true);
await service.onServerShutdown();
});
it('fails loudly when a directory has no webpack config at all', async () => {
const dir = path.join('no-config');
const { mkdirSync } = await import('node:fs');
mkdirSync(path.join(workdir, dir), { recursive: true });
const service = makeService({
env: 'dev',
devwatch: {
root: workdir,
commands: [],
webpack: [{ name: 'missing', directory: dir }],
ready_delay_ms: 0,
},
});
await expect(service.onServerStart()).rejects.toThrow(
'could not find webpack config for: no-config',
);
await service.onServerShutdown();
});
});
+11 -1
View File
@@ -81,6 +81,16 @@ export const handleInstalledApps = async (
extension.get(
'/installedApps',
{ subdomain: 'api', requireUserActor: true, allowFullAccessToken: true },
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: {
scope: 'installed-apps',
limit: 120,
window: 60_000,
key: 'user',
},
},
handleInstalledApps,
);
+29 -5
View File
@@ -3,6 +3,7 @@ import { HttpError } from '@heyputer/backend/src/core/http';
import {
controllersContainers,
driversContainers,
servicesContainers,
} from '@heyputer/backend/src/exports';
import { extension } from '@heyputer/backend/src/extensions';
import type { Request, Response } from 'express';
@@ -18,7 +19,7 @@ function collectAllCosts(): Record<string, unknown>[] {
const all: Record<string, unknown>[] = [];
const collect = (
source: Record<string, unknown>,
kind: 'driver' | 'controller',
kind: 'driver' | 'controller' | 'service',
) => {
for (const [name, instance] of Object.entries(source)) {
const fn = (
@@ -43,6 +44,9 @@ function collectAllCosts(): Record<string, unknown>[] {
};
collect(driversContainers as Record<string, unknown>, 'driver');
collect(controllersContainers as Record<string, unknown>, 'controller');
// Services report the costs that aren't tied to one endpoint — egress,
// which is metered for every response there is.
collect(servicesContainers as Record<string, unknown>, 'service');
return all;
}
@@ -110,26 +114,46 @@ export const handleMeteringAllCosts = async (
res.json({ costs: cachedAllCosts });
};
/** Dashboard reads over the per-actor KV aggregates. */
const USAGE_READ_LIMIT = {
scope: 'metering-usage',
limit: 120,
window: 60_000,
key: 'user' as const,
};
extension.get(
'/metering/usage',
{ subdomain: 'api', requireAuth: true },
{ subdomain: 'api', requireAuth: true, rateLimit: USAGE_READ_LIMIT },
handleMeteringUsage,
);
extension.get(
'/metering/usage/:appIdOrName',
{ subdomain: 'api', requireAuth: true },
{ subdomain: 'api', requireAuth: true, rateLimit: USAGE_READ_LIMIT },
handleMeteringUsageForApp,
);
extension.get(
'/metering/globalUsage',
{ subdomain: 'api', adminOnly: true },
{
subdomain: 'api',
adminOnly: true,
// Sums across every shard of the global aggregate. Admin-gated, so
// this is loop protection — but one accidental poll is an
// expensive minute.
rateLimit: {
scope: 'metering-global-usage',
limit: 10,
window: 60_000,
key: 'user',
},
},
handleMeteringGlobalUsage,
);
extension.get(
'/metering/allCosts',
{ subdomain: 'api', requireAuth: true },
{ subdomain: 'api', requireAuth: true, rateLimit: USAGE_READ_LIMIT },
handleMeteringAllCosts,
);
+15 -3
View File
@@ -71,9 +71,7 @@ describe('serverInfo extension — handleServerInfo', () => {
const statfsSpy = vi
.spyOn(fs.default, 'statfs')
.mockRejectedValue(new Error('statfs unsupported'));
const errSpy = vi
.spyOn(console, 'error')
.mockImplementation(() => {});
const errSpy = vi.spyOn(console, 'error').mockImplementation(() => {});
const { res, captured } = makeRes();
await handleServerInfo({} as Request, res);
@@ -88,4 +86,18 @@ describe('serverInfo extension — handleServerInfo', () => {
statfsSpy.mockRestore();
errSpy.mockRestore();
});
it('reports an unknown CPU model when os.cpus() comes back empty', async () => {
const os = await import('node:os');
const cpusSpy = vi.spyOn(os.default, 'cpus').mockReturnValue([]);
const { res, captured } = makeRes();
await handleServerInfo({} as Request, res);
const cpu = captured.body!.cpu as Record<string, unknown>;
expect(cpu.model).toBe('Unknown');
expect(cpu.cores).toBe(0);
cpusSpy.mockRestore();
});
});
+193 -5
View File
@@ -3,6 +3,7 @@ import {
PutObjectCommand,
type S3Client,
} from '@aws-sdk/client-s3';
import crypto from 'node:crypto';
import {
afterAll,
beforeAll,
@@ -14,6 +15,7 @@ import {
import { PuterServer } from '../src/backend/server.ts';
import { setupTestServer } from '../src/backend/testUtil.ts';
import {
handleFsCopyNodeThumbnail,
handleFsRemoveNodeThumbnail,
handleThumbnailCreated,
handleThumbnailRead,
@@ -26,6 +28,9 @@ const TINY_PNG_BASE64 =
const BUCKET = 'puter-local';
// Keys the extension will accept back: its own `thumbnails/<uuid>` namespace.
const mintedKey = () => `thumbnails/${crypto.randomUUID()}`;
const streamToBuffer = async (
body: { transformToByteArray: () => Promise<Uint8Array> } | undefined,
): Promise<Buffer> => {
@@ -195,7 +200,7 @@ describe('thumbnails extension — handleThumbnailRead', () => {
// Seed an object so the presigned URL points at something real
// (the signer itself doesn't validate existence, but this keeps
// the test honest).
const key = 'thumb-read-test';
const key = mintedKey();
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
@@ -220,6 +225,57 @@ describe('thumbnails extension — handleThumbnailRead', () => {
expect((entry.thumbnail as string).startsWith('http')).toBe(true);
});
// `fsentries.thumbnail` is writable through the FS API, so a stored
// pointer is attacker input. Signing one the extension didn't mint would
// hand out a presigned read of an arbitrary object — including another
// user's file, whose key is its fsentry uuid in this same bucket.
it.each([
// Shaped exactly like an fs object key (and like a legacy thumbnail
// row) — indistinguishable from a planted pointer, so it fails closed.
[
"another user's file object",
`s3://${BUCKET}/${crypto.randomUUID()}`,
],
[
'a key outside the thumbnails namespace',
`s3://${BUCKET}/secrets/dump`,
],
[
'a namespace-lookalike key',
`s3://${BUCKET}/thumbnails/../${crypto.randomUUID()}`,
],
['a non-uuid inside the namespace', `s3://${BUCKET}/thumbnails/etc`],
])('refuses to presign a pointer naming %s', async (_label, thumbnail) => {
const entry: Record<string, unknown> = { thumbnail };
await handleThumbnailRead(entry, {
s3,
s3Presign,
bucketName: BUCKET,
bucketEndpoint: 'http://127.0.0.1:4566/puter-local/',
db: stubDb,
});
expect(entry.thumbnail).toBeNull();
});
it('signs against its own bucket, ignoring the one in the pointer', async () => {
const key = mintedKey();
const entry: Record<string, unknown> = {
thumbnail: `s3://attacker-named-bucket/${key}`,
};
await handleThumbnailRead(entry, {
s3,
s3Presign,
bucketName: BUCKET,
bucketEndpoint: 'http://127.0.0.1:4566/puter-local/',
db: stubDb,
});
// Signed for OUR bucket; `attacker-named-bucket` never reached S3.
const signed = entry.thumbnail as string;
expect(signed.startsWith('http')).toBe(true);
expect(signed).not.toContain('attacker-named-bucket');
expect(signed).toContain(BUCKET);
});
it('leaves the thumbnail untouched when not s3/https/data', async () => {
const entry: Record<string, unknown> = { thumbnail: 'about:blank' };
await handleThumbnailRead(entry, {
@@ -285,7 +341,7 @@ describe('thumbnails extension — handleFsRemoveNodeThumbnail', () => {
});
it('deletes the S3 object referenced by an s3:// thumbnail URL', async () => {
const key = 'thumb-remove-test';
const key = mintedKey();
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
@@ -297,7 +353,7 @@ describe('thumbnails extension — handleFsRemoveNodeThumbnail', () => {
await handleFsRemoveNodeThumbnail(
{ target: { thumbnail: `s3://${BUCKET}/${key}` } },
{ s3 },
{ s3, bucketName: BUCKET },
);
// GetObject should now error because the key was deleted.
@@ -306,15 +362,147 @@ describe('thumbnails extension — handleFsRemoveNodeThumbnail', () => {
).rejects.toThrow();
});
// The destructive half of the same confused deputy: the stored pointer
// decides which object is deleted, so a key the extension didn't mint
// must never reach DeleteObject.
it('does not delete an object the pointer names but we did not mint', async () => {
const victimKey = crypto.randomUUID(); // shaped like an fs object key
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: victimKey,
Body: Buffer.from(TINY_PNG_BASE64, 'base64'),
ContentType: 'image/png',
}),
);
await handleFsRemoveNodeThumbnail(
{ target: { thumbnail: `s3://${BUCKET}/${victimKey}` } },
{ s3, bucketName: BUCKET },
);
const survivor = await s3.send(
new GetObjectCommand({ Bucket: BUCKET, Key: victimKey }),
);
expect(survivor.ContentType).toBe('image/png');
});
it('is a no-op when the target has no thumbnail', async () => {
// Should not throw or attempt a delete.
await handleFsRemoveNodeThumbnail({ target: {} }, { s3 });
await handleFsRemoveNodeThumbnail(
{ target: {} },
{ s3, bucketName: BUCKET },
);
});
it('is a no-op when the thumbnail URL is not an s3:// pointer', async () => {
await handleFsRemoveNodeThumbnail(
{ target: { thumbnail: 'https://cdn.example.com/x.png' } },
{ s3 },
{ s3, bucketName: BUCKET },
);
});
});
describe('thumbnails extension — handleFsCopyNodeThumbnail', () => {
let server: PuterServer;
let s3: S3Client;
beforeAll(async () => {
server = await setupTestServer();
s3 = server.clients.s3.get();
});
afterAll(async () => {
await server?.shutdown();
});
it('duplicates the shared thumbnail object and repoints the copied row', async () => {
const sourceKey = mintedKey();
const body = Buffer.from(TINY_PNG_BASE64, 'base64');
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: sourceKey,
Body: body,
ContentType: 'image/png',
}),
);
const copyUuid = crypto.randomUUID();
const db = { write: vi.fn().mockResolvedValue(undefined) };
await handleFsCopyNodeThumbnail(
{
copy: {
thumbnail: `s3://${BUCKET}/${sourceKey}`,
uuid: copyUuid,
},
},
{ s3, bucketName: BUCKET, db },
);
// The copied row was repointed at a fresh object...
expect(db.write).toHaveBeenCalledTimes(1);
const [, params] = db.write.mock.calls[0] as [string, [string, string]];
const [newPointer, updatedUuid] = params;
expect(updatedUuid).toBe(copyUuid);
expect(newPointer.startsWith(`s3://${BUCKET}/thumbnails/`)).toBe(true);
expect(newPointer).not.toBe(`s3://${BUCKET}/${sourceKey}`);
// ...whose content matches, while the source object survives — so
// deleting either entry can no longer break the other's thumbnail.
const newKey = newPointer.slice(`s3://${BUCKET}/`.length);
const duplicated = await s3.send(
new GetObjectCommand({ Bucket: BUCKET, Key: newKey }),
);
expect(
(await streamToBuffer(duplicated.Body as never)).equals(body),
).toBe(true);
const original = await s3.send(
new GetObjectCommand({ Bucket: BUCKET, Key: sourceKey }),
);
expect(original.ContentType).toBe('image/png');
});
it('drops the pointer when the shared object is already gone', async () => {
const copyUuid = crypto.randomUUID();
const db = { write: vi.fn().mockResolvedValue(undefined) };
await handleFsCopyNodeThumbnail(
{
copy: {
thumbnail: `s3://${BUCKET}/${mintedKey()}`, // never uploaded
uuid: copyUuid,
},
},
{ s3, bucketName: BUCKET, db },
);
expect(db.write).toHaveBeenCalledTimes(1);
const [sql, params] = db.write.mock.calls[0] as [string, [string]];
expect(sql).toContain('NULL');
expect(params).toEqual([copyUuid]);
});
it('does not duplicate an object the pointer names but we did not mint', async () => {
const foreignKey = crypto.randomUUID(); // shaped like an fs object key
const db = { write: vi.fn().mockResolvedValue(undefined) };
await handleFsCopyNodeThumbnail(
{
copy: {
thumbnail: `s3://${BUCKET}/${foreignKey}`,
uuid: crypto.randomUUID(),
},
},
{ s3, bucketName: BUCKET, db },
);
expect(db.write).not.toHaveBeenCalled();
});
it('is a no-op when the copy has no thumbnail', async () => {
const db = { write: vi.fn().mockResolvedValue(undefined) };
await handleFsCopyNodeThumbnail(
{ copy: { thumbnail: null, uuid: crypto.randomUUID() } },
{ s3, bucketName: BUCKET, db },
);
expect(db.write).not.toHaveBeenCalled();
});
});
+152 -29
View File
@@ -1,4 +1,5 @@
import {
CopyObjectCommand,
DeleteObjectCommand,
GetObjectCommand,
PutObjectCommand,
@@ -13,6 +14,54 @@ const clients = extension.import('client');
const MAX_THUMBNAIL_BYTES = 2 * 1024 * 1024;
const MAX_THUMBNAIL_PIXELS = 64e6;
// Namespace every object this extension writes. An fs object's key is its
// bare fsentry uuid and the default config points `thumbnailStore.name` at
// the same bucket as `s3_bucket`, so without a prefix of our own there is no
// way to tell a thumbnail we minted from any other object in the deployment.
const THUMBNAIL_KEY_PREFIX = 'thumbnails/';
const UUID_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
const mintThumbnailKey = (): string =>
`${THUMBNAIL_KEY_PREFIX}${crypto.randomUUID()}`;
/**
* Extract the object key from a stored thumbnail pointer, or null when the
* pointer isn't one this extension minted.
*
* `fsentries.thumbnail` is writable through the FS API, so neither half of the
* stored string is trusted: the bucket is discarded (callers always pass their
* own) and the key must sit under {@link THUMBNAIL_KEY_PREFIX} with a random
* uuid. Honouring an arbitrary key would lend this extension's storage
* credentials to whatever object the caller named — in the shared-bucket layout
* that is every user's file, since an fs object's key is its fsentry uuid.
* Legacy bare-uuid thumbnails fail the check and are treated as absent; they
* are indistinguishable from a planted pointer, so there is nothing safer to do
* with them than stop signing them.
*/
const resolveThumbnailKey = (pointer: string): string | null => {
let key: string;
if (pointer.startsWith('s3://')) {
const rest = pointer.slice('s3://'.length);
const slash = rest.indexOf('/');
if (slash === -1) return null;
key = rest.slice(slash + 1);
} else {
let pathname: string;
try {
pathname = new URL(pointer).pathname;
} catch {
return null;
}
const segments = pathname.replace(/^\/+/, '').split('/');
segments.shift(); // bucket
key = segments.join('/');
}
if (!key.startsWith(THUMBNAIL_KEY_PREFIX)) return null;
const id = key.slice(THUMBNAIL_KEY_PREFIX.length);
return UUID_PATTERN.test(id) ? key : null;
};
// S3 client + bucket config — lazily resolved after boot from config.
let s3Client: S3Client | null = null;
let s3PresignClient: S3Client | null = null;
@@ -112,7 +161,7 @@ export async function handleThumbnailCreated(
return;
}
const key = crypto.randomUUID();
const key = mintThumbnailKey();
event.url = `s3://${deps.bucketName}/${key}`;
await deps.s3.send(
@@ -151,7 +200,7 @@ export const handleThumbnailUploadPrepare = async (
continue;
}
const key = crypto.randomUUID();
const key = mintThumbnailKey();
const command = new PutObjectCommand({
Bucket: deps.bucketName,
Key: key,
@@ -178,27 +227,27 @@ export const handleThumbnailRead = async (
if (typeof thumb !== 'string' || !thumb) return;
const presignClient = deps.s3Presign;
if (thumb.startsWith('s3://')) {
const [bucket, key] = thumb.slice(5).split('/');
entry.thumbnail = await getSignedUrl(
presignClient,
new GetObjectCommand({ Bucket: bucket, Key: key }),
{ expiresIn: 604800 },
);
} else if (
thumb.startsWith('https') &&
thumb.includes(new URL(deps.bucketEndpoint).hostname)
) {
if (
thumb.startsWith('s3://') ||
// Legacy format — remove after full migration
const [bucket, key] = new URL(thumb).pathname.slice(1).split('/');
(thumb.startsWith('https') &&
thumb.includes(new URL(deps.bucketEndpoint).hostname))
) {
const key = resolveThumbnailKey(thumb);
if (!key) {
// Not a pointer we minted — refuse to sign it rather than hand
// out a presigned read of whatever object it names.
entry.thumbnail = null;
return;
}
entry.thumbnail = await getSignedUrl(
presignClient,
new GetObjectCommand({ Bucket: bucket, Key: key }),
new GetObjectCommand({ Bucket: deps.bucketName, Key: key }),
{ expiresIn: 604800 },
);
} else if (thumb.startsWith('data')) {
// Inline data-URL migration: upload to S3 and update the DB entry.
const key = crypto.randomUUID();
const key = mintThumbnailKey();
const { mimeType, data } = base64ParseDataUrl(thumb);
const newUrl = `s3://${deps.bucketName}/${key}`;
@@ -232,15 +281,72 @@ export const handleThumbnailRead = async (
}
};
export const handleFsRemoveNodeThumbnail = async (
payload: { target: Record<string, unknown> },
deps: { s3: S3Client },
export const handleFsCopyNodeThumbnail = async (
payload: { copy?: { thumbnail?: string | null; uuid?: string } | null },
deps: {
s3: S3Client;
bucketName: string;
db: { write: (sql: string, params: unknown[]) => Promise<unknown> };
},
): Promise<void> => {
const thumbnailUrl = payload.target.thumbnail as string | undefined;
if (!thumbnailUrl || !thumbnailUrl.startsWith('s3://')) return;
const copy = payload.copy;
const thumbnailUrl = copy?.thumbnail;
if (!copy || !copy.uuid || typeof thumbnailUrl !== 'string') return;
const [bucket, key] = thumbnailUrl.slice(5).split('/');
await deps.s3.send(new DeleteObjectCommand({ Bucket: bucket, Key: key }));
// Same trust rule as the read and remove paths: only touch objects this
// extension minted.
const sourceKey = resolveThumbnailKey(thumbnailUrl);
if (!sourceKey) return;
// The copied row points at the SAME S3 object as its source, and
// fs.remove.node deletes the pointed-to object — so the first removal
// among the sharers (an overwrite, a trash purge) would break every
// other sharer's thumbnail. Give the copy an object of its own.
const newKey = mintThumbnailKey();
try {
await deps.s3.send(
new CopyObjectCommand({
Bucket: deps.bucketName,
CopySource: `${deps.bucketName}/${sourceKey}`,
Key: newKey,
}),
);
} catch (err) {
// The shared object is already gone (e.g. a sharer was removed
// before this fix existed) — the pointer is dead either way, so
// drop it rather than leave the row advertising a thumbnail it
// doesn't have.
await deps.db.write(
'UPDATE `fsentries` SET `thumbnail` = NULL WHERE `uuid` = ?',
[copy.uuid],
);
console.warn('[thumbnails] failed to duplicate thumbnail on copy', err);
return;
}
await deps.db.write(
'UPDATE `fsentries` SET `thumbnail` = ? WHERE `uuid` = ?',
[`s3://${deps.bucketName}/${newKey}`, copy.uuid],
);
};
export const handleFsRemoveNodeThumbnail = async (
payload: { target: { thumbnail?: string | null } },
deps: { s3: S3Client; bucketName: string },
): Promise<void> => {
const thumbnailUrl = payload.target.thumbnail;
if (!thumbnailUrl) return;
// Same trust rule as the read path, and load-bearing for the same reason:
// the pointer decides which object gets deleted, so a key we didn't mint
// would let the owner of one file destroy an object belonging to someone
// else just by naming it here.
const key = resolveThumbnailKey(thumbnailUrl);
if (!key) return;
await deps.s3.send(
new DeleteObjectCommand({ Bucket: deps.bucketName, Key: key }),
);
};
extension.on(
@@ -279,12 +385,29 @@ extension.on('thumbnail.read', async (_key, entry: Record<string, unknown>) => {
});
});
// -- fs.copy.node ----------------------------------------------------
// A copied entry initially shares its source's thumbnail object; duplicate
// it so removing either entry can't break the other's thumbnail.
extension.on('fs.copy.node', async (_key, payload) => {
await handleFsCopyNodeThumbnail(
payload as {
copy?: { thumbnail?: string | null; uuid?: string } | null;
},
{
s3: getClient(),
bucketName: thumbnailBucketName,
db: clients.db,
},
);
});
// -- fs.remove.node --------------------------------------------------
// Delete S3 thumbnail when the file is removed.
extension.on(
'fs.remove.node',
async (_key, payload: { target: Record<string, unknown> }) => {
await handleFsRemoveNodeThumbnail(payload, { s3: getClient() });
},
);
extension.on('fs.remove.node', async (_key, payload) => {
await handleFsRemoveNodeThumbnail(payload, {
s3: getClient(),
bucketName: thumbnailBucketName,
});
});
+6 -1
View File
@@ -109,6 +109,9 @@ describe('whoami extension — handleWhoami', () => {
expect(body.oidc_only).toBe(false);
// is_user_token is present (true) for user actors.
expect(body.is_user_token).toBe(true);
// Account creation time, in unix seconds.
expect(typeof body.created_ts).toBe('number');
expect(body.created_ts).toBeGreaterThan(0);
// `directories` is only sent to user actors — confirm it’s present.
expect(body.directories).toBeDefined();
// taskbar_items is only sent to user actors.
@@ -132,7 +135,7 @@ describe('whoami extension — handleWhoami', () => {
expect(flags.payment_bypass).toBeUndefined();
});
it('strips desktop_bg_* and human_readable_age fields for app actors', async () => {
it('strips desktop_bg_*, created_ts and human_readable_age fields for app actors', async () => {
const user = await seedUser();
const { res, captured } = makeRes();
@@ -154,6 +157,8 @@ describe('whoami extension — handleWhoami', () => {
expect(body.desktop_bg_color).toBeUndefined();
expect(body.desktop_bg_fit).toBeUndefined();
expect(body.human_readable_age).toBeUndefined();
// Account age, in either form, is not exposed to apps.
expect(body.created_ts).toBeUndefined();
// Directories are user-only.
expect(body.directories).toBeUndefined();
});
+33 -11
View File
@@ -14,6 +14,15 @@ const timeago = (() => {
return new TimeAgo('en-US');
})();
// User timestamps come off the DB as SQL datetime strings; the wire format
// for all of them is unix seconds. Unparseable values are dropped rather
// than sent as NaN.
const toUnixSeconds = (value: unknown): number | undefined => {
if (!value) return undefined;
const ms = new Date(value as string | number | Date).getTime();
return Number.isNaN(ms) ? undefined : Math.round(ms / 1000);
};
// Allowlist of `config.feature_flags` keys safe to surface via /whoami.
// Anything not listed here stays server-side, so internal flags
// (payment_bypass, staff_only_*, etc.) cannot leak by accident. Add a
@@ -98,6 +107,7 @@ export const handleWhoami = async (
: undefined,
otp: !!user.otp_enabled,
feature_flags,
created_ts: toUnixSeconds(user.timestamp),
human_readable_age: user.timestamp
? timeago.format(new Date(user.timestamp as string))
: null,
@@ -141,14 +151,9 @@ export const handleWhoami = async (
}
// Last activity
if (user.last_activity_ts) {
try {
details.last_activity_ts = Math.round(
new Date(user.last_activity_ts as string).getTime() / 1000,
);
} catch {
/* ignore parse error */
}
const lastActivityTs = toUnixSeconds(user.last_activity_ts);
if (lastActivityTs !== undefined) {
details.last_activity_ts = lastActivityTs;
}
// Strip sensitive fields for app actors
@@ -164,6 +169,7 @@ export const handleWhoami = async (
delete details.desktop_bg_color;
delete details.desktop_bg_fit;
delete details.human_readable_age;
delete details.created_ts;
delete details.is_user_token;
delete details.metadata;
}
@@ -183,8 +189,7 @@ export const handleWhoami = async (
}
const subscription = details.subscription as
| { offering?: Record<string, unknown> }
| undefined;
{ offering?: Record<string, unknown> } | undefined;
if (subscription?.offering) {
delete subscription.offering.group;
delete subscription.offering.benefits;
@@ -196,6 +201,23 @@ export const handleWhoami = async (
extension.get(
'/whoami',
{ subdomain: 'api', requireAuth: true, allowUnconfirmed: true },
{
subdomain: 'api',
requireAuth: true,
allowUnconfirmed: true,
// The GUI polls this, and each call fans out to every `whoami`
// event listener — so it costs more than the response suggests.
//
// It is also the call everything else leans on to find out who it is
// talking to, so it rides along with unrelated work rather than
// arriving at its own pace: the ceiling has to clear whatever the
// busiest session is doing, not what a person clicks.
rateLimit: {
scope: 'whoami',
limit: 1_800,
window: 60_000,
key: 'user',
},
},
handleWhoami,
);
+15 -12
View File
@@ -11,7 +11,7 @@
# What this does, in order:
# 1. Checks that docker (with the compose plugin) exists.
# 2. Creates ./puter-selfhosted/ (override with $env:PUTER_DIR).
# 3. Downloads docker-compose.yml + nginx.conf from the OSS repo.
# 3. Downloads docker-compose.yml + caddy/Caddyfile from the OSS repo.
# 4. Generates fresh secrets and writes .env + puter/config/config.json.
# 5. Runs `docker compose up -d` and prints how to find the admin password.
#
@@ -23,7 +23,7 @@
# PUTER_DIR install directory (default: ./puter-selfhosted)
# PUTER_URL base URL to fetch docker-compose.yml (default: GitHub raw, main branch)
# PUTER_DOMAIN domain Puter will serve on (default: puter.localhost)
# PUTER_PORT HTTP port for nginx (default: 80)
# PUTER_PORT HTTP port for Caddy (default: 80)
# PUTER_FORCE set to 1 to overwrite existing .env / config.json
[CmdletBinding()]
@@ -77,10 +77,10 @@ if ($LASTEXITCODE -ne 0) {
# ── Step 2: install dir ─────────────────────────────────────────────
$null = New-Item -ItemType Directory -Force -Path $PuterDir
Set-Location $PuterDir
$null = New-Item -ItemType Directory -Force -Path 'puter/config', 'puter/data', 'puter/tls'
$null = New-Item -ItemType Directory -Force -Path 'puter/config', 'puter/data', 'puter/data/caddy', 'puter/tls'
Write-Log "install dir: $((Get-Location).Path)"
# ── Step 3: docker-compose.yml + nginx config ──────────────────────
# ── Step 3: docker-compose.yml + Caddy config ──────────────────────
Write-Log "downloading docker-compose.yml from $PuterUrl"
try {
Invoke-WebRequest -Uri "$PuterUrl/docker-compose.yml" -OutFile 'docker-compose.yml' -UseBasicParsing
@@ -88,17 +88,19 @@ try {
Die "could not fetch $PuterUrl/docker-compose.yml — $_"
}
Write-Log "downloading nginx/nginx.conf from $PuterUrl"
$null = New-Item -ItemType Directory -Force -Path 'nginx'
# The Caddyfile is domain-agnostic — it answers on every Host and leaves
# the subdomain routing to Puter — so there's nothing to template in.
Write-Log "downloading caddy/Caddyfile from $PuterUrl"
$null = New-Item -ItemType Directory -Force -Path 'caddy'
# If the path was previously auto-created as a directory by a failed
# `compose up`, remove it so we can write the file there.
if (Test-Path 'nginx/nginx.conf' -PathType Container) {
Remove-Item 'nginx/nginx.conf' -Recurse -Force
if (Test-Path 'caddy/Caddyfile' -PathType Container) {
Remove-Item 'caddy/Caddyfile' -Recurse -Force
}
try {
Invoke-WebRequest -Uri "$PuterUrl/nginx/nginx.conf" -OutFile 'nginx/nginx.conf' -UseBasicParsing
Invoke-WebRequest -Uri "$PuterUrl/caddy/Caddyfile" -OutFile 'caddy/Caddyfile' -UseBasicParsing
} catch {
Die "could not fetch $PuterUrl/nginx/nginx.conf — $_"
Die "could not fetch $PuterUrl/caddy/Caddyfile — $_"
}
# ── Step 4: secrets, .env, config.json ──────────────────────────────
@@ -121,7 +123,8 @@ if ($writeConfig) {
$envContent = @"
HTTP_PORT=$PuterPort
# HTTPS_PORT=443 # uncomment after enabling TLS (see doc/selfhosting/full-stack.md)
# HTTPS_PORT=443 # uncomment after enabling TLS in caddy/Caddyfile
# # (see "Step 3 — TLS" in doc/self-hosting.md)
MARIADB_ROOT_PASSWORD=$mariadbRootPw
MARIADB_DATABASE=puter
@@ -205,4 +208,4 @@ Write-Log " cd $PuterDir; docker compose logs -f puter"
Write-Log ''
Write-Log "open http://${PuterDomain}:${PuterPort} once the puter container is healthy."
Write-Log 'first-boot admin password is logged once — grab it with:'
Write-Log " cd $PuterDir; docker compose logs puter | Select-String password"
Write-Log " cd $PuterDir; docker compose logs puter | Select-String password"
+36 -18
View File
@@ -7,7 +7,8 @@
# What this does, in order:
# 1. Checks that docker (with the compose plugin), curl, and openssl exist.
# 2. Creates ./puter-selfhosted/ (override with PUTER_DIR=...).
# 3. Downloads docker-compose.yml from the OSS repo (raw.githubusercontent.com).
# 3. Downloads docker-compose.yml + caddy/Caddyfile from the OSS repo
# (raw.githubusercontent.com).
# 4. Generates fresh secrets and writes .env + puter/config/config.json.
# 5. Runs `docker compose up -d` and prints the first-boot admin password.
#
@@ -19,15 +20,17 @@
# PUTER_DIR install directory (default: ./puter-selfhosted)
# PUTER_URL base URL to fetch docker-compose.yml (default: GitHub raw, main branch)
# PUTER_DOMAIN domain Puter will serve on (default: puter.localhost)
# PUTER_PORT HTTP port for nginx (default: 80)
# PUTER_PORT HTTP port for Caddy (default: 80)
# PUTER_PROTOCOL public scheme: http | https (default: http)
# Set https when a TLS-terminating reverse proxy (Caddy,
# Traefik, nginx, a cloud LB) sits in front — Puter then
# builds https:// origins, S3 URLs, and redirects. See
# "Running behind your own reverse proxy" in doc/self-hosting.md.
# Set https once TLS is terminating in front of Puter —
# either the bundled Caddy with your certs (see "Step 3 —
# TLS" in doc/self-hosting.md) or your own proxy (Traefik,
# a cloud LB). Puter then builds https:// origins, S3
# URLs, and redirects. Leave it http and the installer
# serves plain HTTP on PUTER_PORT.
# PUTER_TRUST_PROXY number of reverse-proxy hops in front (default: 1)
# 1 = the bundled nginx (or a single external proxy);
# 2 = two hops (e.g. Cloudflare → your proxy → Puter).
# 1 = the bundled Caddy (or a single external proxy);
# 2 = two hops (e.g. Cloudflare → Caddy → Puter).
# PUTER_ENV prod | dev (default: prod)
# PUTER_FORCE set to 1 to overwrite existing .env / config.json
@@ -72,25 +75,27 @@ mkdir -p puter/config puter/data puter/tls
# bind-mount roots sidesteps the mismatch without guessing each image's
# internal UID. (Docker Desktop on macOS/Windows papers over this with
# its VM layer; native Linux docker on Debian/Alpine doesn't.)
mkdir -p puter/data/valkey puter/data/mariadb puter/data/dynamo puter/data/s3 puter/data/puter
chmod 0777 puter/data/valkey puter/data/mariadb puter/data/dynamo puter/data/s3 puter/data/puter
mkdir -p puter/data/valkey puter/data/mariadb puter/data/dynamo puter/data/s3 puter/data/puter puter/data/caddy
chmod 0777 puter/data/valkey puter/data/mariadb puter/data/dynamo puter/data/s3 puter/data/puter puter/data/caddy
log "install dir: $(pwd)"
# ── Step 3: docker-compose.yml + nginx config ──────────────────────
# ── Step 3: docker-compose.yml + Caddy config ──────────────────────
log "downloading docker-compose.yml from $PUTER_URL"
curl -fsSL "$PUTER_URL/docker-compose.yml" -o docker-compose.yml \
|| die "could not fetch $PUTER_URL/docker-compose.yml"
# nginx is mounted as `./nginx/nginx.conf:/etc/nginx/nginx.conf:ro` — if
# Caddy is mounted as `./caddy/Caddyfile:/etc/caddy/Caddyfile:ro,z` — if
# the host file is missing, docker silently creates a directory at that
# path and the mount fails with "not a directory" at container start.
log "downloading nginx/nginx.conf from $PUTER_URL"
mkdir -p nginx
# The Caddyfile is domain-agnostic — it answers on every Host and leaves
# the subdomain routing to Puter — so there's nothing to template in.
log "downloading caddy/Caddyfile from $PUTER_URL"
mkdir -p caddy
# If the path was previously auto-created as a dir by a failed `compose up`,
# remove it so curl can write the file.
[ -d nginx/nginx.conf ] && rmdir nginx/nginx.conf 2>/dev/null || true
curl -fsSL "$PUTER_URL/nginx/nginx.conf" -o nginx/nginx.conf \
|| die "could not fetch $PUTER_URL/nginx/nginx.conf"
[ -d caddy/Caddyfile ] && rmdir caddy/Caddyfile 2>/dev/null || true
curl -fsSL "$PUTER_URL/caddy/Caddyfile" -o caddy/Caddyfile \
|| die "could not fetch $PUTER_URL/caddy/Caddyfile"
# ── Step 4: secrets, .env, config.json ──────────────────────────────
write_config=1
@@ -113,7 +118,8 @@ if [ "$write_config" = "1" ]; then
cat > .env <<EOF
HTTP_PORT=$PUTER_PORT
# HTTPS_PORT=443 # uncomment after enabling TLS (see doc/selfhosting/full-stack.md)
# HTTPS_PORT=443 # uncomment after enabling TLS in caddy/Caddyfile
# # (see "Step 3 — TLS" in doc/self-hosting.md)
MARIADB_ROOT_PASSWORD=$MARIADB_ROOT_PASSWORD
MARIADB_DATABASE=puter
@@ -190,6 +196,18 @@ EOF
EOF
fi
# `https` only works if something in front is actually terminating it. The
# bundled Caddy does that from ./puter/tls once you enable its `:443` block
# (see "Step 3 — TLS" in doc/self-hosting.md); your own edge proxy is fine
# too. Neither in place means Puter hands out https:// URLs for an endpoint
# that only speaks HTTP — broken logins and mixed content.
if [ "$PUTER_PROTOCOL" = "https" ] && [ ! -f puter/tls/fullchain.pem ]; then
warn "PUTER_PROTOCOL=https but ./puter/tls/fullchain.pem is missing."
warn "The bundled Caddy is still serving plain HTTP on port $PUTER_PORT."
warn "Either drop a wildcard cert into ./puter/tls and uncomment the :443"
warn "block in caddy/Caddyfile, or terminate TLS in your own proxy."
fi
# ── Step 5: bring it up ─────────────────────────────────────────────
log "docker compose up -d"
docker compose up -d
-135
View File
@@ -1,135 +0,0 @@
# Reverse proxy in front of Puter — mirrors what the prod ALB does:
# accepts every Host header, forwards to the Puter container, and lets
# the Puter app handle subdomain-based routing internally (api.*,
# site.*, app.*, etc).
#
# To enable TLS:
# 1. Drop your fullchain.pem + privkey.pem into ./puter/tls/.
# 2. Uncomment the 443 server{} block below.
# 3. Update server_name to your domain (and wildcard subdomains).
worker_processes auto;
events {
worker_connections 4096;
}
http {
# Required for Puter's WebSocket / socket.io upgrades.
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
# Rough size cap that mirrors prod ALB defaults; tune for your
# uploads. Puter chunks large uploads, so 1 GiB per request is plenty.
client_max_body_size 1024m;
proxy_read_timeout 600s;
proxy_send_timeout 600s;
proxy_buffering off;
server_tokens off;
upstream puter_backend {
server puter:4100;
keepalive 32;
}
upstream s3_backend {
# RustFS — see `s3` service in docker-compose.full.yml. Browsers
# PUT/GET here for presigned-URL uploads / downloads. Routed via
# the `s3.<domain>` subdomain so signature verification works
# (Host header preserved end-to-end) and so HTTPS stays clean
# (no mixed-content from a port-9000 host publish).
server s3:9000;
keepalive 32;
}
# ── HTTP (port 80) ─────────────────────────────────────────────
server {
listen 80;
listen [::]:80;
server_name ~^s3\.;
location / {
proxy_pass http://s3_backend;
proxy_http_version 1.1;
# Critical: preserve the original Host so RustFS validates
# the request against the same host the URL was signed for.
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
# Note: when you enable TLS, replace this block with a redirect:
# return 301 https://$host$request_uri;
location / {
proxy_pass http://puter_backend;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
}
# ── HTTPS (port 443) — uncomment after dropping certs in ./puter/tls/ ─
# server {
# listen 443 ssl;
# listen [::]:443 ssl;
# http2 on;
# server_name ~^s3\.;
#
# ssl_certificate /etc/nginx/tls/fullchain.pem;
# ssl_certificate_key /etc/nginx/tls/privkey.pem;
# ssl_protocols TLSv1.2 TLSv1.3;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_session_cache shared:SSL:10m;
# ssl_session_timeout 10m;
#
# location / {
# proxy_pass http://s3_backend;
# proxy_http_version 1.1;
# proxy_set_header Host $host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header X-Forwarded-Proto $scheme;
# }
# }
#
# server {
# listen 443 ssl default_server;
# listen [::]:443 ssl default_server;
# http2 on;
# server_name _;
#
# ssl_certificate /etc/nginx/tls/fullchain.pem;
# ssl_certificate_key /etc/nginx/tls/privkey.pem;
# ssl_protocols TLSv1.2 TLSv1.3;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_session_cache shared:SSL:10m;
# ssl_session_timeout 10m;
#
# location / {
# proxy_pass http://puter_backend;
# proxy_http_version 1.1;
# proxy_set_header Host $host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header X-Forwarded-Proto $scheme;
# proxy_set_header X-Forwarded-Host $host;
# proxy_set_header X-Forwarded-Port $server_port;
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection $connection_upgrade;
# }
# }
}
+4 -3
View File
@@ -1,12 +1,12 @@
{
"name": "puter.com",
"version": "2.5.1",
"version": "26.07",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "puter.com",
"version": "2.5.1",
"version": "26.07",
"license": "AGPL-3.0-only",
"workspaces": [
"src/*",
@@ -1304,6 +1304,7 @@
},
"node_modules/@clack/prompts/node_modules/is-unicode-supported": {
"version": "1.3.0",
"extraneous": true,
"inBundle": true,
"license": "MIT",
"engines": {
@@ -20252,7 +20253,7 @@
},
"src/puter-js": {
"name": "@heyputer/puter.js",
"version": "2.5.4",
"version": "2.6.1",
"license": "Apache-2.0",
"dependencies": {
"@heyputer/kv.js": "^0.2.1",
+6 -4
View File
@@ -51,8 +51,8 @@
"yaml": "^2.8.1"
},
"scripts": {
"test:backend": "npm run setupExtensions && vitest run --config src/backend/vitest.config.ts ",
"test:backend:postgres": "npm run setupExtensions && PUTER_TEST_DB_ENGINE=postgres vitest run --config src/backend/vitest.config.ts ",
"test:backend": "npm run setupExtensions && npm run build:workerLib && vitest run --config src/backend/vitest.config.ts ",
"test:backend:postgres": "npm run setupExtensions && npm run build:workerLib && PUTER_TEST_DB_ENGINE=postgres vitest run --config src/backend/vitest.config.ts ",
"test:puterjs": "npm run setupExtensions && vitest run --config src/puter-js/tests/api/vitest.config.ts",
"test:puterjs:node": "npm run test:puterjs -- src/puter-js/tests/api/runners/node.test.ts",
"test:puterjs:browser": "npm run test:puterjs -- src/puter-js/tests/api/runners/browser.test.ts",
@@ -60,14 +60,16 @@
"test:puterjs:coverage": "npm run build:workerLib:coverage && npm run setupExtensions && rm -rf src/puter-js/coverage && PUTER_COVERAGE=1 vitest run --config src/puter-js/tests/api/vitest.config.ts && node ./tools/puterjsCoverageReport.mjs",
"build:workerLib:coverage": "cd src/puter-js && npm run build:coverage && cd ../worker && npm run build",
"start:gui": "nodemon --exec \"node dev-server.js\" ",
"start": "node --enable-source-maps -r ./dist/src/backend/telemetry.js ./dist/src/backend/index.js",
"prestart": "npm run setupExtensions && npm run build:ts",
"start": "node ./tools/start.mjs",
"dev": "npm start",
"build": "npm run setupExtensions && npm run build:ts && cd src/gui && node ./build.js && cd ../puter-js && npm run build",
"build:workerLib": "cd src/puter-js && npm run build && cd ../worker && npm run build",
"check-translations": "node tools/check-translations.js",
"prepare": "husky",
"build:ts": "tsc -p tsconfig.build.json && node ./tools/write-dist-package-json.mjs",
"check:puterjs:types": "node tools/checkPuterjsTypes.mjs",
"typecheck": "node tools/typecheck.mjs",
"typecheck:update": "node tools/typecheck.mjs --update",
"setupExtensions": "node ./tools/extensionSetup.mjs"
},
"workspaces": [
@@ -0,0 +1,521 @@
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { AlarmClient } from './AlarmClient';
import type { IConfig, IPagerConfig } from '../../types';
import type { AlertPayload } from './types';
const pdEvent = vi.hoisted(() => vi.fn(async () => ({})));
vi.mock('@pagerduty/pdjs', () => ({ event: pdEvent }));
const makeClient = (pager: IPagerConfig = {}) =>
new AlarmClient({ serverId: 'test-node', pager } as unknown as IConfig);
/** Register a capturing handler in place of a real transport. */
const capture = (
client: AlarmClient,
minSeverity?: 'critical' | 'error' | 'warning' | 'info',
maxSeverity?: 'critical' | 'error' | 'warning' | 'info',
) => {
const seen: AlertPayload[] = [];
client.addAlertHandler(
async (alert) => {
seen.push(alert);
},
{ name: 'capture', minSeverity, maxSeverity },
);
return seen;
};
describe('AlarmClient severity routing', () => {
beforeEach(() => {
vi.spyOn(console, 'error').mockImplementation(() => {});
vi.spyOn(console, 'warn').mockImplementation(() => {});
vi.spyOn(console, 'log').mockImplementation(() => {});
pdEvent.mockClear();
});
it('defaults to critical when the call site says nothing', () => {
const client = makeClient();
const seen = capture(client);
client.create('boom', 'everything is on fire');
expect(seen).toHaveLength(1);
expect(seen[0].severity).toBe('critical');
});
it('honours a configured default severity', () => {
const client = makeClient({ defaultSeverity: 'info' });
const seen = capture(client);
client.create('boom', 'everything is on fire');
expect(seen[0].severity).toBe('info');
});
it('skips handlers whose floor the alarm does not reach', () => {
const client = makeClient();
const paging = capture(client, 'warning');
const chat = capture(client, 'info');
client.create('rate-limit', 'a user hit a limit', {}, 'info');
expect(paging).toHaveLength(0);
expect(chat).toHaveLength(1);
});
it('skips handlers whose ceiling the alarm exceeds', () => {
const client = makeClient();
const chat = capture(client, 'info', 'info');
client.create('rate-limit', 'a user hit a limit', {}, 'info');
client.create('outage', 'everything is on fire', {}, 'critical');
expect(chat.map((alert) => alert.id)).toEqual(['rate-limit']);
});
it('retiers an alarm from config', () => {
const client = makeClient({
severityOverrides: { 'noisy:*': 'info' },
});
const paging = capture(client, 'warning');
const chat = capture(client, 'info');
client.create('noisy:thing', 'used to page', {}, 'critical');
expect(paging).toHaveLength(0);
expect(chat[0].severity).toBe('info');
});
it('mutes an alarm from config', () => {
const client = makeClient({
severityOverrides: { 'noisy:thing': 'mute' },
});
const chat = capture(client, 'info');
client.create('noisy:thing', 'not worth reporting');
client.create('noisy:thing', 'still not worth reporting');
expect(chat).toHaveLength(0);
});
it('lets config override a known-error rule', () => {
const client = makeClient({
severityOverrides: { 'known:thing': 'critical' },
});
client.setKnownErrors([
{
match: { id: 'known:thing' },
action: { type: 'severity', value: 'info' },
},
]);
const paging = capture(client, 'warning');
client.create('known:thing', 'known but escalated');
expect(paging[0].severity).toBe('critical');
});
it('still respects a no-alert known-error rule', () => {
const client = makeClient();
client.setKnownErrors([
{ match: { id: 'known:quiet' }, action: { type: 'no-alert' } },
]);
const chat = capture(client, 'info');
client.create('known:quiet', 'suppressed');
expect(chat).toHaveLength(0);
});
});
describe('AlarmClient alert payload', () => {
beforeEach(() => {
vi.spyOn(console, 'error').mockImplementation(() => {});
vi.spyOn(console, 'warn').mockImplementation(() => {});
});
it('reports occurrence counts across repeats', () => {
const client = makeClient();
const seen = capture(client);
client.create('flap', 'first');
client.create('flap', 'second');
expect(seen[0]).toMatchObject({ repeatCount: 1, isRepeat: false });
expect(seen[1]).toMatchObject({ repeatCount: 2, isRepeat: true });
});
it('renders fields as strings and lifts the stack out of the error', () => {
const client = makeClient();
const seen = capture(client);
const error = new Error('kaboom');
client.create('boom', 'failed', { error, status: 500 }, 'critical');
expect(seen[0].fields.status).toBe('500');
expect(seen[0].trace).toBe(error.stack);
expect(seen[0].shortId).toMatch(/^[a-z]+-[a-z]+-[a-z]+$/);
});
it('keeps one failing handler from starving the others', async () => {
const client = makeClient();
client.addAlertHandler(
async () => {
throw new Error('transport down');
},
{ name: 'broken' },
);
const seen = capture(client);
client.create('boom', 'failed');
await Promise.resolve();
expect(seen).toHaveLength(1);
});
it('suppresses alarms once draining', () => {
const client = makeClient();
const seen = capture(client);
client.onServerPrepareShutdown();
client.create('boom', 'too late');
expect(seen).toHaveLength(0);
});
});
describe('AlarmClient transport registration', () => {
beforeEach(() => {
vi.spyOn(console, 'error').mockImplementation(() => {});
vi.spyOn(console, 'warn').mockImplementation(() => {});
vi.spyOn(console, 'log').mockImplementation(() => {});
pdEvent.mockClear();
});
it('splits info to Slack and everything above it to PagerDuty', async () => {
const fetchMock = vi.fn(async () => ({ ok: true, status: 200 }));
vi.stubGlobal('fetch', fetchMock);
const client = makeClient({
pagerduty: { enabled: true, routingKey: 'rk' },
slack: { enabled: true, webhookUrl: 'https://hooks.example/abc' },
});
await client.onServerStart();
client.create('quiet:thing', 'informational', {}, 'info');
await vi.waitFor(() => expect(fetchMock).toHaveBeenCalledTimes(1));
expect(pdEvent).not.toHaveBeenCalled();
client.create('loud:thing', 'paging', {}, 'critical');
await vi.waitFor(() => expect(pdEvent).toHaveBeenCalledTimes(1));
// The pager has it; chat doesn't repeat it.
expect(fetchMock).toHaveBeenCalledTimes(1);
vi.unstubAllGlobals();
});
it('sends every severity to Slack when there is no pager', async () => {
const fetchMock = vi.fn(async () => ({ ok: true, status: 200 }));
vi.stubGlobal('fetch', fetchMock);
const client = makeClient({
slack: { enabled: true, webhookUrl: 'https://hooks.example/abc' },
});
await client.onServerStart();
client.create(
'loud:thing',
'nowhere else to send this',
{},
'critical',
);
await vi.waitFor(() => expect(fetchMock).toHaveBeenCalledTimes(1));
vi.unstubAllGlobals();
});
it('lets config widen the Slack ceiling back out', async () => {
const fetchMock = vi.fn(async () => ({ ok: true, status: 200 }));
vi.stubGlobal('fetch', fetchMock);
const client = makeClient({
pagerduty: { enabled: true, routingKey: 'rk' },
slack: {
enabled: true,
webhookUrl: 'https://hooks.example/abc',
maxSeverity: 'critical',
},
});
await client.onServerStart();
client.create('loud:thing', 'paging', {}, 'critical');
await vi.waitFor(() => expect(fetchMock).toHaveBeenCalledTimes(1));
vi.unstubAllGlobals();
});
it('gives each occurrence its own PagerDuty incident by default', async () => {
const client = makeClient({
pagerduty: { enabled: true, routingKey: 'rk' },
});
await client.onServerStart();
client.create('scan:failed', 'first', {}, 'warning');
client.create('scan:failed', 'second', {}, 'warning');
await vi.waitFor(() => expect(pdEvent).toHaveBeenCalledTimes(2));
const keys = pdEvent.mock.calls.map(
([arg]: [{ data: { dedup_key: string } }]) => arg.data.dedup_key,
);
expect(keys[0]).not.toBe(keys[1]);
});
it('collapses repeats of a dedup alarm onto one incident', async () => {
const client = makeClient({
pagerduty: { enabled: true, routingKey: 'rk' },
});
await client.onServerStart();
const raise = () =>
client.create(
'http_500:POST:/notif/mark-ack:deadlock',
'HTTP 500 on POST /notif/mark-ack: deadlock',
{},
'critical',
{ dedup: true },
);
raise();
raise();
await vi.waitFor(() => expect(pdEvent).toHaveBeenCalledTimes(2));
const keys = pdEvent.mock.calls.map(
([arg]: [{ data: { dedup_key: string } }]) => arg.data.dedup_key,
);
expect(keys[0]).toBe('http_500:POST:/notif/mark-ack:deadlock');
expect(keys[1]).toBe(keys[0]);
});
it('skips transports that are enabled but not configured', async () => {
const client = makeClient({
pagerduty: { enabled: true },
slack: { enabled: true },
});
await client.onServerStart();
const seen = capture(client);
client.create('boom', 'nowhere to send this');
// Only the capturing handler is registered.
expect(seen).toHaveLength(1);
expect(pdEvent).not.toHaveBeenCalled();
});
});
describe('AlarmClient alarm registry', () => {
beforeEach(() => {
vi.spyOn(console, 'error').mockImplementation(() => {});
vi.spyOn(console, 'warn').mockImplementation(() => {});
vi.spyOn(console, 'log').mockImplementation(() => {});
});
it('finds an alarm by its full id or its short id', () => {
const client = makeClient();
client.create('disk-pressure', 'nearly full');
const alarm = client.get('disk-pressure');
expect(alarm?.message).toBe('nearly full');
expect(client.get(alarm!.shortId)).toBe(alarm);
});
it('returns nothing for an id it has never seen', () => {
expect(makeClient().get('never-raised')).toBeUndefined();
});
it('forgets an alarm once cleared, under both ids', () => {
const client = makeClient();
client.create('disk-pressure', 'nearly full');
const shortId = client.get('disk-pressure')!.shortId;
client.clear('disk-pressure');
expect(client.get('disk-pressure')).toBeUndefined();
expect(client.get(shortId)).toBeUndefined();
});
it('ignores a clear for an alarm that is not active', () => {
const client = makeClient();
expect(() => client.clear('never-raised')).not.toThrow();
});
it('abbreviates long ids in the log line but keeps the short id usable', () => {
const log = vi.spyOn(console, 'error').mockImplementation(() => {});
const client = makeClient();
const longId = 'a-very-long-alarm-identifier-that-wraps';
client.create(longId, 'noisy');
const shortId = client.get(longId)!.shortId;
expect(log).toHaveBeenCalledWith(
`[alarm] ACTIVE ${shortId} (${longId.slice(0, 20)}...) :: noisy`,
);
});
it('accumulates fields across repeats of the same alarm', () => {
const client = makeClient();
const seen = capture(client);
client.create('flap', 'first', { a: 1 });
client.create('flap', 'second', { b: 2 });
expect(seen[1].fields).toEqual({ a: '1', b: '2' });
expect(client.get('flap')?.occurrences).toHaveLength(2);
});
it('caps retained occurrences while still counting every repeat', () => {
const client = makeClient();
const seen = capture(client);
for (let i = 0; i < 50; i++) {
client.create('hot', `occurrence ${i}`, { i });
}
const alarm = client.get('hot')!;
expect(alarm.count).toBe(50);
expect(alarm.occurrences).toHaveLength(20);
expect(alarm.timestamps).toHaveLength(20);
// The window kept is the most recent one, not the oldest.
expect(alarm.occurrences[19].message).toBe('occurrence 49');
// Trimming history must not rewind what the transports are told.
expect(seen[49]).toMatchObject({ repeatCount: 50, isRepeat: true });
});
it('names anonymous handlers by their registration order', () => {
const client = makeClient();
client.addAlertHandler(async () => {
throw new Error('down');
});
expect(() => client.create('boom', 'x')).not.toThrow();
});
it('only logs the drain notice once', () => {
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
log.mockClear();
const client = makeClient();
client.onServerPrepareShutdown();
// A second prepare must not restart the drain.
client.onServerPrepareShutdown();
client.create('a', 'x');
client.create('b', 'y');
const drainLogs = log.mock.calls.filter(
([message]) =>
message === '[alarm] suppressing alarm while draining',
);
expect(drainLogs).toHaveLength(1);
});
it('substitutes placeholders for an empty id and message', () => {
const client = makeClient();
const seen = capture(client);
client.create('', '');
expect(seen[0]).toMatchObject({
id: 'something-bad',
message: 'something bad happened',
});
});
});
describe('AlarmClient known-error rules', () => {
beforeEach(() => {
vi.spyOn(console, 'error').mockImplementation(() => {});
vi.spyOn(console, 'warn').mockImplementation(() => {});
vi.spyOn(console, 'log').mockImplementation(() => {});
});
it('suppresses alerts for a matching rule', () => {
const client = makeClient();
const seen = capture(client);
client.setKnownErrors([
{ match: { id: 'noisy' }, action: { type: 'no-alert' } },
]);
client.create('noisy', 'ignore me');
client.create('noisy', 'ignore me again');
expect(seen).toHaveLength(0);
});
it('leaves alarms with a different id alone', () => {
const client = makeClient();
const seen = capture(client);
client.setKnownErrors([
{ match: { id: 'noisy' }, action: { type: 'no-alert' } },
]);
client.create('real', 'page me');
expect(seen).toHaveLength(1);
});
it('only matches when the message matches too', () => {
const rules = [
{
match: { id: 'timeout', message: 'upstream timed out' },
action: { type: 'no-alert' as const },
},
];
// Separate clients: suppression is recorded on the alarm, so a
// second occurrence of the same id would inherit it.
const matching = makeClient();
const matchingSeen = capture(matching);
matching.setKnownErrors(rules);
matching.create('timeout', 'upstream timed out');
expect(matchingSeen).toHaveLength(0);
const differing = makeClient();
const differingSeen = capture(differing);
differing.setKnownErrors(rules);
differing.create('timeout', 'something else entirely');
expect(differingSeen.map((alert) => alert.message)).toEqual([
'something else entirely',
]);
});
it('only matches when every named field matches', () => {
const client = makeClient();
const seen = capture(client);
client.setKnownErrors([
{
match: { id: 'http', fields: { status: 404 } },
action: { type: 'no-alert' },
},
]);
client.create('http', 'not found', { status: 404 });
client.create('http-2', 'server error', { status: 500 });
expect(seen.map((alert) => alert.id)).toEqual(['http-2']);
});
it('retiers a matching alarm to a lower severity', () => {
const client = makeClient();
const paging = capture(client, 'error');
const chat = capture(client, 'info');
client.setKnownErrors([
{
match: { id: 'flaky' },
action: { type: 'severity', value: 'info' },
},
]);
client.create('flaky', 'transient', {}, 'critical');
expect(paging).toHaveLength(0);
expect(chat).toHaveLength(1);
expect(chat[0].severity).toBe('info');
});
});
+243 -112
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -20,64 +20,65 @@
import { event as pdEvent } from '@pagerduty/pdjs';
import { inspect } from 'node:util';
import { createHash } from 'node:crypto';
import type { IConfig } from '../../types';
import type { IConfig, PagerSeverity, SeverityRule } from '../../types';
import { PuterClient } from '../types';
import {
meetsMinSeverity,
resolveSeverityOverride,
withinMaxSeverity,
} from './severity';
import { createSlackAlertHandler } from './slack';
import type {
Alarm,
AlarmFields,
AlarmOptions,
AlertHandler,
AlertPayload,
KnownErrorRule,
} from './types';
export type {
Alarm,
AlarmFields,
AlarmOptions,
AlertHandler,
AlertPayload,
KnownErrorRule,
} from './types';
export type { PagerSeverity } from '../../types';
// -- Types ------------------------------------------------------------
export interface AlarmFields {
error?: Error;
[key: string]: unknown;
interface RegisteredHandler {
name: string;
/** Lowest severity this transport accepts. */
minSeverity: PagerSeverity;
/** Highest severity this transport accepts. */
maxSeverity: PagerSeverity;
handler: AlertHandler;
}
interface AlarmOccurrence {
message: string;
fields: AlarmFields;
timestamp: number;
}
interface Alarm {
id: string;
shortId: string;
message: string;
fields: AlarmFields;
error?: Error;
started: number;
timestamps: number[];
occurrences: AlarmOccurrence[];
severity?: PagerSeverity;
noAlert?: boolean;
}
export type PagerSeverity = 'critical' | 'error' | 'warning' | 'info';
export interface AlertPayload {
id: string;
message: string;
source: string;
severity: PagerSeverity;
custom?: Record<string, unknown>;
}
type AlertHandler = (alert: AlertPayload) => Promise<void>;
interface KnownErrorRule {
match: {
id: string;
message?: string;
fields?: Record<string, unknown>;
};
action: {
type: 'no-alert' | 'severity';
value?: PagerSeverity;
};
}
/** Severity used when neither the call site nor config picks one. */
const FALLBACK_SEVERITY: PagerSeverity = 'critical';
/**
* How many recent occurrences an alarm keeps. An alarm is never cleared unless
* something calls `clear`, and a hot one repeats for as long as the fault lasts
* — so retaining every occurrence means retaining every message and field set
* it was ever raised with, request bodies and actors included, for the life of
* the process. The last few are what a human reads; the rest is only a count,
* and `count` keeps that.
*/
const OCCURRENCE_HISTORY_LIMIT = 20;
/** Keeps `info` alarms out of the paging system unless config says otherwise. */
const DEFAULT_PAGERDUTY_MIN_SEVERITY: PagerSeverity = 'warning';
/** Slack's ceiling once a pager exists: chat gets what doesn't page. */
const DEFAULT_SLACK_MAX_SEVERITY_WITH_PAGER: PagerSeverity = 'info';
// -- Helpers ----------------------------------------------------------
/**
* Deterministic short identifier derived from an alarm ID.
* Produces a readable 3-word slug like "amber-delta-fox".
* Deterministic short identifier derived from an alarm ID. Produces a readable
* 3-word slug like "amber-delta-fox".
*/
const WORD_POOL = [
'alpha',
@@ -169,13 +170,17 @@ function cleanFields(fields: AlarmFields): Record<string, string> {
// -- AlarmClient ------------------------------------------------------
/**
* Manages system alarms and dispatches alerts to external paging
* services (PagerDuty, or any registered handler).
* Manages system alarms and routes them to alert transports by severity.
*
* Severity is the routing decision: each transport declares the severity window
* it accepts, so `critical` pages on-call while `info` only reaches the chat
* channel. Config can retier or mute any alarm id after the fact — see
* `pager.severityOverrides` in {@link IConfig}.
*/
export class AlarmClient extends PuterClient {
private alarms = new Map<string, Alarm>();
private aliases = new Map<string, Alarm>();
private alertHandlers: AlertHandler[] = [];
private alertHandlers: RegisteredHandler[] = [];
private knownErrors: KnownErrorRule[] = [];
private draining = false;
private drainLogged = false;
@@ -187,39 +192,8 @@ export class AlarmClient extends PuterClient {
// -- Lifecycle ----------------------------------------------------
override async onServerStart(): Promise<void> {
const pagerConf = this.config.pager;
if (!pagerConf?.pagerduty?.enabled) return;
const routingKey = pagerConf.pagerduty.routingKey;
if (!routingKey) {
console.warn(
'[alarm] PagerDuty enabled but no routingKey configured',
);
return;
}
const serverId = this.config.serverId;
this.alertHandlers.push(async (alert) => {
await pdEvent({
data: {
routing_key: routingKey,
event_action: 'trigger',
dedup_key: alert.id,
payload: {
summary: alert.message,
source: alert.source,
severity: alert.severity,
custom_details: {
...alert.custom,
server_id: serverId,
},
},
},
});
});
console.log('[alarm] PagerDuty handler registered');
const paging = this.registerPagerDuty();
this.registerSlack({ paging });
}
override onServerPrepareShutdown(): void {
@@ -228,22 +202,107 @@ export class AlarmClient extends PuterClient {
console.log('[alarm] entering drain mode — suppressing new alarms');
}
/** @returns Whether a working PagerDuty transport was registered. */
private registerPagerDuty(): boolean {
const pagerDutyConf = this.config.pager?.pagerduty;
if (!pagerDutyConf?.enabled) return false;
const routingKey = pagerDutyConf.routingKey;
if (!routingKey) {
console.warn(
'[alarm] PagerDuty enabled but no routingKey configured',
);
return false;
}
const serverId = this.config.serverId;
const minSeverity =
pagerDutyConf.minSeverity ?? DEFAULT_PAGERDUTY_MIN_SEVERITY;
this.addAlertHandler(
async (alert) => {
await pdEvent({
data: {
routing_key: routingKey,
event_action: 'trigger',
dedup_key: alert.dedupKey,
payload: {
summary: alert.message,
source: alert.source,
severity: alert.severity,
custom_details: {
...alert.custom,
server_id: serverId,
},
},
},
});
},
{ name: 'pagerduty', minSeverity },
);
console.log(
`[alarm] PagerDuty handler registered (min severity: ${minSeverity})`,
);
return true;
}
private registerSlack({ paging }: { paging: boolean }): void {
const slackConf = this.config.pager?.slack;
if (!slackConf?.enabled) return;
if (!slackConf.webhookUrl) {
console.warn('[alarm] Slack enabled but no webhookUrl configured');
return;
}
const minSeverity = slackConf.minSeverity ?? 'info';
// With a pager taking everything from `warning` up, chat is where the
// rest is recorded — reposting the paging tiers there only trains
// people to skim the channel. Without one, Slack is the only place an
// alarm can land, so it takes all of them.
const maxSeverity =
slackConf.maxSeverity ??
(paging ? DEFAULT_SLACK_MAX_SEVERITY_WITH_PAGER : 'critical');
this.addAlertHandler(
createSlackAlertHandler(slackConf, {
serverId: this.config.serverId,
}),
{ name: 'slack', minSeverity, maxSeverity },
);
console.log(
`[alarm] Slack handler registered (severity ${minSeverity}..${maxSeverity})`,
);
}
// -- Public API ---------------------------------------------------
/**
* Create or update an alarm. If the alarm ID already exists, the
* occurrence count is incremented and a repeat alert is dispatched.
* Create or update an alarm. If the alarm ID already exists, the occurrence
* count is incremented and a repeat alert is dispatched.
*
* `severity` is the PagerDuty severity for this alarm; omit for the
* default 'critical'. Use 'info' / 'warning' for expected-but-worth-
* tracking signals (e.g. a user hitting a rate limit) so they record
* and de-dupe like any other alarm but don't page on-call.
* `severity` decides where the alarm lands:
*
* Critical — a real outage; pages on-call. Reserve it for unhandled server
* errors. error — pages on-call as well; prefer `critical` or `warning`.
* warning — worth a look soon, but nobody gets woken up. info — a record in
* the chat channel; never pages.
*
* Omit it to take `pager.defaultSeverity` (itself defaulting to
* 'critical'). Operators can retier or mute any alarm id from config
* afterwards, so the value here is the starting point, not the last word.
*
* Pass `{ dedup: true }` when repeats of this id are one recurring fault
* that should collapse into a single incident — see {@link AlarmOptions}.
*/
create(
id: string,
message: string,
fields: AlarmFields = {},
severity?: PagerSeverity,
opts: AlarmOptions = {},
): void {
if (this.draining) {
if (!this.drainLogged) {
@@ -267,8 +326,12 @@ export class AlarmClient extends PuterClient {
message,
fields,
severity,
dedup: opts.dedup,
started: Date.now(),
timestamps: [Date.now()],
// `recordOccurrence` below stamps the first occurrence; seeding one
// here too would report every alarm as one occurrence ahead.
count: 0,
timestamps: [],
occurrences: [],
};
if (fields.error) alarm.error = fields.error;
@@ -295,16 +358,28 @@ export class AlarmClient extends PuterClient {
}
/**
* Register an additional alert handler. Handlers are called for
* every alarm that isn't suppressed by a known-error rule.
* Register an additional alert handler. Handlers are called for every alarm
* that isn't suppressed by a known-error rule or muted by config, and whose
* severity falls inside the handler's own `minSeverity`..`maxSeverity`
* window (default: everything).
*/
addAlertHandler(handler: AlertHandler): void {
this.alertHandlers.push(handler);
addAlertHandler(
handler: AlertHandler,
opts: {
name?: string;
minSeverity?: PagerSeverity;
maxSeverity?: PagerSeverity;
} = {},
): void {
this.alertHandlers.push({
name: opts.name ?? `handler-${this.alertHandlers.length}`,
minSeverity: opts.minSeverity ?? 'info',
maxSeverity: opts.maxSeverity ?? 'critical',
handler,
});
}
/**
* Add rules that can suppress or adjust severity of known errors.
*/
/** Add rules that can suppress or adjust severity of known errors. */
setKnownErrors(rules: KnownErrorRule[]): void {
this.knownErrors = rules;
}
@@ -316,16 +391,25 @@ export class AlarmClient extends PuterClient {
message: string,
fields: AlarmFields,
): void {
const now = Date.now();
alarm.message = message;
alarm.fields = { ...alarm.fields, ...fields };
alarm.timestamps.push(Date.now());
alarm.count++;
if (fields.error) alarm.error = fields.error;
alarm.occurrences.push({
message,
fields,
timestamp: Date.now(),
});
alarm.timestamps.push(now);
alarm.occurrences.push({ message, fields, timestamp: now });
if (alarm.timestamps.length > OCCURRENCE_HISTORY_LIMIT) {
alarm.timestamps.splice(
0,
alarm.timestamps.length - OCCURRENCE_HISTORY_LIMIT,
);
alarm.occurrences.splice(
0,
alarm.occurrences.length - OCCURRENCE_HISTORY_LIMIT,
);
}
}
private applyKnownErrors(alarm: Alarm): void {
@@ -373,7 +457,7 @@ export class AlarmClient extends PuterClient {
this.applyKnownErrors(alarm);
console.warn(
`[alarm] REPEAT ${displayId(alarm)} :: ${alarm.message} (${alarm.timestamps.length})`,
`[alarm] REPEAT ${displayId(alarm)} :: ${alarm.message} (${alarm.count})`,
);
if (alarm.noAlert) return;
@@ -381,25 +465,72 @@ export class AlarmClient extends PuterClient {
this.dispatchAlert(alarm);
}
/**
* Call-site severity, then any known-error rule (both already on the
* alarm), then the config override — so an operator always has the last
* word over what the code asked for.
*/
private resolveSeverity(alarm: Alarm): SeverityRule {
const base =
alarm.severity ??
this.config.pager?.defaultSeverity ??
FALLBACK_SEVERITY;
return (
resolveSeverityOverride(
alarm.id,
this.config.pager?.severityOverrides,
) ?? base
);
}
private dispatchAlert(alarm: Alarm): void {
const severity = alarm.severity ?? 'critical';
const resolved = this.resolveSeverity(alarm);
if (resolved === 'mute') {
if (!alarm.muteLogged) {
alarm.muteLogged = true;
console.log(`[alarm] MUTED by config ${displayId(alarm)}`);
}
return;
}
alarm.severity = resolved;
const fieldsClean = cleanFields(alarm.fields);
const repeatCount = alarm.count;
const id = alarm.id || 'something-bad';
const payload: AlertPayload = {
id: alarm.id || 'something-bad',
id,
// A de-duplicating transport should fold repeats of a recurring
// fault into one incident, but everything else is a fresh event
// each time it fires — the start time keeps occurrence numbering
// from colliding with an incident left open by an earlier boot.
dedupKey: alarm.dedup
? id
: `${id}#${alarm.started}.${repeatCount}`,
shortId: alarm.shortId,
message: alarm.message || alarm.id || 'something bad happened',
source: 'alarm',
severity,
severity: resolved,
fields: fieldsClean,
trace: alarm.error?.stack,
repeatCount,
isRepeat: repeatCount > 1,
custom: {
fields: fieldsClean,
trace: alarm.error?.stack,
repeat_count: alarm.timestamps.length,
repeat_count: repeatCount,
},
};
for (const handler of this.alertHandlers) {
for (const { name, minSeverity, maxSeverity, handler } of this
.alertHandlers) {
if (!meetsMinSeverity(resolved, minSeverity)) continue;
if (!withinMaxSeverity(resolved, maxSeverity)) continue;
handler(payload).catch((err) => {
console.error(`[alarm] alert handler failed: ${err?.message}`);
console.error(
`[alarm] ${name} alert handler failed: ${err?.message}`,
);
});
}
}
@@ -0,0 +1,76 @@
import { describe, expect, it, vi } from 'vitest';
import { meetsMinSeverity, resolveSeverityOverride } from './severity';
describe('meetsMinSeverity', () => {
it('accepts everything at or above the floor', () => {
expect(meetsMinSeverity('critical', 'warning')).toBe(true);
expect(meetsMinSeverity('error', 'warning')).toBe(true);
expect(meetsMinSeverity('warning', 'warning')).toBe(true);
expect(meetsMinSeverity('info', 'warning')).toBe(false);
});
it('lets an info floor take every severity', () => {
for (const severity of [
'critical',
'error',
'warning',
'info',
] as const) {
expect(meetsMinSeverity(severity, 'info')).toBe(true);
}
});
});
describe('resolveSeverityOverride', () => {
it('returns undefined without overrides or on no match', () => {
expect(resolveSeverityOverride('a:b', undefined)).toBeUndefined();
expect(
resolveSeverityOverride('a:b', { 'c:*': 'info' }),
).toBeUndefined();
});
it('matches an exact id', () => {
expect(
resolveSeverityOverride('cronMonitor:lowSignupRate', {
'cronMonitor:lowSignupRate': 'info',
}),
).toBe('info');
});
it('matches a prefix pattern', () => {
expect(
resolveSeverityOverride('cronMonitor:high_aiLogEntries', {
'cronMonitor:*': 'warning',
}),
).toBe('warning');
});
it('prefers the exact id over a prefix pattern', () => {
expect(
resolveSeverityOverride('cronMonitor:lowSignupRate', {
'cronMonitor:*': 'warning',
'cronMonitor:lowSignupRate': 'mute',
}),
).toBe('mute');
});
it('prefers the longest matching prefix', () => {
expect(
resolveSeverityOverride('abuse:card-verification:setup-failed', {
'abuse:*': 'info',
'abuse:card-verification:*': 'mute',
}),
).toBe('mute');
});
it('drops an unrecognized rule rather than guessing', () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
expect(
resolveSeverityOverride('a:b', {
'a:b': 'silent' as unknown as 'mute',
}),
).toBeUndefined();
expect(warn).toHaveBeenCalled();
warn.mockRestore();
});
});
+94
View File
@@ -0,0 +1,94 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { PagerSeverity, SeverityRule } from '../../types';
/** Ascending urgency. A transport takes everything at or above its floor. */
const RANK: Record<PagerSeverity, number> = {
info: 0,
warning: 1,
error: 2,
critical: 3,
};
const SEVERITIES = Object.keys(RANK) as PagerSeverity[];
export function isPagerSeverity(value: unknown): value is PagerSeverity {
return typeof value === 'string' && value in RANK;
}
/** True when `severity` is urgent enough for a transport whose floor is `min`. */
export function meetsMinSeverity(
severity: PagerSeverity,
min: PagerSeverity,
): boolean {
return RANK[severity] >= RANK[min];
}
/**
* True when `severity` is quiet enough for a transport whose ceiling is `max`.
* A ceiling is what keeps a chat transport from repeating everything the pager
* already delivered.
*/
export function withinMaxSeverity(
severity: PagerSeverity,
max: PagerSeverity,
): boolean {
return RANK[severity] <= RANK[max];
}
/**
* Look up the operator override for an alarm id. Exact ids win over prefix
* patterns (`cronMonitor:*`); among patterns the longest prefix wins, so a
* specific rule can carve an exception out of a broad one.
*/
export function resolveSeverityOverride(
id: string,
overrides: Record<string, SeverityRule> | undefined,
): SeverityRule | undefined {
if (!overrides) return undefined;
const exact = overrides[id];
if (exact !== undefined) return validRule(id, exact);
let bestLength = -1;
let best: SeverityRule | undefined;
for (const [pattern, rule] of Object.entries(overrides)) {
if (!pattern.endsWith('*')) continue;
const prefix = pattern.slice(0, -1);
if (!id.startsWith(prefix)) continue;
if (prefix.length <= bestLength) continue;
bestLength = prefix.length;
best = rule;
}
return best === undefined ? undefined : validRule(id, best);
}
/**
* A typo in the override map would otherwise silently mute or escalate an
* alarm, so an unrecognized value is dropped with a warning instead.
*/
function validRule(id: string, rule: SeverityRule): SeverityRule | undefined {
if (rule === 'mute' || isPagerSeverity(rule)) return rule;
console.warn(
`[alarm] ignoring invalid severity override "${rule}" for ${id} ` +
`(expected mute or one of ${SEVERITIES.join(', ')})`,
);
return undefined;
}
+206
View File
@@ -0,0 +1,206 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { buildSlackMessage, createSlackAlertHandler } from './slack';
import type { AlertPayload } from './types';
const alert = (over: Partial<AlertPayload> = {}): AlertPayload => ({
id: 'cronMonitor:high_aiLogEntries',
dedupKey: 'cronMonitor:high_aiLogEntries#1.1',
shortId: 'amber-delta-fox',
message: 'High AI log entries: 1200 in the last 10 minutes',
source: 'alarm',
severity: 'warning',
fields: { count: '1200', threshold: '1000' },
repeatCount: 1,
isRepeat: false,
...over,
});
describe('buildSlackMessage', () => {
it('renders severity, message and fields', () => {
const msg = buildSlackMessage(alert(), {
channel: '#alerts',
username: 'puter-alarms',
serverId: 'oregon',
});
expect(msg.text).toContain('[WARNING]');
expect(msg.text).toContain('High AI log entries');
expect(msg.channel).toBe('#alerts');
expect(msg.username).toBe('puter-alarms');
expect(msg.attachments[0].fields).toEqual([
{ title: 'count', value: '1200', short: true },
{ title: 'threshold', value: '1000', short: true },
]);
expect(msg.attachments[0].footer).toBe(
'amber-delta-fox • cronMonitor:high_aiLogEntries • oregon',
);
});
it('marks repeats with an occurrence count', () => {
const msg = buildSlackMessage(
alert({ repeatCount: 7, isRepeat: true }),
);
expect(msg.text).toContain('(x7)');
});
it('colours each severity differently', () => {
const colors = (['critical', 'error', 'warning', 'info'] as const).map(
(severity) =>
buildSlackMessage(alert({ severity })).attachments[0].color,
);
expect(new Set(colors).size).toBe(4);
});
it('puts the stack in a code block and keeps it out of the fields', () => {
const msg = buildSlackMessage(
alert({
fields: { error: 'Error: boom', path: '/api/x' },
trace: 'Error: boom\n at handler',
}),
);
expect(msg.attachments[0].text).toBe(
'```Error: boom\n at handler```',
);
expect(msg.attachments[0].fields.map((f) => f.title)).toEqual(['path']);
});
it('truncates long values and caps the field count', () => {
const fields: Record<string, string> = { long: 'x'.repeat(1000) };
for (let i = 0; i < 30; i++) fields[`f${i}`] = String(i);
const msg = buildSlackMessage(alert({ fields }));
expect(msg.attachments[0].fields.length).toBe(12);
expect(msg.attachments[0].fields[0].value).toHaveLength(400);
expect(msg.attachments[0].fields[0].value.endsWith('…')).toBe(true);
});
it('omits channel and username when unset', () => {
const msg = buildSlackMessage(alert());
expect(msg.channel).toBeUndefined();
expect(msg.username).toBeUndefined();
});
});
describe('createSlackAlertHandler', () => {
let fetchMock: ReturnType<typeof vi.fn>;
beforeEach(() => {
vi.useFakeTimers();
fetchMock = vi.fn(async () => ({ ok: true, status: 200 }));
vi.stubGlobal('fetch', fetchMock);
});
afterEach(() => {
vi.useRealTimers();
vi.unstubAllGlobals();
});
it('posts the payload to the webhook', async () => {
const handler = createSlackAlertHandler({
webhookUrl: 'https://hooks.example/abc',
channel: '#alerts',
});
await handler(alert());
expect(fetchMock).toHaveBeenCalledTimes(1);
const [url, init] = fetchMock.mock.calls[0];
expect(url).toBe('https://hooks.example/abc');
expect(init.method).toBe('POST');
expect(JSON.parse(init.body).channel).toBe('#alerts');
});
it('throttles repeats of the same alarm and lets other ids through', async () => {
const handler = createSlackAlertHandler({
webhookUrl: 'https://hooks.example/abc',
repeatThrottleMs: 60_000,
});
await handler(alert());
await handler(alert({ repeatCount: 2, isRepeat: true }));
expect(fetchMock).toHaveBeenCalledTimes(1);
await handler(alert({ id: 'other:alarm' }));
expect(fetchMock).toHaveBeenCalledTimes(2);
vi.advanceTimersByTime(60_000);
await handler(alert({ repeatCount: 3, isRepeat: true }));
expect(fetchMock).toHaveBeenCalledTimes(3);
});
it('posts every occurrence when throttling is disabled', async () => {
const handler = createSlackAlertHandler({
webhookUrl: 'https://hooks.example/abc',
repeatThrottleMs: 0,
});
await handler(alert());
await handler(alert({ repeatCount: 2, isRepeat: true }));
expect(fetchMock).toHaveBeenCalledTimes(2);
});
it('does not let a failed post consume the throttle slot', async () => {
fetchMock.mockResolvedValueOnce({ ok: false, status: 500 });
const handler = createSlackAlertHandler({
webhookUrl: 'https://hooks.example/abc',
repeatThrottleMs: 60_000,
});
await expect(handler(alert())).rejects.toThrow('500');
await handler(alert());
expect(fetchMock).toHaveBeenCalledTimes(2);
});
it('releases the throttle slot when the post itself errors', async () => {
fetchMock.mockRejectedValueOnce(new Error('socket hang up'));
const handler = createSlackAlertHandler({
webhookUrl: 'https://hooks.example/abc',
repeatThrottleMs: 60_000,
});
await expect(handler(alert())).rejects.toThrow('socket hang up');
await handler(alert());
expect(fetchMock).toHaveBeenCalledTimes(2);
});
it('evicts throttle entries that have aged out once the map fills', async () => {
const throttleMs = 60_000;
const handler = createSlackAlertHandler({
webhookUrl: 'https://hooks.example/abc',
repeatThrottleMs: throttleMs,
});
// Fill to the high-water mark, then age every entry past the window.
for (let i = 0; i < 5000; i++) {
await handler(alert({ id: `aged:${i}` }));
}
vi.advanceTimersByTime(throttleMs);
await handler(alert({ id: 'fresh' }));
expect(fetchMock).toHaveBeenCalledTimes(5001);
// The aged ids were pruned, so they post again immediately.
await handler(alert({ id: 'aged:0' }));
expect(fetchMock).toHaveBeenCalledTimes(5002);
});
it('drops the oldest entry when the map is full of live ones', async () => {
const handler = createSlackAlertHandler({
webhookUrl: 'https://hooks.example/abc',
repeatThrottleMs: 60_000,
});
for (let i = 0; i < 5000; i++) {
await handler(alert({ id: `live:${i}` }));
}
// Nothing has aged out, so the oldest id makes room for the new one.
await handler(alert({ id: 'newcomer' }));
expect(fetchMock).toHaveBeenCalledTimes(5001);
await handler(alert({ id: 'live:0' }));
expect(fetchMock).toHaveBeenCalledTimes(5002);
// The newcomer is still throttled.
await handler(alert({ id: 'newcomer' }));
expect(fetchMock).toHaveBeenCalledTimes(5002);
});
});
+178
View File
@@ -0,0 +1,178 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { ISlackAlertConfig, PagerSeverity } from '../../types';
import type { AlertHandler, AlertPayload } from './types';
const REQUEST_TIMEOUT_MS = 5000;
export const DEFAULT_REPEAT_THROTTLE_MS = 15 * 60 * 1000;
/** Beyond this many tracked alarm ids, throttle state is pruned. */
const MAX_THROTTLE_ENTRIES = 5000;
const MAX_FIELDS = 12;
const MAX_FIELD_VALUE = 400;
const MAX_TRACE = 1500;
/** Values under this length sit two-per-row in the Slack attachment. */
const SHORT_FIELD_LENGTH = 40;
const STYLE: Record<PagerSeverity, { emoji: string; color: string }> = {
critical: { emoji: ':rotating_light:', color: '#d64545' },
error: { emoji: ':red_circle:', color: '#e08d4c' },
warning: { emoji: ':warning:', color: '#e0b84c' },
info: { emoji: ':information_source:', color: '#4c8de0' },
};
interface SlackField {
title: string;
value: string;
short: boolean;
}
export interface SlackMessage {
text: string;
channel?: string;
username?: string;
icon_emoji?: string;
attachments: Array<{
color: string;
fallback: string;
fields: SlackField[];
footer: string;
mrkdwn_in: string[];
text?: string;
}>;
}
function truncate(value: string, max: number): string {
return value.length <= max ? value : `${value.slice(0, max - 1)}…`;
}
/**
* Render an alert as an incoming-webhook payload: severity-coloured attachment,
* the alarm's fields as a table, and the stack (when there is one) as a code
* block. Split out from the handler so the formatting is testable on its own.
*/
export function buildSlackMessage(
alert: AlertPayload,
opts: { channel?: string; username?: string; serverId?: string } = {},
): SlackMessage {
const style = STYLE[alert.severity] ?? STYLE.info;
const repeat = alert.isRepeat ? ` (x${alert.repeatCount})` : '';
const headline = `${style.emoji} *[${alert.severity.toUpperCase()}]* ${alert.message}${repeat}`;
const fields: SlackField[] = [];
for (const [key, value] of Object.entries(alert.fields)) {
if (fields.length >= MAX_FIELDS) break;
// `error` is already rendered as the trace block below.
if (key === 'error') continue;
const rendered = truncate(value, MAX_FIELD_VALUE);
fields.push({
title: key,
value: rendered,
short: rendered.length <= SHORT_FIELD_LENGTH,
});
}
const footerParts = [alert.shortId, alert.id];
if (opts.serverId) footerParts.push(opts.serverId);
return {
text: headline,
...(opts.channel ? { channel: opts.channel } : {}),
...(opts.username ? { username: opts.username } : {}),
attachments: [
{
color: style.color,
fallback: `[${alert.severity}] ${alert.message}`,
fields,
footer: footerParts.join(' • '),
mrkdwn_in: ['text'],
...(alert.trace
? { text: '```' + truncate(alert.trace, MAX_TRACE) + '```' }
: {}),
},
],
};
}
/**
* Post alerts to a Slack incoming webhook. Repeats of the same alarm id are
* throttled so a hot loop doesn't flood the channel — the occurrence count on
* the next post that gets through tells the reader what they missed.
*/
export function createSlackAlertHandler(
conf: ISlackAlertConfig,
opts: { serverId?: string } = {},
): AlertHandler {
const webhookUrl = conf.webhookUrl as string;
const throttleMs = conf.repeatThrottleMs ?? DEFAULT_REPEAT_THROTTLE_MS;
const lastPosted = new Map<string, number>();
const shouldPost = (alert: AlertPayload): boolean => {
if (throttleMs <= 0) return true;
const now = Date.now();
const previous = lastPosted.get(alert.id);
if (previous !== undefined && now - previous < throttleMs) return false;
if (lastPosted.size >= MAX_THROTTLE_ENTRIES) {
for (const [id, at] of lastPosted) {
if (now - at >= throttleMs) lastPosted.delete(id);
}
// Still full of live entries — drop the oldest to stay bounded.
if (lastPosted.size >= MAX_THROTTLE_ENTRIES) {
const oldest = lastPosted.keys().next().value;
if (oldest !== undefined) lastPosted.delete(oldest);
}
}
lastPosted.set(alert.id, now);
return true;
};
return async (alert) => {
if (!shouldPost(alert)) return;
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
try {
const res = await fetch(webhookUrl, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(
buildSlackMessage(alert, {
channel: conf.channel,
username: conf.username,
serverId: opts.serverId,
}),
),
signal: controller.signal,
});
if (!res.ok) {
// A rejected post shouldn't leave the id marked as delivered.
lastPosted.delete(alert.id);
throw new Error(`Slack webhook returned ${res.status}`);
}
} catch (err) {
lastPosted.delete(alert.id);
throw err;
} finally {
clearTimeout(timer);
}
};
}
+105
View File
@@ -0,0 +1,105 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { PagerSeverity } from '../../types';
export interface AlarmFields {
error?: Error;
[key: string]: unknown;
}
export interface AlarmOccurrence {
message: string;
fields: AlarmFields;
timestamp: number;
}
export interface AlarmOptions {
/**
* Collapse repeats of this alarm id into a single incident on transports
* that de-duplicate (PagerDuty). Only right when the id already pins down
* one recurring fault — an uncaught error on a route, say — so that N
* occurrences really are one thing to fix. Off by default: every other
* alarm raises its own incident per occurrence.
*/
dedup?: boolean;
}
export interface Alarm extends AlarmOptions {
id: string;
shortId: string;
message: string;
fields: AlarmFields;
error?: Error;
started: number;
/**
* Every occurrence ever counted, including those aged out of the two lists
* below.
*/
count: number;
/**
* Timestamps of the most recent occurrences only — see
* `OCCURRENCE_HISTORY_LIMIT`.
*/
timestamps: number[];
/** The most recent occurrences only — see `OCCURRENCE_HISTORY_LIMIT`. */
occurrences: AlarmOccurrence[];
severity?: PagerSeverity;
noAlert?: boolean;
/** Set once a config mute has been logged, so it's reported only once. */
muteLogged?: boolean;
}
export interface AlertPayload {
id: string;
/**
* Key a de-duplicating transport groups by. Equal to `id` for alarms raised
* with `dedup`, and unique per occurrence for everything else.
*/
dedupKey: string;
/** Readable slug for the same alarm, for humans quoting it back. */
shortId: string;
message: string;
source: string;
severity: PagerSeverity;
/** Field values rendered as strings, ready to display. */
fields: Record<string, string>;
/** Stack of the attached error, when the alarm carried one. */
trace?: string;
/** Total occurrences of this alarm so far, including this one. */
repeatCount: number;
/** False for the first occurrence of an alarm id. */
isRepeat: boolean;
/** PagerDuty `custom_details` payload. */
custom?: Record<string, unknown>;
}
export type AlertHandler = (alert: AlertPayload) => Promise<void>;
export interface KnownErrorRule {
match: {
id: string;
message?: string;
fields?: Record<string, unknown>;
};
action: {
type: 'no-alert' | 'severity';
value?: PagerSeverity;
};
}
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -0,0 +1,247 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { describe, expect, it } from 'vitest';
import type { IConfig } from '../../types';
import { AbstractDatabaseClient, type WriteResult } from './DatabaseClient';
import { DatabaseClientFactory } from './index.js';
import { MySQLDatabaseClient } from './MySQLDatabaseClient.js';
import { PostgresDatabaseClient } from './PostgresDatabaseClient.js';
import { SqliteDatabaseClient } from './SqliteDatabaseClient.js';
const config = (engine: string): IConfig =>
({ port: 0, extensions: [], database: { engine } }) as IConfig;
const sqlite = () => new SqliteDatabaseClient(config('sqlite'));
const mysql = () => new MySQLDatabaseClient(config('mysql'));
const postgres = () => new PostgresDatabaseClient(config('postgres'));
/**
* Minimal concrete client backed by fixture rows — exercises the shared helpers
* on `AbstractDatabaseClient` that the real engines inherit.
*/
class FixtureDatabaseClient extends AbstractDatabaseClient {
override readonly engineName = 'fixture';
readonly reads: string[] = [];
readonly preads: string[] = [];
readonly writes: { query: string; params: unknown[] }[] = [];
constructor(
private readonly replicaRows: Record<string, unknown>[] | Error,
private readonly primaryRows: Record<string, unknown>[] = [],
) {
super(config('fixture'));
}
override async read(query: string): Promise<Record<string, unknown>[]> {
this.reads.push(query);
if (this.replicaRows instanceof Error) throw this.replicaRows;
return this.replicaRows;
}
override async pread(query: string): Promise<Record<string, unknown>[]> {
this.preads.push(query);
return this.primaryRows;
}
override async write(
query: string,
params: unknown[] = [],
): Promise<WriteResult> {
this.writes.push({ query, params });
return { insertId: 1, affectedRows: 1, anyRowsAffected: true };
}
}
describe('DatabaseClientFactory', () => {
it('picks sqlite when no engine is configured', () => {
expect(
new DatabaseClientFactory({ port: 0, extensions: [] } as IConfig),
).toBeInstanceOf(SqliteDatabaseClient);
});
it('refuses an unknown engine name', () => {
expect(() => new DatabaseClientFactory(config('cassandra'))).toThrow(
'Unknown database engine: cassandra',
);
});
});
describe('AbstractDatabaseClient — unimplemented surface', () => {
const bare = () => new AbstractDatabaseClient(config('none'));
it('refuses to run queries until a subclass implements them', async () => {
await expect(bare().read('SELECT 1')).rejects.toThrow(
'DatabaseClient.read() not implemented',
);
await expect(bare().pread('SELECT 1')).rejects.toThrow(
'DatabaseClient.pread() not implemented',
);
await expect(bare().write('DELETE FROM x')).rejects.toThrow(
'DatabaseClient.write() not implemented',
);
await expect(bare().batchWrite([])).rejects.toThrow(
'DatabaseClient.batchWrite() not implemented',
);
});
});
describe('AbstractDatabaseClient — replica-aware reads', () => {
it('returns replica rows without waiting on the primary', async () => {
const client = new FixtureDatabaseClient([{ id: 1 }], [{ id: 2 }]);
await expect(client.tryHardRead('SELECT 1')).resolves.toEqual([
{ id: 1 },
]);
});
it('falls back to the primary when the replica has not caught up', async () => {
const client = new FixtureDatabaseClient([], [{ id: 2 }]);
await expect(client.tryHardRead('SELECT 1')).resolves.toEqual([
{ id: 2 },
]);
});
it('falls back to the primary when the replica read throws', async () => {
const client = new FixtureDatabaseClient(
new Error('replica unavailable'),
[{ id: 3 }],
);
await expect(client.tryHardRead('SELECT 1')).resolves.toEqual([
{ id: 3 },
]);
});
it('names the failing query when a required read finds nothing', async () => {
const client = new FixtureDatabaseClient([], []);
await expect(
client.requireRead('SELECT * FROM `user` WHERE `id` = ?'),
).rejects.toThrow(
'required read returned no rows: SELECT * FROM `user` WHERE `id` = ?',
);
});
});
describe('AbstractDatabaseClient — SQL generation', () => {
it('generates an INSERT with quoted columns and placeholders', async () => {
const client = new FixtureDatabaseClient([]);
await client.insert('user', { username: 'ada', email: null });
expect(client.writes).toEqual([
{
query: 'INSERT INTO `user` (`username`, `email`) VALUES (?, ?)',
params: ['ada', null],
},
]);
});
it('quotes each dotted segment and escapes embedded backticks', () => {
const client = sqlite();
expect(client.quoteIdentifier('user')).toBe('`user`');
expect(client.quoteIdentifier('db.user.id')).toBe('`db`.`user`.`id`');
expect(client.quoteIdentifier('user.*')).toBe('`user`.*');
expect(client.quoteIdentifier('we`ird')).toBe('`we``ird`');
});
it('quotes postgres identifiers with double quotes', () => {
const client = postgres();
expect(client.quoteIdentifier('db.user')).toBe('"db"."user"');
expect(client.quoteIdentifier('we"ird')).toBe('"we""ird"');
expect(client.quoteIdentifier('user.*')).toBe('"user".*');
});
it('renders booleans the way each engine expects', () => {
expect(sqlite().booleanLiteral(true)).toBe('1');
expect(sqlite().booleanValue(false)).toBe(0);
expect(postgres().booleanLiteral(true)).toBe('TRUE');
expect(postgres().booleanLiteral(false)).toBe('FALSE');
expect(postgres().booleanValue(true)).toBe(true);
});
it('picks the engine-specific ignore-conflict syntax', () => {
expect(sqlite().insertIgnoreInto('kv')).toBe(
'INSERT OR IGNORE INTO `kv`',
);
expect(sqlite().insertIgnoreSuffix()).toBe('');
expect(postgres().insertIgnoreInto('kv')).toBe('INSERT INTO "kv"');
expect(postgres().insertIgnoreSuffix()).toBe(' ON CONFLICT DO NOTHING');
expect(mysql().insertIgnoreInto('kv')).toBe('INSERT IGNORE INTO `kv`');
expect(mysql().insertIgnoreSuffix()).toBe('');
});
it('builds an upsert clause per engine', () => {
expect(mysql().upsertClause(['user_id'], ['value', 'dt'])).toBe(
'ON DUPLICATE KEY UPDATE `value` = ?, `dt` = ?',
);
expect(sqlite().upsertClause(['user_id', 'app'], ['value'])).toBe(
'ON CONFLICT(`user_id`, `app`) DO UPDATE SET `value` = ?',
);
expect(postgres().upsertClause(['user_id'], ['value'])).toBe(
'ON CONFLICT("user_id") DO UPDATE SET "value" = ?',
);
});
it('rejects an upsert with nothing to update', () => {
expect(() => sqlite().upsertClause(['user_id'], [])).toThrow(
'upsertClause requires at least one update column',
);
});
it('extracts JSON text using each engine dialect', () => {
expect(sqlite().jsonTextExtract('`metadata`', ['a', 'b'])).toBe(
"json_extract(`metadata`, '$.a.b')",
);
expect(mysql().jsonTextExtract('`metadata`', ['a'])).toBe(
"JSON_UNQUOTE(JSON_EXTRACT(`metadata`, '$.a'))",
);
expect(postgres().jsonTextExtract('"metadata"', ['a', 'b'])).toBe(
`"metadata" #>> ARRAY['a', 'b']`,
);
});
it('escapes quotes inside JSON path segments', () => {
expect(sqlite().jsonTextExtract('`m`', ["it's"])).toBe(
"json_extract(`m`, '$.it''s')",
);
});
it('coalesces expressions and rejects an empty list', () => {
expect(sqlite().nullCoalesce('`a`', '`b`', '0')).toBe(
'COALESCE(`a`, `b`, 0)',
);
expect(() => sqlite().nullCoalesce()).toThrow(
'nullCoalesce requires at least one expression',
);
});
it('only postgres needs a RETURNING clause to learn the insert id', () => {
expect(postgres().returningIdClause()).toBe(' RETURNING id');
expect(sqlite().returningIdClause()).toBe('');
expect(mysql().returningIdClause()).toBe('');
});
it('falls back to `otherwise` for an engine with no explicit choice', () => {
const client = new FixtureDatabaseClient([]);
expect(
client.case({ sqlite: 'a', mysql: 'b', otherwise: 'fallback' }),
).toBe('fallback');
expect(client.case({ fixture: 'exact', otherwise: 'fallback' })).toBe(
'exact',
);
});
});
+19 -24
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -37,8 +37,8 @@ type SqlJsonPath = readonly [string, ...string[]];
* Base database client. Subclasses must override every method that throws here.
*
* Do not instantiate directly — use the factory exported from
* `clients/database/index.ts` which picks the right implementation
* based on `config.database.engine`.
* `clients/database/index.ts` which picks the right implementation based on
* `config.database.engine`.
*/
export class AbstractDatabaseClient extends PuterClient {
/** Short name used by `case()` to pick engine-specific values. */
@@ -52,9 +52,7 @@ export class AbstractDatabaseClient extends PuterClient {
// Abstract interface — subclasses MUST override
// ------------------------------------------------------------------
/**
* Execute a read query. Returns an array of row objects.
*/
/** Execute a read query. Returns an array of row objects. */
async read(
_query: string,
_params: unknown[] = [],
@@ -63,9 +61,9 @@ export class AbstractDatabaseClient extends PuterClient {
}
/**
* Read that prefers the primary database (useful when read-replicas
* may have replication lag). In single-node setups this is identical
* to `read()`.
* Read that prefers the primary database (useful when read-replicas may
* have replication lag). In single-node setups this is identical to
* `read()`.
*/
async pread(
_query: string,
@@ -74,16 +72,12 @@ export class AbstractDatabaseClient extends PuterClient {
throw new Error('DatabaseClient.pread() not implemented');
}
/**
* Execute a write query (INSERT / UPDATE / DELETE).
*/
/** Execute a write query (INSERT / UPDATE / DELETE). */
async write(_query: string, _params: unknown[] = []): Promise<WriteResult> {
throw new Error('DatabaseClient.write() not implemented');
}
/**
* Execute multiple write statements in a single transaction.
*/
/** Execute multiple write statements in a single transaction. */
async batchWrite(_entries: BatchEntry[]): Promise<void> {
throw new Error('DatabaseClient.batchWrite() not implemented');
}
@@ -111,9 +105,9 @@ export class AbstractDatabaseClient extends PuterClient {
}
/**
* Like `read()` but falls back to the primary when read-replicas are
* in use. Subclasses may override with replica-aware logic; the
* default delegates to `pread()`.
* Like `read()` but falls back to the primary when read-replicas are in
* use. Subclasses may override with replica-aware logic; the default
* delegates to `pread()`.
*/
async tryHardRead(
query: string,
@@ -133,9 +127,7 @@ export class AbstractDatabaseClient extends PuterClient {
return primary;
}
/**
* Like `tryHardRead()` but throws when the result set is empty.
*/
/** Like `tryHardRead()` but throws when the result set is empty. */
async requireRead(
query: string,
params: unknown[] = [],
@@ -151,9 +143,12 @@ export class AbstractDatabaseClient extends PuterClient {
* Return the value from `choices` that matches the current engine.
*
* Usage:
* ```
* db.case({ sqlite: "datetime('now')", mysql: 'NOW()', otherwise: 'NOW()' })
* ```
*
* db.case({
* sqlite: "datetime('now')",
* mysql: 'NOW()',
* otherwise: 'NOW()',
* });
*
* If the engine name isn't present in `choices`, falls back to
* `choices.otherwise`.
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -17,13 +17,95 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { describe, expect, it, vi } from 'vitest';
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import type { IConfig } from '../../types';
import {
MySQLDatabaseClient,
compareMigrationFilenames,
} from './MySQLDatabaseClient.js';
interface FakeConnection {
query: ReturnType<typeof vi.fn>;
execute: ReturnType<typeof vi.fn>;
beginTransaction: ReturnType<typeof vi.fn>;
commit: ReturnType<typeof vi.fn>;
rollback: ReturnType<typeof vi.fn>;
release: ReturnType<typeof vi.fn>;
}
interface FakePool {
poolConfig: Record<string, unknown>;
connectionHandlers: ((conn: unknown) => void)[];
connections: FakeConnection[];
calls: { sql: string; values?: unknown[] }[];
ended: boolean;
endError: unknown;
endThrows: unknown;
respond: (sql: string, values?: unknown[]) => unknown;
on: (event: string, handler: (conn: unknown) => void) => FakePool;
promise: () => { getConnection: () => Promise<FakeConnection> };
end: (cb: (err?: unknown) => void) => void;
}
// mysql2 is the wire boundary: stub the driver's pool so we can assert on
// the exact SQL, parameters and connection choreography the adapter emits.
const { createPoolMock, createdPools } = vi.hoisted(() => {
const createdPools: FakePool[] = [];
const createPoolMock = vi.fn((poolConfig: Record<string, unknown>) => {
const pool: FakePool = {
poolConfig,
connectionHandlers: [],
connections: [],
calls: [],
ended: false,
endError: null,
endThrows: null,
respond: () => [[], undefined],
on(event, handler) {
if (event === 'connection') {
pool.connectionHandlers.push(handler);
}
return pool;
},
promise() {
return {
getConnection: async () => {
const run = async (sql: string, values?: unknown[]) => {
pool.calls.push({ sql, values });
return pool.respond(sql, values);
};
const conn: FakeConnection = {
query: vi.fn(run),
execute: vi.fn(run),
beginTransaction: vi.fn(async () => {}),
commit: vi.fn(async () => {}),
rollback: vi.fn(async () => {}),
release: vi.fn(),
};
pool.connections.push(conn);
return conn;
},
};
},
end(cb) {
if (pool.endThrows) throw pool.endThrows;
pool.ended = true;
cb(pool.endError ?? undefined);
},
};
createdPools.push(pool);
return pool;
});
return { createPoolMock, createdPools };
});
vi.mock('mysql2', () => ({ createPool: createPoolMock }));
describe('compareMigrationFilenames', () => {
it('orders numbered migrations numerically, not lexically', () => {
const input = [
@@ -132,7 +214,9 @@ const codedError = (code: string) => {
describe('MySQLDatabaseClient.read — replica failover', () => {
it('fails over to the primary on batcher load-shed errors', async () => {
const replica = { execute: vi.fn().mockRejectedValue(codedError('dbBatchFailed')) };
const replica = {
execute: vi.fn().mockRejectedValue(codedError('dbBatchFailed')),
};
const primary = { execute: vi.fn().mockResolvedValue([[{ ok: 1 }]]) };
const client = makeClient({ replica, primary });
@@ -141,7 +225,9 @@ describe('MySQLDatabaseClient.read — replica failover', () => {
});
it('fails over on transient connection errors', async () => {
const replica = { execute: vi.fn().mockRejectedValue(codedError('ECONNRESET')) };
const replica = {
execute: vi.fn().mockRejectedValue(codedError('ECONNRESET')),
};
const primary = { execute: vi.fn().mockResolvedValue([[{ ok: 1 }]]) };
const client = makeClient({ replica, primary });
@@ -149,7 +235,9 @@ describe('MySQLDatabaseClient.read — replica failover', () => {
});
it('rethrows deterministic SQL errors without touching the primary', async () => {
const replica = { execute: vi.fn().mockRejectedValue(codedError('ER_PARSE_ERROR')) };
const replica = {
execute: vi.fn().mockRejectedValue(codedError('ER_PARSE_ERROR')),
};
const primary = { execute: vi.fn() };
const client = makeClient({ replica, primary });
@@ -160,7 +248,9 @@ describe('MySQLDatabaseClient.read — replica failover', () => {
});
it('does not fail over in single-node configuration', async () => {
const replica = { execute: vi.fn().mockRejectedValue(codedError('dbBatchFailed')) };
const replica = {
execute: vi.fn().mockRejectedValue(codedError('dbBatchFailed')),
};
const primary = { execute: vi.fn() };
const client = makeClient({ replica, primary, multiNode: false });
@@ -170,3 +260,588 @@ describe('MySQLDatabaseClient.read — replica failover', () => {
expect(primary.execute).not.toHaveBeenCalled();
});
});
// -- Driver-level behaviour (stubbed mysql2 pool) ---------------------
const mysqlConfig = (
database: Partial<NonNullable<IConfig['database']>> = {},
): IConfig =>
({
port: 0,
extensions: [],
database: { engine: 'mysql', ...database },
}) as IConfig;
const startClient = async (
database: Partial<NonNullable<IConfig['database']>> = {},
): Promise<MySQLDatabaseClient> => {
const client = new MySQLDatabaseClient(mysqlConfig(database));
await client.onServerStart();
return client;
};
// The batcher coalesces a batch into `stmt1;stmt2; SELECT 1`, so the driver
// must answer with one row-set per statement plus one for the SELECT 1.
const rowSets = (...sets: unknown[][]) => [[...sets, [{ 1: 1 }]], undefined];
beforeEach(() => {
createdPools.length = 0;
createPoolMock.mockClear();
});
describe('MySQLDatabaseClient — pool construction', () => {
it('builds the primary pool from config with multi-statement support', async () => {
await startClient({
host: 'db.internal',
port: 3307,
user: 'puter',
password: 'hunter2',
database: 'puterdb',
});
expect(createPoolMock).toHaveBeenCalledTimes(1);
expect(createdPools[0].poolConfig).toEqual({
maxPreparedStatements: 900,
connectionLimit: 30,
enableKeepAlive: true,
host: 'db.internal',
port: 3307,
user: 'puter',
password: 'hunter2',
database: 'puterdb',
multipleStatements: true,
});
});
it('falls back to loopback defaults when the endpoint is unspecified', async () => {
await startClient();
expect(createdPools[0].poolConfig).toMatchObject({
host: '127.0.0.1',
port: 3306,
user: 'root',
password: '',
database: 'puter',
});
});
it('arms a server-side statement timeout on every new connection', async () => {
await startClient({ selectTimeoutMs: 12_000 });
const pool = createdPools[0];
expect(pool.connectionHandlers).toHaveLength(1);
const conn = { query: vi.fn() };
pool.connectionHandlers[0](conn);
expect(conn.query).toHaveBeenCalledWith(
'SET SESSION max_execution_time = 12000',
);
});
it('omits the statement timeout when configured to 0', async () => {
await startClient({ selectTimeoutMs: 0 });
expect(createdPools[0].connectionHandlers).toHaveLength(0);
});
it('shares one pool between reads and writes without a replica', async () => {
await startClient();
expect(createPoolMock).toHaveBeenCalledTimes(1);
});
it('creates a second pool when a read-replica is configured', async () => {
await startClient({
replica: { host: 'replica.internal', port: 3306 },
});
expect(createPoolMock).toHaveBeenCalledTimes(2);
expect(createdPools[1].poolConfig).toMatchObject({
host: 'replica.internal',
multipleStatements: true,
});
});
});
describe('MySQLDatabaseClient — query interface', () => {
it('sends the query and its parameters through to the driver', async () => {
const client = await startClient();
const pool = createdPools[0];
pool.respond = () => rowSets([{ id: 1, username: 'ada' }]);
await expect(
client.read('SELECT * FROM `user` WHERE `id` = ?', [1]),
).resolves.toEqual([{ id: 1, username: 'ada' }]);
expect(pool.calls).toEqual([
{
sql: 'SELECT * FROM `user` WHERE `id` = ?; SELECT 1',
values: [1],
},
]);
});
it('returns an empty array when the driver yields no row-set', async () => {
const client = await startClient();
createdPools[0].respond = () => [[], undefined];
await expect(client.read('SELECT 1')).resolves.toEqual([]);
await expect(client.pread('SELECT 1')).resolves.toEqual([]);
});
it('reads from the replica pool and primary-reads from the primary', async () => {
const client = await startClient({ replica: { host: 'replica' } });
const [primary, replica] = createdPools;
primary.respond = () => rowSets([{ from: 'primary' }]);
replica.respond = () => rowSets([{ from: 'replica' }]);
await expect(client.read('SELECT 1')).resolves.toEqual([
{ from: 'replica' },
]);
await expect(client.pread('SELECT 1')).resolves.toEqual([
{ from: 'primary' },
]);
});
it('maps the mysql result header onto the shared write result', async () => {
const client = await startClient();
createdPools[0].respond = () =>
rowSets({ insertId: 42, affectedRows: 2 } as unknown as unknown[]);
await expect(
client.write('UPDATE `user` SET `username` = ? WHERE `id` = ?', [
'ada',
1,
]),
).resolves.toEqual({
insertId: 42,
affectedRows: 2,
anyRowsAffected: true,
});
});
it('reports no rows affected when the header omits the counters', async () => {
const client = await startClient();
createdPools[0].respond = () => rowSets({} as unknown as unknown[]);
await expect(client.write('DELETE FROM `user`')).resolves.toEqual({
insertId: 0,
affectedRows: 0,
anyRowsAffected: false,
});
});
it('builds an INSERT with quoted identifiers and positional params', async () => {
const client = await startClient();
const pool = createdPools[0];
pool.respond = () =>
rowSets({ insertId: 7, affectedRows: 1 } as unknown as unknown[]);
await expect(
client.insert('user', { username: 'ada', email: null }),
).resolves.toMatchObject({ insertId: 7 });
expect(pool.calls[0]).toEqual({
sql:
'INSERT INTO `user` (`username`, `email`) VALUES (?, ?); ' +
'SELECT 1',
values: ['ada', null],
});
});
});
describe('MySQLDatabaseClient — batchWrite transactions', () => {
it('runs every statement on one connection inside a transaction', async () => {
const client = await startClient();
const pool = createdPools[0];
await client.batchWrite([
{ statement: 'UPDATE `user` SET `x` = ?', values: [1] },
{
statement: 'DELETE FROM `sessions` WHERE `uuid` = ?',
values: ['s'],
},
]);
expect(pool.connections).toHaveLength(1);
const conn = pool.connections[0];
expect(conn.beginTransaction).toHaveBeenCalledTimes(1);
expect(conn.execute.mock.calls).toEqual([
['UPDATE `user` SET `x` = ?', [1]],
['DELETE FROM `sessions` WHERE `uuid` = ?', ['s']],
]);
expect(conn.commit).toHaveBeenCalledTimes(1);
expect(conn.rollback).not.toHaveBeenCalled();
expect(conn.release).toHaveBeenCalledTimes(1);
});
it('rolls back and releases the connection when a statement fails', async () => {
const client = await startClient();
const pool = createdPools[0];
pool.respond = (sql) => {
if (sql.startsWith('DELETE')) throw new Error('constraint blew up');
return [[], undefined];
};
await expect(
client.batchWrite([
{ statement: 'UPDATE `user` SET `x` = ?', values: [1] },
{ statement: 'DELETE FROM `user`', values: [] },
]),
).rejects.toThrow('constraint blew up');
const conn = pool.connections[0];
expect(conn.commit).not.toHaveBeenCalled();
expect(conn.rollback).toHaveBeenCalledTimes(1);
expect(conn.release).toHaveBeenCalledTimes(1);
});
it('still rethrows the original failure when the rollback also fails', async () => {
const client = await startClient();
const pool = createdPools[0];
pool.respond = () => {
throw new Error('statement blew up');
};
const original = client.batchWrite([
{ statement: 'UPDATE `user` SET `x` = ?', values: [1] },
]);
// The rollback is attached after the connection is handed out, so
// patch it once the adapter has acquired one.
await Promise.resolve();
pool.connections[0]?.rollback.mockRejectedValue(
new Error('rollback blew up'),
);
await expect(original).rejects.toThrow('statement blew up');
expect(pool.connections[0].release).toHaveBeenCalledTimes(1);
});
it('never acquires a connection for an empty batch', async () => {
const client = await startClient();
await client.batchWrite([]);
expect(createdPools[0].connections).toHaveLength(0);
});
});
describe('MySQLDatabaseClient — tryHardRead', () => {
it('issues exactly one query in single-node configuration', async () => {
const client = await startClient();
const pool = createdPools[0];
pool.respond = () => rowSets([{ id: 1 }]);
await expect(client.tryHardRead('SELECT 1')).resolves.toEqual([
{ id: 1 },
]);
expect(pool.calls).toHaveLength(1);
});
it('prefers replica rows when the replica has them', async () => {
const client = await startClient({ replica: { host: 'replica' } });
const [primary, replica] = createdPools;
primary.respond = () => rowSets([{ from: 'primary' }]);
replica.respond = () => rowSets([{ from: 'replica' }]);
await expect(client.tryHardRead('SELECT 1')).resolves.toEqual([
{ from: 'replica' },
]);
});
it('falls back to the primary when the replica is behind', async () => {
const client = await startClient({ replica: { host: 'replica' } });
const [primary, replica] = createdPools;
primary.respond = () => rowSets([{ from: 'primary' }]);
replica.respond = () => rowSets([]);
await expect(client.tryHardRead('SELECT 1')).resolves.toEqual([
{ from: 'primary' },
]);
});
it('falls back to the primary when the replica query throws', async () => {
const client = await startClient({ replica: { host: 'replica' } });
const [primary, replica] = createdPools;
primary.respond = () => rowSets([{ from: 'primary' }]);
replica.respond = () => {
throw new Error('replica down');
};
await expect(client.tryHardRead('SELECT 1')).resolves.toEqual([
{ from: 'primary' },
]);
});
});
describe('MySQLDatabaseClient — migrations', () => {
let dir: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), 'puter-mysql-mig-'));
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it('applies mysql migrations in numeric order, one statement at a time', async () => {
writeFileSync(
join(dir, 'mysql_mig_2.sql'),
'CREATE TABLE b (id INT);\nCREATE TABLE c (id INT);\n',
);
writeFileSync(
join(dir, 'mysql_mig_10.sql'),
'CREATE TABLE d (id INT);',
);
// Other engines' files and non-SQL noise must be ignored.
writeFileSync(join(dir, 'postgres_mig_1.sql'), 'CREATE TABLE nope ();');
writeFileSync(join(dir, 'mysql_notes.txt'), 'not sql');
await startClient({ migrationPaths: [dir] });
const pool = createdPools[0];
expect(pool.calls.map((c) => c.sql)).toEqual([
'CREATE TABLE b (id INT)',
'CREATE TABLE c (id INT)',
'CREATE TABLE d (id INT)',
]);
expect(pool.connections[0].release).toHaveBeenCalledTimes(1);
});
it('reports which file and statement index failed', async () => {
writeFileSync(
join(dir, 'mysql_mig_1.sql'),
'CREATE TABLE ok (id INT);\nCREATE TABLE bad (id INT);',
);
const client = new MySQLDatabaseClient(
mysqlConfig({ migrationPaths: [dir] }),
);
const cause = new Error('ER_PARSE_ERROR');
// The pool only exists once onServerStart runs, so arm the failure
// through the factory.
createPoolMock.mockImplementationOnce((poolConfig) => {
const pool = createPoolMock.getMockImplementation()!(poolConfig);
pool.respond = (sql: string) => {
if (sql.includes('bad')) throw cause;
return [[], undefined];
};
return pool;
});
await expect(client.onServerStart()).rejects.toThrow(
'[mysql] failed to apply mysql_mig_1.sql at statement 1',
);
});
it('fails loudly when a configured migration path is unreadable', async () => {
await expect(
startClient({ migrationPaths: [join(dir, 'does-not-exist')] }),
).rejects.toThrow('[mysql] migration path is unreadable');
});
it('skips a configured directory that holds no mysql migrations', async () => {
writeFileSync(join(dir, 'readme.md'), '# nothing here');
await startClient({ migrationPaths: [dir] });
expect(createdPools[0].calls).toEqual([]);
});
});
describe('MySQLDatabaseClient — pool lifecycle', () => {
it('replaces and closes the previous primary pool on reinit', async () => {
const client = await startClient();
const original = createdPools[0];
client.reinitPrimary();
expect(createdPools).toHaveLength(2);
expect(original.ended).toBe(true);
// Single-node: the replica batcher must follow the new primary.
createdPools[1].respond = () => rowSets([{ from: 'new-primary' }]);
await expect(client.read('SELECT 1')).resolves.toEqual([
{ from: 'new-primary' },
]);
});
it('replaces only the replica pool on replica reinit', async () => {
const client = await startClient({ replica: { host: 'replica' } });
const [primary, originalReplica] = createdPools;
client.reinitReplica();
expect(createdPools).toHaveLength(3);
expect(originalReplica.ended).toBe(true);
expect(primary.ended).toBe(false);
});
it('ignores a replica reinit when no replica is configured', async () => {
const client = await startClient();
client.reinitReplica();
expect(createdPools).toHaveLength(1);
});
it('ignores reinit once shutdown has started', async () => {
const client = await startClient();
await client.onServerPrepareShutdown();
client.reinitPrimary();
expect(createdPools).toHaveLength(1);
await client.onServerShutdown();
});
it('closes both pools on shutdown when a replica is configured', async () => {
const client = await startClient({ replica: { host: 'replica' } });
await client.onServerShutdown();
expect(createdPools[0].ended).toBe(true);
expect(createdPools[1].ended).toBe(true);
});
it('closes the pools once the drain window elapses', async () => {
vi.useFakeTimers();
try {
const client = await startClient();
await client.onServerPrepareShutdown();
expect(createdPools[0].ended).toBe(false);
await vi.advanceTimersByTimeAsync(60_000);
expect(createdPools[0].ended).toBe(true);
} finally {
vi.useRealTimers();
}
});
it('cancels the drain timer when shutdown arrives first', async () => {
vi.useFakeTimers();
try {
const client = await startClient();
await client.onServerPrepareShutdown();
// A second prepare is a no-op — only one drain timer may exist.
await client.onServerPrepareShutdown();
expect(vi.getTimerCount()).toBe(1);
await client.onServerShutdown();
expect(vi.getTimerCount()).toBe(0);
expect(createdPools[0].ended).toBe(true);
} finally {
vi.useRealTimers();
}
});
it('surfaces a driver error raised while closing the pool', async () => {
const client = await startClient();
createdPools[0].endError = new Error('pool refused to close');
await expect(client.onServerShutdown()).rejects.toThrow(
'pool refused to close',
);
});
it('surfaces a driver error thrown synchronously by end()', async () => {
const client = await startClient();
createdPools[0].endThrows = new Error('end exploded');
await expect(client.onServerShutdown()).rejects.toThrow('end exploded');
});
});
describe('MySQLDatabaseClient.readWithRetry', () => {
const codedFailure = (code: string) => {
const err = new Error(code) as Error & { code: string };
err.code = code;
return err;
};
it('classifies transient connection failures as retriable', () => {
expect(
MySQLDatabaseClient.isRetriableError(codedFailure('ECONNRESET')),
).toBe(true);
expect(
MySQLDatabaseClient.isRetriableError(codedFailure('ER_DUP_ENTRY')),
).toBe(false);
expect(
MySQLDatabaseClient.isRetriableError(new Error('Connection lost')),
).toBe(true);
});
it('retries a transient failure and returns the eventual result', async () => {
const client = await startClient();
let attempts = 0;
const operation = vi.fn(async () => {
attempts += 1;
if (attempts < 3) throw codedFailure('PROTOCOL_CONNECTION_LOST');
return [{ ok: 1 }];
});
await expect(
client.readWithRetry('health', operation, { baseBackoffMs: 0 }),
).resolves.toEqual([{ ok: 1 }]);
expect(operation).toHaveBeenCalledTimes(3);
});
it('gives up after the attempt budget is spent', async () => {
const client = await startClient();
const operation = vi.fn(async () => {
throw codedFailure('ETIMEDOUT');
});
await expect(
client.readWithRetry('health', operation, {
maxAttempts: 2,
baseBackoffMs: 0,
}),
).rejects.toMatchObject({ code: 'ETIMEDOUT' });
expect(operation).toHaveBeenCalledTimes(2);
});
it('never retries a deterministic SQL error', async () => {
const client = await startClient();
const operation = vi.fn(async () => {
throw codedFailure('ER_PARSE_ERROR');
});
await expect(
client.readWithRetry('health', operation, { baseBackoffMs: 0 }),
).rejects.toMatchObject({ code: 'ER_PARSE_ERROR' });
expect(operation).toHaveBeenCalledTimes(1);
});
it('stops retrying once shutdown has started', async () => {
const client = await startClient();
await client.onServerPrepareShutdown();
const operation = vi.fn(async () => {
throw codedFailure('ECONNRESET');
});
await expect(
client.readWithRetry('health', operation, { baseBackoffMs: 0 }),
).rejects.toMatchObject({ code: 'ECONNRESET' });
expect(operation).toHaveBeenCalledTimes(1);
await client.onServerShutdown();
});
it('applies jittered exponential backoff capped at maxBackoffMs', async () => {
const client = await startClient();
const delays: number[] = [];
const realSetTimeout = globalThis.setTimeout;
const timeoutSpy = vi
.spyOn(globalThis, 'setTimeout')
.mockImplementation(((fn: () => void, ms?: number) => {
delays.push(ms ?? 0);
return realSetTimeout(fn, 0);
}) as typeof setTimeout);
try {
const operation = vi.fn(async () => {
throw codedFailure('ECONNRESET');
});
await expect(
client.readWithRetry('health', operation, {
maxAttempts: 4,
baseBackoffMs: 100,
maxBackoffMs: 150,
jitterRatio: 0,
}),
).rejects.toMatchObject({ code: 'ECONNRESET' });
expect(delays).toEqual([100, 150, 150]);
} finally {
timeoutSpy.mockRestore();
}
});
});
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -248,9 +248,9 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
/**
* Apply `.sql` files from each configured migration directory in order.
* Files within a directory are sorted lexically. Files MUST be
* idempotent — there is no per-file applied-state tracking. Failures
* abort startup so operators see schema problems loud.
* Files within a directory are sorted lexically. Files MUST be idempotent —
* there is no per-file applied-state tracking. Failures abort startup so
* operators see schema problems loud.
*/
private async runMigrations(): Promise<void> {
const paths = this.config.database?.migrationPaths;
@@ -414,8 +414,10 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
return isRetriableError(error);
}
/** Replica failures worth retrying on the primary: batcher load-shed
* or transient connection errors — never deterministic SQL errors. */
/**
* Replica failures worth retrying on the primary: batcher load-shed or
* transient connection errors — never deterministic SQL errors.
*/
private static isFailoverWorthy(error: unknown): boolean {
const code = (error as { code?: string })?.code;
return code === 'dbBatchFailed' || isRetriableError(error);
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
+24 -15
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -22,6 +22,7 @@ import {
POOL_ACQUIRE_TIMEOUT,
isNeverSentError,
isRetriableError,
isRolledBackError,
} from './retriableErrors.js';
const DEFAULT_MAX_QUEUE_SIZE = 1000;
@@ -89,20 +90,22 @@ export class SQLBatcher {
#metricAttrs;
/**
* @param {object} dbPool mysql2 pool
* @param {object} dbPool Mysql2 pool
* @param {object} [opts]
* @param {number} [opts.maxTimeInQueue] ms an item may wait before flush
* @param {number} [opts.maxBatchSize] items coalesced per flush
* @param {number} [opts.maxQueueSize] drop-oldest high-water mark
* @param {number} [opts.failureThreshold] consecutive failures to open the breaker
* @param {number} [opts.cooldownMs] breaker open duration after last failure
* @param {'primary'|'replica'} [opts.poolLabel] role label on metrics; in
* @param {number} [opts.maxTimeInQueue] Ms an item may wait before flush
* @param {number} [opts.maxBatchSize] Items coalesced per flush
* @param {number} [opts.maxQueueSize] Drop-oldest high-water mark
* @param {number} [opts.failureThreshold] Consecutive failures to open the
* breaker
* @param {number} [opts.cooldownMs] Breaker open duration after last
* failure
* @param {'primary' | 'replica'} [opts.poolLabel] Role label on metrics; in
* single-node setups the 'replica' batcher shares the primary pool, so
* this reflects the read/write role rather than a physical instance
* @param {boolean} [opts.readOnly] this batcher only ever carries SELECTs,
* @param {boolean} [opts.readOnly] This batcher only ever carries SELECTs,
* so any transient failure is safe to retry
* @param {number} [opts.acquireTimeoutMs] max wait for a pooled
* connection; 0 disables the bound
* @param {number} [opts.acquireTimeoutMs] Max wait for a pooled connection;
* 0 disables the bound
*/
constructor(
dbPool,
@@ -359,8 +362,14 @@ export class SQLBatcher {
// Run one fallback item, retrying transient failures with backoff.
// A read-only batcher may retry anything transient; a batcher that
// carries writes only retries failures where the statement provably
// never reached the server — a write that died mid-flight may have
// committed, and re-running it would double-apply.
// did not apply — either it never reached the server, or the server
// rolled it back itself. A write that died mid-flight may have
// committed, and re-running that one would double-apply.
//
// Lock contention lands in the second group and is worth retrying rather
// than surfacing: an item is a single statement, so a deadlock victim has
// been fully undone, and the caller sees an unhandled 500 for what the
// database is telling us to just run again.
async #runFallbackItem(b) {
let attempt = 0;
while (true) {
@@ -377,8 +386,8 @@ export class SQLBatcher {
};
} catch (error) {
const canRetry = this.readOnly
? isRetriableError(error)
: isNeverSentError(error);
? isRetriableError(error) || isRolledBackError(error)
: isNeverSentError(error) || isRolledBackError(error);
if (!canRetry || attempt >= ITEM_RETRY_ATTEMPTS) {
return { ok: false, error };
}
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -208,6 +208,36 @@ describe('SQLBatcher', () => {
expect(fallbackAttempts).toBe(2);
});
it('retries a deadlocked write instead of surfacing it', async () => {
let fallbackAttempts = 0;
const conn = makeConnection((sql) => {
if (isBatchQuery(sql)) throw makeError('ER_LOCK_DEADLOCK');
fallbackAttempts++;
if (fallbackAttempts === 1) throw makeError('ER_LOCK_DEADLOCK');
return [[{ ok: 1 }], undefined];
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, { maxTimeInQueue: 5 });
// The victim statement was rolled back by the server, so re-running it
// can't double-apply — the caller should never see the deadlock.
const result = await batcher.query('UPDATE notification SET x', []);
expect(result[0]).toEqual([{ ok: 1 }]);
expect(fallbackAttempts).toBe(2);
});
it('gives up on a write that deadlocks past the retry budget', async () => {
const conn = makeConnection(() => {
throw makeError('ER_LOCK_DEADLOCK');
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, { maxTimeInQueue: 5 });
await expect(batcher.query('UPDATE hot SET x', [])).rejects.toMatchObject(
{ code: 'ER_LOCK_DEADLOCK' },
);
});
it('never retries deterministic row-level errors and does not escalate the breaker', async () => {
let fallbackAttempts = 0;
const conn = makeConnection((sql) => {
@@ -0,0 +1,437 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import Database from 'better-sqlite3';
import { existsSync, mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import type { IConfig } from '../../types';
import { DatabaseClientFactory } from './index.js';
import { SqliteDatabaseClient } from './SqliteDatabaseClient.js';
/** Highest schema version the migration table can reach. */
const CURRENT_SCHEMA_VERSION = 61;
const SYSTEM_USER_UUID = '5d4adce0-a381-4982-9c02-6e2540026238';
const sqliteConfig = (
database: Partial<NonNullable<IConfig['database']>> = {},
): IConfig =>
({
port: 0,
extensions: [],
database: { engine: 'sqlite', inMemory: true, ...database },
}) as IConfig;
const bootClient = async (
database: Partial<NonNullable<IConfig['database']>> = {},
): Promise<SqliteDatabaseClient> => {
const client = new SqliteDatabaseClient(sqliteConfig(database));
await client.onServerStart();
return client;
};
const userVersionOf = async (client: SqliteDatabaseClient): Promise<number> => {
const [row] = await client.read('PRAGMA user_version');
return row.user_version as number;
};
describe('SqliteDatabaseClient — boot and migrations', () => {
let client: SqliteDatabaseClient;
beforeEach(async () => {
client = await bootClient();
});
afterEach(() => {
client.onServerShutdown();
});
it('is what the factory picks for the sqlite engine', () => {
expect(new DatabaseClientFactory(sqliteConfig())).toBeInstanceOf(
SqliteDatabaseClient,
);
});
it('migrates a fresh database all the way to the current version', async () => {
expect(await userVersionOf(client)).toBe(CURRENT_SCHEMA_VERSION);
});
it('runs the javascript migrations, not just the .sql ones', async () => {
// The `system` user only exists because 0025 (a .dbmig.js file) ran
// inside the migration VM.
const rows = await client.read(
'SELECT `username` FROM `user` WHERE `uuid` = ?',
[SYSTEM_USER_UUID],
);
expect(rows).toEqual([{ username: 'system' }]);
});
it('leaves an already-migrated database untouched on a second boot', async () => {
const dir = mkdtempSync(join(tmpdir(), 'puter-sqlite-'));
const path = join(dir, 'nested', 'puter.sqlite');
try {
const first = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path }),
);
await first.onServerStart();
await first.write(
'INSERT INTO `kv` (`user_id`, `kkey_hash`, `kkey`, `value`) ' +
'VALUES (?, ?, ?, ?)',
[1, 1, 'boot-marker', '"kept"'],
);
first.onServerShutdown();
expect(existsSync(path)).toBe(true);
const second = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path }),
);
await second.onServerStart();
await expect(
second.read('SELECT `value` FROM `kv` WHERE `kkey` = ?', [
'boot-marker',
]),
).resolves.toEqual([{ value: '"kept"' }]);
expect(await userVersionOf(second)).toBe(CURRENT_SCHEMA_VERSION);
second.onServerShutdown();
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it('stops early at a configured target version', async () => {
const partial = await bootClient({ targetVersion: 5 });
try {
expect(await userVersionOf(partial)).toBe(5);
// 0005 landed (apps.background); 0011 (notification) did not.
await expect(
partial.read(
"SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'notification'",
),
).resolves.toEqual([]);
await expect(
partial.read(
"SELECT 1 FROM pragma_table_info('apps') WHERE name = 'background'",
),
).resolves.toHaveLength(1);
} finally {
partial.onServerShutdown();
}
});
});
describe('SqliteDatabaseClient — legacy version inference', () => {
let dir: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), 'puter-sqlite-legacy-'));
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it('treats user_version=0 with no bootstrap tables as uninitialized', async () => {
const path = join(dir, 'blank.sqlite');
new Database(path).close();
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
const client = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path }),
);
try {
await client.onServerStart();
expect(warn).toHaveBeenCalledWith(
expect.stringContaining('bootstrap tables are missing'),
);
// A full migration run followed, so the schema is current.
expect(await userVersionOf(client)).toBe(CURRENT_SCHEMA_VERSION);
} finally {
client.onServerShutdown();
warn.mockRestore();
}
});
it('infers the schema version from table and column markers', async () => {
const path = join(dir, 'legacy.sqlite');
const seed = new Database(path);
seed.exec(`
CREATE TABLE user (
id INTEGER PRIMARY KEY,
username TEXT,
otp_secret TEXT,
otp_enabled INTEGER,
otp_recovery_codes TEXT
);
CREATE TABLE apps (
id INTEGER PRIMARY KEY,
uid TEXT,
background INTEGER,
metadata TEXT
);
CREATE TABLE user_to_user_permissions (id INTEGER PRIMARY KEY);
CREATE TABLE audit_user_to_user_permissions (
id INTEGER PRIMARY KEY,
issuer_user_id INTEGER,
holder_user_id INTEGER
);
CREATE TABLE sessions (
id INTEGER PRIMARY KEY,
created_at INTEGER,
last_activity INTEGER
);
CREATE TABLE kv (id INTEGER PRIMARY KEY, value JSON);
INSERT INTO apps (uid)
VALUES ('app-e3ac5486-da8c-42ad-8377-8728086e0980');
`);
seed.close();
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
// targetVersion just past the inferred version keeps the run a
// no-op, so the assertion is about inference alone.
const client = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path, targetVersion: 22 }),
);
try {
await client.onServerStart();
expect(warn).toHaveBeenCalledWith(
'[sqlite] user_version=0; inferred legacy schema version 21',
);
// Nothing was applied, so user_version stays where it was.
expect(await userVersionOf(client)).toBe(0);
} finally {
client.onServerShutdown();
warn.mockRestore();
}
});
it('reports version 0 when only the bootstrap tables exist', async () => {
const path = join(dir, 'bootstrap-only.sqlite');
const seed = new Database(path);
seed.exec(`
CREATE TABLE user (id INTEGER PRIMARY KEY);
CREATE TABLE apps (id INTEGER PRIMARY KEY);
`);
seed.close();
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
const client = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path, targetVersion: 1 }),
);
try {
await client.onServerStart();
expect(log).toHaveBeenCalledWith('[sqlite] database version: 0');
} finally {
client.onServerShutdown();
log.mockRestore();
}
});
});
describe('SqliteDatabaseClient — query interface', () => {
let client: SqliteDatabaseClient;
beforeEach(async () => {
client = await bootClient();
await client.write(
'CREATE TABLE `widget` (`id` INTEGER PRIMARY KEY, ' +
'`name` TEXT, `enabled` INTEGER, `seen_at` TEXT)',
);
});
afterEach(() => {
client.onServerShutdown();
});
it('reports lastInsertRowid and change counts on write', async () => {
const inserted = await client.write(
'INSERT INTO `widget` (`name`) VALUES (?)',
['spanner'],
);
expect(inserted).toEqual({
insertId: 1,
affectedRows: 1,
anyRowsAffected: true,
});
const missed = await client.write(
'UPDATE `widget` SET `name` = ? WHERE `id` = ?',
['nope', 999],
);
expect(missed).toMatchObject({
affectedRows: 0,
anyRowsAffected: false,
});
});
it('binds booleans as sqlite integers', async () => {
await client.write(
'INSERT INTO `widget` (`name`, `enabled`) VALUES (?, ?)',
['toggled', true],
);
await client.write(
'INSERT INTO `widget` (`name`, `enabled`) VALUES (?, ?)',
['untoggled', false],
);
await expect(
client.read('SELECT `name`, `enabled` FROM `widget` ORDER BY `id`'),
).resolves.toEqual([
{ name: 'toggled', enabled: 1 },
{ name: 'untoggled', enabled: 0 },
]);
});
it('rewrites now() into the sqlite equivalent', async () => {
await client.write(
'INSERT INTO `widget` (`name`, `seen_at`) VALUES (?, NOW())',
['stamped'],
);
const [row] = await client.read(
'SELECT `seen_at` FROM `widget` WHERE `name` = ?',
['stamped'],
);
expect(row.seen_at).toMatch(/^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}$/);
});
it('generates an INSERT from a data object', async () => {
await client.insert('widget', { name: 'generated', enabled: true });
await expect(
client.read('SELECT `name`, `enabled` FROM `widget`'),
).resolves.toEqual([{ name: 'generated', enabled: 1 }]);
});
it('primary-reads through the same single-node connection', async () => {
await client.write('INSERT INTO `widget` (`name`) VALUES (?)', ['p']);
await expect(
client.pread('SELECT `name` FROM `widget`'),
).resolves.toEqual([{ name: 'p' }]);
});
it('applies a batch write atomically', async () => {
await client.batchWrite([
{
statement:
'INSERT INTO `widget` (`name`, `enabled`) VALUES (?, ?)',
values: ['one', true],
},
{
statement: 'INSERT INTO `widget` (`name`) VALUES (?)',
values: ['two'],
},
]);
await expect(
client.read('SELECT `name` FROM `widget` ORDER BY `id`'),
).resolves.toEqual([{ name: 'one' }, { name: 'two' }]);
});
it('rolls the whole batch back when one statement fails', async () => {
await expect(
client.batchWrite([
{
statement: 'INSERT INTO `widget` (`name`) VALUES (?)',
values: ['kept-if-broken'],
},
{
statement: 'INSERT INTO `nonexistent` (`name`) VALUES (?)',
values: ['boom'],
},
]),
).rejects.toThrow(/no such table/i);
await expect(
client.read('SELECT `name` FROM `widget`'),
).resolves.toEqual([]);
});
it('surfaces sqlite errors for malformed SQL', async () => {
await expect(client.read('SELEC oops')).rejects.toThrow(
/syntax error/i,
);
});
it('closes the database on shutdown', async () => {
const closable = await bootClient();
closable.onServerShutdown();
await expect(closable.read('SELECT 1')).rejects.toThrow(
'The database connection is not open',
);
});
});
// Single-node engines have no replica to race, so `tryHardRead` must not
// issue the query twice — the base-class default fires `pread()` and
// `read()` in parallel, which on sqlite is the same connection.
describe('SqliteDatabaseClient — tryHardRead', () => {
let client: SqliteDatabaseClient;
beforeEach(async () => {
client = await bootClient();
await client.write(
'CREATE TABLE `widget` (`id` INTEGER PRIMARY KEY, `name` TEXT)',
);
});
afterEach(() => {
client.onServerShutdown();
});
it('executes the statement exactly once', async () => {
await client.write('INSERT INTO `widget` (`name`) VALUES (?)', ['one']);
const readSpy = vi.spyOn(client, 'read');
const preadSpy = vi.spyOn(client, 'pread');
await expect(
client.tryHardRead('SELECT `name` FROM `widget`'),
).resolves.toEqual([{ name: 'one' }]);
expect(readSpy).toHaveBeenCalledTimes(1);
expect(preadSpy).not.toHaveBeenCalled();
});
it('executes the statement exactly once when it matches no rows', async () => {
const readSpy = vi.spyOn(client, 'read');
const preadSpy = vi.spyOn(client, 'pread');
await expect(
client.tryHardRead('SELECT `name` FROM `widget`'),
).resolves.toEqual([]);
expect(readSpy).toHaveBeenCalledTimes(1);
expect(preadSpy).not.toHaveBeenCalled();
});
it('throws from requireRead when nothing matches', async () => {
await expect(
client.requireRead(
'SELECT `name` FROM `widget` WHERE `id` = ?',
[42],
),
).rejects.toThrow('required read returned no rows');
});
it('returns the rows from requireRead when something matches', async () => {
await client.write('INSERT INTO `widget` (`name`) VALUES (?)', ['ok']);
await expect(
client.requireRead('SELECT `name` FROM `widget`'),
).resolves.toEqual([{ name: 'ok' }]);
});
});
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -27,9 +27,9 @@ import { AbstractDatabaseClient, type WriteResult } from './DatabaseClient';
const MIGRATIONS_DIR = resolve(__dirname, './migrations/sqlite');
/**
* Ordered list of [threshold_version, files[]] pairs.
* A database whose `user_version` is <= threshold_version will have
* the corresponding files applied.
* Ordered list of [threshold_version, files[]] pairs. A database whose
* `user_version` is <= threshold_version will have the corresponding files
* applied.
*/
const AVAILABLE_MIGRATIONS: [number, string[]][] = [
[-1, ['0001_create-tables.sql', '0002_add-default-apps.sql']],
@@ -94,6 +94,7 @@ const AVAILABLE_MIGRATIONS: [number, string[]][] = [
[57, ['0062_blocked-app-origins.sql']],
[58, ['0063_add_suspended_reason.sql']],
[59, ['0064_abuse-moderation-events.sql']],
[60, ['0065_app-feedback.sql']],
];
export class SqliteDatabaseClient extends AbstractDatabaseClient {
@@ -158,6 +159,16 @@ export class SqliteDatabaseClient extends AbstractDatabaseClient {
return this.read(query, params);
}
override async tryHardRead(
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
// No replica to race, so the base class's parallel
// primary-plus-replica read would run the same statement twice on
// the one connection.
return this.read(query, params);
}
@Span('db.write', (query: string) => ({ 'db.statement': query }))
override async write(
query: string,
+4 -4
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -33,9 +33,9 @@ import { PostgresDatabaseClient } from './PostgresDatabaseClient';
import { SqliteDatabaseClient } from './SqliteDatabaseClient';
/**
* Factory class registered in `puterClients`. PuterServer calls
* `new DatabaseClientFactory(config)` — the constructor returns the
* concrete subclass selected by `config.database.engine`.
* Factory class registered in `puterClients`. PuterServer calls `new
* DatabaseClientFactory(config)` — the constructor returns the concrete
* subclass selected by `config.database.engine`.
*/
export const DatabaseClientFactory = class DatabaseClientFactory {
constructor(config: IConfig) {
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -0,0 +1,48 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- User-to-developer feedback for apps that opt in. Mirrors SQLite migration
-- 0065. Opt-in is the new `apps.feedback_enabled` column (developer-writable
-- through the regular `puter.apps.update` path). Each row of `app_feedback`
-- is one message a signed-in user submitted through the GUI feedback dialog;
-- a copy is emailed to the app owner unless the per-app daily email cap
-- suppressed it (`email_sent` records which). `app_uid` is denormalized
-- alongside `app_id` so rows stay attributable after an app is deleted.
-- `created_at` is unix seconds.
--
-- Idempotent: the column add uses _puter_add_col (defined in mig_1, which
-- always runs first) and the table uses `CREATE TABLE IF NOT EXISTS`, so the
-- directory can replay safely.
CALL _puter_add_col('apps', 'feedback_enabled', '`feedback_enabled` tinyint(1) DEFAULT ''0''');
CREATE TABLE IF NOT EXISTS `app_feedback` (
`id` INT NOT NULL AUTO_INCREMENT,
`uid` CHAR(36) NOT NULL,
`app_id` INT NOT NULL,
`app_uid` CHAR(40) NOT NULL,
`user_id` INT NOT NULL,
`message` TEXT NOT NULL,
`source_env` VARCHAR(16) DEFAULT NULL,
`source_origin` VARCHAR(2048) DEFAULT NULL,
`email_sent` TINYINT(1) NOT NULL DEFAULT 0,
`created_at` BIGINT NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `uq_app_feedback_uid` (`uid`),
KEY `idx_app_feedback_app_created` (`app_id`, `created_at`),
KEY `idx_app_feedback_user_created` (`user_id`, `created_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
@@ -0,0 +1,47 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- User-to-developer feedback for apps that opt in. Mirrors SQLite migration
-- 0065 / MySQL mysql_mig_20. Opt-in is the new `apps.feedback_enabled` column
-- (developer-writable through the regular `puter.apps.update` path). Each row
-- of `app_feedback` is one message a signed-in user submitted through the GUI
-- feedback dialog; a copy is emailed to the app owner unless the per-app
-- daily email cap suppressed it (`email_sent` records which). `app_uid` is
-- denormalized alongside `app_id` so rows stay attributable after an app is
-- deleted. `created_at` is unix seconds.
--
-- Idempotent via IF NOT EXISTS.
ALTER TABLE apps ADD COLUMN IF NOT EXISTS feedback_enabled boolean NOT NULL DEFAULT FALSE;
CREATE TABLE IF NOT EXISTS app_feedback (
id BIGSERIAL PRIMARY KEY,
uid CHAR(36) NOT NULL UNIQUE,
app_id BIGINT NOT NULL,
app_uid VARCHAR(40) NOT NULL,
user_id BIGINT NOT NULL,
message TEXT NOT NULL,
source_env VARCHAR(16) DEFAULT NULL,
source_origin VARCHAR(2048) DEFAULT NULL,
email_sent BOOLEAN NOT NULL DEFAULT FALSE,
created_at BIGINT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_app_feedback_app_created
ON app_feedback (app_id, created_at);
CREATE INDEX IF NOT EXISTS idx_app_feedback_user_created
ON app_feedback (user_id, created_at);
@@ -0,0 +1,51 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- User-to-developer feedback for apps that opt in. Opt-in is the new
-- `apps.feedback_enabled` column (developer-writable through the regular
-- `puter.apps.update` path; a dedicated column rather than a `metadata` key
-- because Dev Center saves replace the whole metadata blob and would erase
-- it). Each row of `app_feedback` is one message a signed-in user submitted
-- through the GUI feedback dialog; a copy is emailed to the app owner unless
-- the per-app daily email cap suppressed it (`email_sent` records which).
-- `app_uid` is denormalized alongside `app_id` so rows stay attributable
-- after an app is deleted (abuse forensics). `source_env` is 'app' (desktop
-- dialog) or 'web' (puter.com popup opened from an external site);
-- `source_origin` is the popup opener's browser-attested origin, null for
-- desktop submissions. `created_at` is unix seconds. The (user_id,
-- created_at) and (app_id, created_at) indexes serve the sliding-window
-- rate-limit counts in AppFeedbackStore.
ALTER TABLE apps ADD COLUMN "feedback_enabled" tinyint(1) DEFAULT '0';
CREATE TABLE IF NOT EXISTS `app_feedback` (
"id" INTEGER PRIMARY KEY AUTOINCREMENT,
"uid" TEXT NOT NULL UNIQUE,
"app_id" INTEGER NOT NULL,
"app_uid" TEXT NOT NULL,
"user_id" INTEGER NOT NULL,
"message" TEXT NOT NULL,
"source_env" TEXT DEFAULT NULL, -- 'app' | 'web'
"source_origin" TEXT DEFAULT NULL, -- attested opener origin (web popups)
"email_sent" INTEGER NOT NULL DEFAULT 0,
"created_at" INTEGER NOT NULL -- unix seconds
);
CREATE INDEX IF NOT EXISTS idx_app_feedback_app_created
ON `app_feedback` (`app_id`, `created_at`);
CREATE INDEX IF NOT EXISTS idx_app_feedback_user_created
ON `app_feedback` (`user_id`, `created_at`);
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
+38 -10
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -41,11 +41,10 @@ const RETRIABLE_ERROR_MESSAGES = [
];
/**
* Failures where the statement provably never reached the server, so a
* retry can never double-apply it — safe even for writes. Anything that
* can occur after the statement was sent (resets, protocol drops) is
* deliberately excluded: the server may have committed before the
* connection died.
* Failures where the statement provably never reached the server, so a retry
* can never double-apply it — safe even for writes. Anything that can occur
* after the statement was sent (resets, protocol drops) is deliberately
* excluded: the server may have committed before the connection died.
*/
const NEVER_SENT_ERROR_CODES = new Set([
'ECONNREFUSED',
@@ -55,14 +54,33 @@ const NEVER_SENT_ERROR_CODES = new Set([
POOL_ACQUIRE_TIMEOUT,
]);
/**
* Failures the server itself rolled back before returning. The statement did
* reach the database, so this is not `NEVER_SENT`, but InnoDB guarantees it
* left no effect — which makes a retry just as safe for writes.
*
* Only true for a _single_ statement: under autocommit each statement is its
* own transaction, so "the transaction was rolled back" means "this statement
* was rolled back". Retrying a multi-statement string on one of these would
* re-run the statements that already committed ahead of the failure.
*/
const ROLLED_BACK_ERROR_CODES = new Set([
// Deadlock — InnoDB picked this transaction as the victim and undid it.
// MySQL's own message for it is "try restarting transaction".
'ER_LOCK_DEADLOCK',
// Lock wait timeout. Rolls back the statement rather than the transaction
// unless innodb_rollback_on_timeout is set — the same thing when the
// transaction is one statement.
'ER_LOCK_WAIT_TIMEOUT',
]);
const errorCode = (error: unknown): string | undefined =>
(error as { code?: string } | null)?.code;
/**
* Transient connection-level failures worth retrying — but only for
* statements that are safe to run twice (reads). Row-level errors
* (duplicate key, constraint violations) are deterministic and never
* match.
* Transient connection-level failures worth retrying — but only for statements
* that are safe to run twice (reads). Row-level errors (duplicate key,
* constraint violations) are deterministic and never match.
*/
export const isRetriableError = (error: unknown): boolean => {
const code = errorCode(error);
@@ -76,3 +94,13 @@ export const isNeverSentError = (error: unknown): boolean => {
const code = errorCode(error);
return Boolean(code && NEVER_SENT_ERROR_CODES.has(code));
};
/**
* Lock-contention failures the server rolled back on its own. Safe to retry a
* single statement on, writes included — see `ROLLED_BACK_ERROR_CODES` for the
* one-statement precondition.
*/
export const isRolledBackError = (error: unknown): boolean => {
const code = errorCode(error);
return Boolean(code && ROLLED_BACK_ERROR_CODES.has(code));
};
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -20,12 +20,11 @@
/**
* DELIMITER-aware splitter for MySQL dump / migration files.
*
* Splits `sql` into individual statements using the active statement
* delimiter (default `;`). Recognises `DELIMITER X` lines, single-quoted
* strings, double-quoted strings, backtick-quoted identifiers, line
* comments (`-- `, `#`) and block comments (`/* ... *\/`). DELIMITER
* directives are stripped from the output (they're a client-side concept,
* not server SQL).
* Splits `sql` into individual statements using the active statement delimiter
* (default `;`). Recognises `DELIMITER X` lines, single-quoted strings,
* double-quoted strings, backtick-quoted identifiers, line comments (`-- `,
* `#`) and block comments (`/* ... *\/`). DELIMITER directives are stripped
* from the output (they're a client-side concept, not server SQL).
*
* Returns trimmed, non-empty statements without the trailing delimiter.
*/
@@ -0,0 +1,116 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { describe, expect, it } from 'vitest';
import { splitPostgresStatements } from './splitPostgresStatements';
describe('splitPostgresStatements', () => {
it('splits on semicolons and trims each statement', () => {
expect(
splitPostgresStatements('SELECT 1;\n SELECT 2 ;\nSELECT 3'),
).toEqual(['SELECT 1', 'SELECT 2', 'SELECT 3']);
});
it('drops empty statements from stray semicolons', () => {
expect(splitPostgresStatements(';;SELECT 1;;')).toEqual(['SELECT 1']);
});
it('returns nothing for whitespace-only input', () => {
expect(splitPostgresStatements(' \n\t ')).toEqual([]);
});
it('keeps semicolons inside single-quoted literals', () => {
expect(
splitPostgresStatements("INSERT INTO t VALUES ('a;b'); SELECT 1"),
).toEqual(["INSERT INTO t VALUES ('a;b')", 'SELECT 1']);
});
it('handles doubled and backslash-escaped quotes in literals', () => {
expect(
splitPostgresStatements("SELECT 'it''s; fine'; SELECT 2"),
).toEqual(["SELECT 'it''s; fine'", 'SELECT 2']);
expect(splitPostgresStatements("SELECT E'a\\'; b'; SELECT 2")).toEqual([
"SELECT E'a\\'; b'",
'SELECT 2',
]);
});
it('keeps semicolons inside quoted identifiers', () => {
expect(
splitPostgresStatements('SELECT "we;ird" FROM t; SELECT 2'),
).toEqual(['SELECT "we;ird" FROM t', 'SELECT 2']);
});
it('handles a doubled quote inside a quoted identifier', () => {
expect(
splitPostgresStatements('SELECT "we""ird;x" FROM t; SELECT 2'),
).toEqual(['SELECT "we""ird;x" FROM t', 'SELECT 2']);
});
it('ignores semicolons inside line comments', () => {
expect(
splitPostgresStatements('SELECT 1 -- trailing; note\n; SELECT 2'),
).toEqual(['SELECT 1 -- trailing; note', 'SELECT 2']);
});
it('ignores semicolons inside block comments', () => {
expect(
splitPostgresStatements('SELECT /* a; b */ 1; SELECT 2'),
).toEqual(['SELECT /* a; b */ 1', 'SELECT 2']);
});
it('keeps a whole dollar-quoted function body together', () => {
const sql = [
'CREATE FUNCTION bump() RETURNS trigger AS $$',
'BEGIN',
' NEW.updated_at := now();',
' RETURN NEW;',
'END;',
'$$ LANGUAGE plpgsql;',
'SELECT 1',
].join('\n');
const statements = splitPostgresStatements(sql);
expect(statements).toHaveLength(2);
expect(statements[0]).toContain('RETURN NEW;');
expect(statements[0].endsWith('$$ LANGUAGE plpgsql')).toBe(true);
expect(statements[1]).toBe('SELECT 1');
});
it('respects a tagged dollar quote and its nested untagged pair', () => {
const sql = 'SELECT $body$ inner $$ still; inside $body$; SELECT 2';
expect(splitPostgresStatements(sql)).toEqual([
'SELECT $body$ inner $$ still; inside $body$',
'SELECT 2',
]);
});
it('treats a bare dollar sign as ordinary text', () => {
expect(splitPostgresStatements('SELECT 1 $ 2; SELECT 3')).toEqual([
'SELECT 1 $ 2',
'SELECT 3',
]);
});
it('emits an unterminated final statement', () => {
expect(
splitPostgresStatements('SELECT 1; SELECT 2 -- no newline'),
).toEqual(['SELECT 1', 'SELECT 2 -- no newline']);
});
});
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -0,0 +1,531 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { createServer, type Server } from 'node:http';
import type { AddressInfo } from 'node:net';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import type { IConfig } from '../../types';
import { DDBClient } from './DDBClient';
const TABLE = 'kv-items';
const GSI = 'by-kind';
const localConfig = (): IConfig =>
({
port: 0,
extensions: [],
dynamo: { inMemory: true, bootstrapTables: true },
}) as unknown as IConfig;
/**
* The emulator reports a table as CREATING for a tick after `CreateTable`
* returns, so wait until it actually answers a read before using it.
*/
const awaitTable = async (
client: DDBClient,
table: string,
key: Record<string, unknown>,
): Promise<void> => {
for (let attempt = 0; attempt < 100; attempt++) {
try {
await client.get(table, key);
return;
} catch (error) {
if ((error as Error).name !== 'ResourceNotFoundException')
throw error;
await new Promise((resolve) => setTimeout(resolve, 20));
}
}
throw new Error(`table ${table} never became queryable`);
};
const tableSchema = {
TableName: TABLE,
KeySchema: [
{ AttributeName: 'pk', KeyType: 'HASH' as const },
{ AttributeName: 'sk', KeyType: 'RANGE' as const },
],
AttributeDefinitions: [
{ AttributeName: 'pk', AttributeType: 'S' as const },
{ AttributeName: 'sk', AttributeType: 'S' as const },
{ AttributeName: 'kind', AttributeType: 'S' as const },
],
GlobalSecondaryIndexes: [
{
IndexName: GSI,
KeySchema: [{ AttributeName: 'kind', KeyType: 'HASH' as const }],
Projection: { ProjectionType: 'ALL' as const },
ProvisionedThroughput: {
ReadCapacityUnits: 5,
WriteCapacityUnits: 5,
},
},
],
ProvisionedThroughput: { ReadCapacityUnits: 5, WriteCapacityUnits: 5 },
};
describe('DDBClient — item operations', () => {
let client: DDBClient;
beforeAll(async () => {
client = new DDBClient(localConfig());
await client.createTableIfNotExists(tableSchema);
await awaitTable(client, TABLE, { pk: 'ready', sk: 'ready' });
});
it('round-trips an item and reports consumed capacity', async () => {
await client.put(TABLE, { pk: 'u1', sk: 'profile', name: 'ada' });
const result = await client.get(TABLE, { pk: 'u1', sk: 'profile' });
expect(result.Item).toEqual({ pk: 'u1', sk: 'profile', name: 'ada' });
expect(result.ConsumedCapacity?.TableName).toBe(TABLE);
});
it('returns no Item for a key that does not exist', async () => {
const result = await client.get(TABLE, { pk: 'nobody', sk: 'here' });
expect(result.Item).toBeUndefined();
});
it('honours a strongly consistent read', async () => {
await client.put(TABLE, { pk: 'u2', sk: 'profile', name: 'grace' });
const result = await client.get(
TABLE,
{ pk: 'u2', sk: 'profile' },
true,
);
expect(result.Item?.name).toBe('grace');
});
it('applies an update expression and returns the new item', async () => {
await client.put(TABLE, { pk: 'u3', sk: 'counter', hits: 1 });
const updated = await client.update(
TABLE,
{ pk: 'u3', sk: 'counter' },
'SET #h = :next',
{ ':next': 5 },
{ '#h': 'hits' },
);
expect(updated.Attributes).toMatchObject({ hits: 5 });
});
it('supports an update expression with no values or names', async () => {
await client.put(TABLE, { pk: 'u4', sk: 'doc', stale: true });
const updated = await client.update(
TABLE,
{ pk: 'u4', sk: 'doc' },
'REMOVE stale',
);
expect(updated.Attributes).toEqual({ pk: 'u4', sk: 'doc' });
});
it('deletes an item', async () => {
await client.put(TABLE, { pk: 'u5', sk: 'temp' });
await client.del(TABLE, { pk: 'u5', sk: 'temp' });
const result = await client.get(TABLE, { pk: 'u5', sk: 'temp' });
expect(result.Item).toBeUndefined();
});
it('fetches keys from several tables in one batch', async () => {
await client.put(TABLE, { pk: 'b1', sk: 'a', v: 1 });
await client.put(TABLE, { pk: 'b1', sk: 'b', v: 2 });
const result = await client.batchGet([
{ table: TABLE, items: { pk: 'b1', sk: 'a' } },
{ table: TABLE, items: { pk: 'b1', sk: 'b' } },
]);
const values = (result.Responses?.[TABLE] ?? [])
.map((item) => item.v as number)
.sort();
expect(values).toEqual([1, 2]);
});
});
describe('DDBClient — queries', () => {
let client: DDBClient;
beforeAll(async () => {
client = new DDBClient(localConfig());
await client.createTableIfNotExists(tableSchema);
await awaitTable(client, TABLE, { pk: 'ready', sk: 'ready' });
for (const sk of ['post#1', 'post#2', 'post#3', 'note#1']) {
await client.put(TABLE, {
pk: 'q1',
sk,
kind: sk.split('#')[0],
size: sk === 'post#2' ? 10 : 1,
});
}
});
it('queries by partition key', async () => {
const result = await client.query(TABLE, { pk: 'q1' });
expect(result.Items).toHaveLength(4);
expect(result.ConsumedCapacity?.TableName).toBe(TABLE);
});
it('narrows the sort key with begins_with', async () => {
const result = await client.query(
TABLE,
{ pk: 'q1' },
0,
undefined,
'',
false,
{ beginsWith: { key: 'sk', value: 'post#' } },
);
expect(result.Items?.map((item) => item.sk)).toEqual([
'post#1',
'post#2',
'post#3',
]);
});
it('ignores an empty begins_with value', async () => {
const result = await client.query(
TABLE,
{ pk: 'q1' },
0,
undefined,
'',
false,
{ beginsWith: { key: 'sk', value: '' } },
);
expect(result.Items).toHaveLength(4);
});
it('applies a filter expression with its own names and values', async () => {
const result = await client.query(
TABLE,
{ pk: 'q1' },
0,
undefined,
'',
false,
{
filter: {
expression: '#s > :min',
names: { '#s': 'size' },
values: { ':min': 5 },
},
},
);
expect(result.Items?.map((item) => item.sk)).toEqual(['post#2']);
});
it('counts without returning items', async () => {
const result = await client.query(
TABLE,
{ pk: 'q1' },
0,
undefined,
'',
false,
{ select: 'COUNT' },
);
expect(result.Count).toBe(4);
expect(result.Items).toBeUndefined();
});
it('pages through results with the last evaluated key', async () => {
const seen: string[] = [];
let pageKey: Record<string, unknown> | undefined;
let pages = 0;
do {
const page = await client.query(TABLE, { pk: 'q1' }, 2, pageKey);
pages += 1;
expect(page.Items?.length).toBeLessThanOrEqual(2);
for (const item of page.Items ?? []) seen.push(item.sk as string);
pageKey = page.LastEvaluatedKey;
} while (pageKey && pages < 10);
expect(pages).toBeGreaterThan(1);
expect(new Set(seen).size).toBe(4);
});
it('queries a secondary index', async () => {
const result = await client.query(
TABLE,
{ kind: 'note' },
0,
undefined,
GSI,
);
expect(result.Items?.map((item) => item.sk)).toEqual(['note#1']);
});
});
describe('DDBClient — batch writes', () => {
let client: DDBClient;
beforeAll(async () => {
client = new DDBClient(localConfig());
await client.createTableIfNotExists(tableSchema);
await awaitTable(client, TABLE, { pk: 'ready', sk: 'ready' });
});
it('reports no consumed capacity for an empty batch', async () => {
await expect(client.batchPut([])).resolves.toEqual({
ConsumedCapacity: [],
});
});
it('splits a batch larger than the service limit into chunks', async () => {
const params = Array.from({ length: 60 }, (_, index) => ({
table: TABLE,
item: { pk: 'bulk', sk: `item-${index}`, index },
}));
const result = await client.batchPut(params);
expect(result.ConsumedCapacity).toHaveLength(1);
expect(result.ConsumedCapacity[0].TableName).toBe(TABLE);
expect(result.ConsumedCapacity[0].CapacityUnits).toBeGreaterThan(0);
const stored = await client.query(TABLE, { pk: 'bulk' });
expect(stored.Items).toHaveLength(60);
});
});
describe('DDBClient — expired item sweep', () => {
it('deletes only the items past their ttl when the table already existed', async () => {
const client = new DDBClient(localConfig());
const table = 'ttl-items';
const schema = {
TableName: table,
KeySchema: [{ AttributeName: 'pk', KeyType: 'HASH' as const }],
AttributeDefinitions: [
{ AttributeName: 'pk', AttributeType: 'S' as const },
],
ProvisionedThroughput: {
ReadCapacityUnits: 5,
WriteCapacityUnits: 5,
},
};
await client.createTableIfNotExists(schema, 'expireAt');
await awaitTable(client, table, { pk: 'ready' });
const now = Math.floor(Date.now() / 1000);
await client.put(table, { pk: 'stale', expireAt: now - 60 });
await client.put(table, { pk: 'fresh', expireAt: now + 3600 });
await client.put(table, { pk: 'eternal' });
// Second call finds the table in use, so the sweep runs.
await client.createTableIfNotExists(schema, 'expireAt');
expect((await client.get(table, { pk: 'stale' })).Item).toBeUndefined();
expect((await client.get(table, { pk: 'fresh' })).Item).toBeDefined();
expect((await client.get(table, { pk: 'eternal' })).Item).toBeDefined();
});
});
describe('DDBClient — expired item sweep on an empty table', () => {
it('completes without deleting anything', async () => {
const client = new DDBClient(localConfig());
const table = 'ttl-empty';
const schema = {
TableName: table,
KeySchema: [{ AttributeName: 'pk', KeyType: 'HASH' as const }],
AttributeDefinitions: [
{ AttributeName: 'pk', AttributeType: 'S' as const },
],
ProvisionedThroughput: {
ReadCapacityUnits: 5,
WriteCapacityUnits: 5,
},
};
await client.createTableIfNotExists(schema, 'expireAt');
await awaitTable(client, table, { pk: 'ready' });
await expect(
client.createTableIfNotExists(schema, 'expireAt'),
).resolves.toBeUndefined();
});
});
describe('DDBClient — credentialled configuration', () => {
it('refuses an aws config that is missing a key', () => {
expect(
() =>
new DDBClient({
port: 0,
extensions: [],
dynamo: { aws: { access_key: 'only-one' } },
} as unknown as IConfig),
).toThrow('requires both `access_key` and `secret_key`');
});
it('skips table creation on a managed deployment', async () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
const client = new DDBClient({
port: 0,
extensions: [],
dynamo: {
aws: { access_key: 'a', secret_key: 'b', region: 'us-west-2' },
endpoint: 'http://127.0.0.1:1',
},
} as unknown as IConfig);
// Would fail against the unreachable endpoint if it actually tried.
await client.createTableIfNotExists(tableSchema);
expect(warn).toHaveBeenCalledWith(
expect.stringContaining(
'Creating DynamoDB tables is disabled by default',
),
);
warn.mockRestore();
});
it('rebinds an aws-credentialled client on recreate', async () => {
const client = new DDBClient({
port: 0,
extensions: [],
dynamo: {
aws: { access_key: 'a', secret_key: 'b' },
endpoint: 'http://127.0.0.1:1',
},
} as unknown as IConfig);
await expect(client.recreateClient()).resolves.toBeUndefined();
});
it('rebinds the underlying client on recreate', async () => {
const client = new DDBClient(localConfig());
await client.createTableIfNotExists(tableSchema);
await awaitTable(client, TABLE, { pk: 'ready', sk: 'ready' });
await client.put(TABLE, { pk: 'recreate', sk: 'a', v: 1 });
await client.recreateClient();
// Same in-memory store: the recreated client sees earlier writes.
const result = await client.get(TABLE, { pk: 'recreate', sk: 'a' });
expect(result.Item?.v).toBe(1);
});
});
// Unprocessed items are a throttling signal the emulator never produces, so
// the retry loop is driven against a stub speaking the DynamoDB wire format.
describe('DDBClient — unprocessed batch items', () => {
let server: Server;
let endpoint: string;
let responses: unknown[];
let requestCount = 0;
beforeAll(async () => {
server = createServer((req, res) => {
let body = '';
req.on('data', (chunk) => {
body += chunk;
});
req.on('end', () => {
const next = responses.shift() ?? {};
requestCount += 1;
res.writeHead(200, {
'content-type': 'application/x-amz-json-1.0',
});
res.end(JSON.stringify(next));
});
});
await new Promise<void>((resolve) =>
server.listen(0, '127.0.0.1', resolve),
);
endpoint = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
});
afterAll(async () => {
await new Promise<void>((resolve) => server.close(() => resolve()));
});
const stubClient = () =>
new DDBClient({
port: 0,
extensions: [],
dynamo: {
aws: { access_key: 'a', secret_key: 'b', region: 'us-west-2' },
endpoint,
},
} as unknown as IConfig);
const unprocessed = (pk: string) => ({
UnprocessedItems: {
[TABLE]: [{ PutRequest: { Item: { pk: { S: pk } } } }],
},
ConsumedCapacity: [{ TableName: TABLE, CapacityUnits: 1 }],
});
it('retries the leftover items and sums capacity across attempts', async () => {
requestCount = 0;
responses = [
unprocessed('retry-me'),
{
UnprocessedItems: {},
ConsumedCapacity: [{ TableName: TABLE, CapacityUnits: 2 }],
},
];
const result = await stubClient().batchPut([
{ table: TABLE, item: { pk: 'retry-me' } },
]);
expect(requestCount).toBe(2);
expect(result.ConsumedCapacity).toEqual([
{ TableName: TABLE, CapacityUnits: 3 },
]);
});
it('gives up with a bad-request error when items never drain', async () => {
requestCount = 0;
responses = Array.from({ length: 12 }, () => unprocessed('stuck'));
await expect(
stubClient().batchPut([{ table: TABLE, item: { pk: 'stuck' } }]),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'bad_request',
message: 'Failed to batch write all items to DynamoDB',
});
// One initial attempt plus the full retry budget.
expect(requestCount).toBe(9);
}, 30_000);
it('tolerates a response that reports no capacity at all', async () => {
requestCount = 0;
responses = [{ UnprocessedItems: {} }];
await expect(
stubClient().batchPut([{ table: TABLE, item: { pk: 'quiet' } }]),
).resolves.toEqual({ ConsumedCapacity: [] });
expect(requestCount).toBe(1);
});
it('ignores capacity entries with no table name', async () => {
requestCount = 0;
responses = [
{
UnprocessedItems: {},
ConsumedCapacity: [{ CapacityUnits: 7 }],
},
];
await expect(
stubClient().batchPut([{ table: TABLE, item: { pk: 'anon' } }]),
).resolves.toEqual({ ConsumedCapacity: [] });
});
});
+10 -11
View File
@@ -1,21 +1,20 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify it under the
* terms of the GNU Affero General Public License as published by the Free
* Software Foundation, either version 3 of the License, or (at your option) any
* later version.
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
* details.
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import {
@@ -0,0 +1,322 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { describe, expect, it, vi } from 'vitest';
import type { IConfig } from '../../types';
import { EmailClient } from './EmailClient';
const FROM = '"Puter" <[email protected]>';
/**
* Nodemailer's JSON transport is a real transport that serializes the message
* instead of talking SMTP — the send path stays genuine while the wire stays
* local.
*/
const jsonTransportConfig = (overrides: Partial<IConfig> = {}): IConfig =>
({
port: 0,
extensions: [],
env: 'prod',
email: { jsonTransport: true, from: FROM },
...overrides,
}) as unknown as IConfig;
const startClient = (overrides: Partial<IConfig> = {}): EmailClient => {
const client = new EmailClient(jsonTransportConfig(overrides));
client.onServerStart();
return client;
};
/** The JSON transport reports the serialized message on `.message`. */
const sentMessage = (info: unknown) =>
JSON.parse((info as { message: string }).message) as {
from: { address: string; name: string };
to: { address: string }[];
subject: string;
html?: string;
headers?: Record<string, string>;
};
describe('EmailClient — transport lifecycle', () => {
it('reports as configured once a transport is wired up', () => {
const client = startClient();
expect(client.isConfigured).toBe(true);
client.onServerShutdown();
expect(client.isConfigured).toBe(false);
});
it('warns and stays unconfigured when no transport is set', () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
const client = new EmailClient({
port: 0,
extensions: [],
} as unknown as IConfig);
client.onServerStart();
expect(client.isConfigured).toBe(false);
expect(warn).toHaveBeenCalledWith(
expect.stringContaining('no email transport configured'),
);
warn.mockRestore();
});
it('drops the send and returns null when no transport exists', async () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
const client = new EmailClient({
port: 0,
extensions: [],
} as unknown as IConfig);
client.onServerStart();
await expect(
client.sendRaw({ to: '[email protected]', subject: 'hi', text: 'hi' }),
).resolves.toBeNull();
expect(warn).toHaveBeenCalledWith(
expect.stringContaining('without transport'),
expect.objectContaining({ to: '[email protected]' }),
);
warn.mockRestore();
});
});
describe('EmailClient — sending', () => {
it('sends a rendered template through the transport', async () => {
const client = startClient();
const info = await client.sendRaw({
to: '[email protected]',
subject: 'raw subject',
html: '<p>hello</p>',
headers: { 'List-Unsubscribe': '<https://puter.test/u>' },
});
const message = sentMessage(info);
expect(message.subject).toBe('raw subject');
expect(message.to[0].address).toBe('[email protected]');
expect(message.from.address).toBe('[email protected]');
expect(message.headers?.['List-Unsubscribe']).toBe(
'<https://puter.test/u>',
);
});
it('lets the caller override the from address', async () => {
const client = startClient();
const info = await client.sendRaw({
from: '[email protected]',
to: '[email protected]',
subject: 's',
text: 't',
});
expect(sentMessage(info).from.address).toBe('[email protected]');
});
it('falls back to the built-in sender when config has none', async () => {
const client = startClient({
email: { jsonTransport: true },
} as unknown as Partial<IConfig>);
const info = await client.sendRaw({
to: '[email protected]',
subject: 's',
text: 't',
});
expect(sentMessage(info).from.address).toBe('[email protected]');
});
it('refuses an unknown template name', async () => {
const client = startClient();
await expect(
client.send('[email protected]', 'not-a-template' as never),
).rejects.toThrow('Unknown email template: not-a-template');
});
it('surfaces transport failures to the caller', async () => {
// Port 1 on loopback is never listening, so the send really fails.
const client = startClient({
email: { host: '127.0.0.1', port: 1, from: FROM },
} as unknown as Partial<IConfig>);
await expect(
client.sendRaw({
to: '[email protected]',
subject: 's',
text: 't',
}),
).rejects.toThrow();
});
});
describe('EmailClient — template rendering', () => {
it('renders the code into both subject and body', async () => {
const client = startClient();
const captured: { subject?: string; html?: string } = {};
const original = client.sendRaw.bind(client);
vi.spyOn(client, 'sendRaw').mockImplementation(async (options) => {
captured.subject = options.subject;
captured.html = options.html;
return original(options);
});
await client.send('[email protected]', 'email_verification_code', {
code: '424242',
});
expect(captured.subject).toBe('424242 is your confirmation code');
expect(captured.html).toContain('<strong>424242</strong>');
});
it('does not HTML-escape values in the plain-text subject header', async () => {
const client = startClient();
const captured: { subject?: string } = {};
vi.spyOn(client, 'sendRaw').mockImplementation(async (options) => {
captured.subject = options.subject;
return null;
});
await client.send('[email protected]', 'app-user-feedback', {
owner_username: 'dev',
sender_username: 'user',
sender_email: null,
app_title: "Bob's App & Games",
app_name: 'bobs-app',
app_link: 'https://puter.example/app/bobs-app',
message: 'hi',
});
// A subject is not HTML — entities would render literally in the
// recipient's mail client.
expect(captured.subject).toBe("New user feedback for Bob's App & Games");
});
it('escapes html and converts newlines in nl2br values', async () => {
const client = startClient();
let html = '';
vi.spyOn(client, 'sendRaw').mockImplementation(async (options) => {
html = options.html ?? '';
return null;
});
await client.send('[email protected]', 'listing-rejected', {
app_name: 'demo',
app_title: 'Demo',
reason: 'first & "only"\n<b>line</b>',
});
expect(html).toContain(
'first &amp; &quot;only&quot;<br />&lt;b&gt;line&lt;/b&gt;',
);
});
it('renders an empty string for a missing nl2br value', async () => {
const client = startClient();
let html = '';
vi.spyOn(client, 'sendRaw').mockImplementation(async (options) => {
html = options.html ?? '';
return null;
});
await client.send('[email protected]', 'listing-rejected', {
app_name: 'demo',
app_title: 'Demo',
});
expect(html).toContain('<blockquote></blockquote>');
});
});
describe('EmailClient.clean', () => {
const clean = (email: string) => startClient().clean(email);
it('strips subaddressing on ordinary domains', () => {
expect(clean('[email protected]')).toBe('[email protected]');
});
it('ignores dots and subaddressing for gmail', () => {
expect(clean('[email protected]')).toBe('[email protected]');
});
it('canonicalizes googlemail onto gmail', () => {
expect(clean('[email protected]')).toBe('[email protected]');
});
it('keeps yahoo subaddressing, which yahoo treats as distinct', () => {
expect(clean('[email protected]')).toBe('[email protected]');
});
it('ignores dots for icloud aliases', () => {
expect(clean('[email protected]')).toBe('[email protected]');
});
it('returns anything that is not an address unchanged', () => {
expect(clean('not-an-email')).toBe('not-an-email');
expect(clean('@nolocal.test')).toBe('@nolocal.test');
});
});
describe('EmailClient.validate', () => {
it('waves everything through in a dev environment', async () => {
const client = startClient({ env: 'dev' } as Partial<IConfig>);
await expect(client.validate('[email protected]')).resolves.toBe(
true,
);
});
it('rejects a blocked domain suffix', async () => {
const client = startClient({
blockedEmailDomains: ['blocked.test'],
} as unknown as Partial<IConfig>);
await expect(client.validate('[email protected]')).resolves.toBe(
false,
);
await expect(client.validate('[email protected]')).resolves.toBe(
true,
);
});
it('matches the blocklist against the cleaned address', async () => {
const client = startClient({
blockedEmailDomains: ['@gmail.com'],
} as unknown as Partial<IConfig>);
await expect(
client.validate('[email protected]'),
).resolves.toBe(false);
});
it('lets a registered validator veto an address', async () => {
const client = startClient();
const seen: string[] = [];
client.addValidator((email) => {
seen.push(email);
return !email.startsWith('spam');
});
await expect(client.validate('[email protected]')).resolves.toBe(
false,
);
await expect(client.validate('[email protected]')).resolves.toBe(true);
// Validators always see the canonical form.
expect(seen).toEqual(['[email protected]', '[email protected]']);
});
it('supports asynchronous validators', async () => {
const client = startClient();
client.addValidator(async (email) => email !== '[email protected]');
await expect(client.validate('[email protected]')).resolves.toBe(false);
await expect(client.validate('[email protected]')).resolves.toBe(true);
});
});
+32 -21
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -110,10 +110,11 @@ const DOMAIN_ALIASES: Record<string, string> = {
/**
* Unified email client. Handles:
* - Template-based outbound mail (via `send`)
* - Raw nodemailer passthrough (via `sendRaw`)
* - Canonical-form normalization for dedup (via `clean`)
* - Policy + extensible validation (via `validate`)
*
* - Template-based outbound mail (via `send`)
* - Raw nodemailer passthrough (via `sendRaw`)
* - Canonical-form normalization for dedup (via `clean`)
* - Policy + extensible validation (via `validate`)
*/
export class EmailClient extends PuterClient {
private transport: ReturnType<typeof nodemailer.createTransport> | null =
@@ -152,12 +153,15 @@ export class EmailClient extends PuterClient {
// -- Public API: sending ------------------------------------------
/**
* Render a template and send it to `to`.
* Render a template and send it to `to`. `options.replyTo` sets the
* Reply-To header (e.g. so a recipient can respond to the originator of the
* message rather than the no-reply From address).
*/
async send<T extends EmailTemplateName>(
to: string,
template: T,
values: Record<string, unknown> = {},
options: { replyTo?: string } = {},
): Promise<void> {
const compiled = this.compiledTemplates[template];
if (!compiled) {
@@ -169,16 +173,17 @@ export class EmailClient extends PuterClient {
to,
subject: compiled.subject(values),
html: compiled.html(values),
...(options.replyTo ? { replyTo: options.replyTo } : {}),
});
}
/**
* Raw send — bypasses the template system. Useful for one-off
* admin emails that don't warrant a named template.
* Raw send — bypasses the template system. Useful for one-off admin emails
* that don't warrant a named template.
*
* Returns the transport's send result, or `null` when no transport
* is configured (the send is a no-op in that case — callers that
* must not silently drop mail should check `isConfigured` first).
* Returns the transport's send result, or `null` when no transport is
* configured (the send is a no-op in that case — callers that must not
* silently drop mail should check `isConfigured` first).
*/
async sendRaw(options: SendMailOptions) {
if (!this.transport) {
@@ -202,9 +207,9 @@ export class EmailClient extends PuterClient {
// -- Public API: clean / validate ---------------------------------
/**
* Normalize an email to its canonical form for dedup comparisons.
* Applies provider-specific rules (e.g. Gmail ignores dots in
* the local part) plus generic subaddressing removal.
* Normalize an email to its canonical form for dedup comparisons. Applies
* provider-specific rules (e.g. Gmail ignores dots in the local part) plus
* generic subaddressing removal.
*/
clean(email: string): string {
let [local, domain] = email.split('@');
@@ -233,9 +238,9 @@ export class EmailClient extends PuterClient {
}
/**
* Check whether an email is allowed to be used. Checks domain
* blocklist plus any registered validators (services can call
* `addValidator()` to register custom policy hooks).
* Check whether an email is allowed to be used. Checks domain blocklist
* plus any registered validators (services can call `addValidator()` to
* register custom policy hooks).
*/
async validate(email: string): Promise<boolean> {
if (this.config.env === 'dev') return true;
@@ -258,9 +263,8 @@ export class EmailClient extends PuterClient {
}
/**
* Register a custom validation hook. Services can call this
* during their startup to veto specific emails (e.g. a
* disposable-email service).
* Register a custom validation hook. Services can call this during their
* startup to veto specific emails (e.g. a disposable-email service).
*/
addValidator(fn: EmailValidator): void {
this.validators.push(fn);
@@ -287,7 +291,14 @@ export class EmailClient extends PuterClient {
private compileTemplates(): void {
for (const [name, template] of Object.entries(EMAIL_TEMPLATES)) {
this.compiledTemplates[name as EmailTemplateName] = {
subject: handlebars.compile(template.subject),
// Subjects are plain-text headers: HTML-escaping would put
// literal entities in front of the recipient (&amp; etc.).
// Header safety is handled elsewhere — the transport encodes
// newlines, and free-form values (e.g. app_title) collapse
// whitespace upstream.
subject: handlebars.compile(template.subject, {
noEscape: true,
}),
html: handlebars.compile(dedent(template.html)),
};
}
+25 -6
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -18,12 +18,12 @@
*/
/**
* Email template definitions. Keys are the template names; values are
* the Handlebars-compilable `subject` and `html` strings.
* Email template definitions. Keys are the template names; values are the
* Handlebars-compilable `subject` and `html` strings.
*
* Rendered values are supplied by callers of `EmailClient.send()`.
* Variables use standard Handlebars syntax: `{{var}}`, `{{#if cond}}…{{/if}}`,
* and the custom helper `{{{nl2br text}}}` for HTML-safe newline conversion.
* Rendered values are supplied by callers of `EmailClient.send()`. Variables
* use standard Handlebars syntax: `{{var}}`, `{{#if cond}}…{{/if}}`, and the
* custom helper `{{{nl2br text}}}` for HTML-safe newline conversion.
*/
export interface EmailTemplate {
@@ -75,6 +75,25 @@ Please update <a href="https://puter.com/app/{{app_name}}">{{app_title}}</a>.
<blockquote>{{nl2br message}}</blockquote>
<p>Best,<br />
The Puter Team
</p>
`,
},
'app-user-feedback': {
subject: 'New user feedback for {{app_title}}',
html: `
<p>Hi{{#if owner_username}} {{owner_username}}{{/if}},</p>
<p>
<strong>{{sender_username}}</strong>{{#if sender_email}} (<a href="mailto:{{sender_email}}">{{sender_email}}</a>){{/if}} sent feedback about <a href="{{app_link}}">{{app_title}}</a>:
</p>
<blockquote>{{{nl2br message}}}</blockquote>
{{#if sender_email}}<p>Just reply to this email to respond to them directly.</p>{{/if}}
<p>
You're receiving this because feedback is enabled for
<a href="{{app_link}}">{{app_title}}</a> — manage it in the
<a href="{{dev_center_link}}">Dev Center</a> under your app's settings.
</p>
<p>Best,<br />
The Puter Team
</p>
`,
},
+103 -2
View File
@@ -1,5 +1,6 @@
import {
afterAll,
afterEach,
beforeAll,
beforeEach,
describe,
@@ -7,6 +8,7 @@ import {
it,
vi,
} from 'vitest';
import { extensionStore } from '../../extensions.ts';
import { PuterServer } from '../../server.ts';
import { setupTestServer } from '../../testUtil.ts';
import type { EventClient } from './EventClient.js';
@@ -25,8 +27,7 @@ describe('EventClient', () => {
});
// Each test gets a fresh key so listeners registered by earlier tests
// never collide with later ones (the client has no public way to
// unsubscribe).
// never collide with later ones.
let key: string;
beforeEach(() => {
key = `test.${Math.random().toString(36).slice(2)}`;
@@ -184,6 +185,106 @@ describe('EventClient', () => {
});
});
describe('off', () => {
it('stops delivering to a removed listener', () => {
const kept = vi.fn();
const removed = vi.fn();
target.on(key, kept);
target.on(key, removed);
target.off(key, removed);
target.emit(key, {}, {});
expect(kept).toHaveBeenCalledTimes(1);
expect(removed).not.toHaveBeenCalled();
});
it('ignores a key nobody ever subscribed to', () => {
expect(() => target.off(key, vi.fn())).not.toThrow();
});
it('leaves the list alone when the listener was never registered', () => {
const listener = vi.fn();
target.on(key, listener);
target.off(key, vi.fn());
target.emit(key, {}, {});
expect(listener).toHaveBeenCalledTimes(1);
});
});
describe('hasListeners', () => {
it('is false for a key nobody subscribed to', () => {
expect(target.hasListeners(key)).toBe(false);
});
it('is true for an exact subscriber', () => {
target.on(key, vi.fn());
expect(target.hasListeners(key)).toBe(true);
});
it('is true when only a wildcard prefix matches', () => {
target.on(`${key}.*`, vi.fn());
expect(target.hasListeners(`${key}.child.deep`)).toBe(true);
});
it('is true for an extension-registered listener', () => {
extensionStore.events[key] = [vi.fn()];
try {
expect(target.hasListeners(key)).toBe(true);
} finally {
delete extensionStore.events[key];
}
});
it('goes back to false once the last listener is removed', () => {
const listener = vi.fn();
target.on(key, listener);
target.off(key, listener);
expect(target.hasListeners(key)).toBe(false);
});
});
describe('extension-registered listeners', () => {
afterEach(() => {
delete extensionStore.events[key];
});
it('fires alongside listeners registered on the client', () => {
const fromExtension = vi.fn();
const fromClient = vi.fn();
extensionStore.events[key] = [fromExtension];
target.on(key, fromClient);
target.emit(key, { n: 1 }, { source: 'test' });
expect(fromExtension).toHaveBeenCalledWith(
key,
{ n: 1 },
{ source: 'test' },
);
expect(fromClient).toHaveBeenCalledTimes(1);
});
it('is awaited by emitAndWait', async () => {
const order: string[] = [];
extensionStore.events[key] = [
async () => {
await new Promise((resolve) => setTimeout(resolve, 5));
order.push('extension');
},
];
target.on(key, () => {
order.push('client');
});
await target.emitAndWait(key, {}, {});
expect(order).toEqual(['client', 'extension']);
});
});
describe('lifecycle hooks', () => {
it('onServerStart emits a serverStart event', () => {
const listener = vi.fn();
+46 -28
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -44,20 +44,19 @@ export class EventClient extends PuterClient {
/**
* Dispatch an event to every matching subscriber.
*
* Match semantics: emit walks every dot-separated prefix of `key`,
* looking up `<prefix>.*` listeners for prefixes shorter than the full
* key, and exact-key listeners on the final iteration. So emitting
* Match semantics: emit walks every dot-separated prefix of `key`, looking
* up `<prefix>.*` listeners for prefixes shorter than the full key, and
* exact-key listeners on the final iteration. So emitting
* `outer.gui.item.removed` fires subscribers on:
*
* - `outer.*`
* - `outer.gui.*`
* - `outer.gui.item.*`
* - `outer.gui.item.removed`
*
* Subscribers are still keyed in a single map — wildcards just live
* under their literal `<prefix>.*` string. No regex, no per-emit
* scan of every listener.
* - `outer.*`
* - `outer.gui.*`
* - `outer.gui.item.*`
* - `outer.gui.item.removed`
*
* Subscribers are still keyed in a single map — wildcards just live under
* their literal `<prefix>.*` string. No regex, no per-emit scan of every
* listener.
*/
emit<T extends keyof EventMap>(
key: T,
@@ -85,15 +84,15 @@ export class EventClient extends PuterClient {
/**
* Like `emit`, but awaits every matched listener before resolving.
*
* Use this when the emitter needs to act on mutations the handlers made
* to `data` — e.g. validation hooks where a listener can set
* `data.allow = false` to reject, or pre-commit pipelines where every
* stage must complete before the next step runs. Regular `emit` is
* fire-and-forget and can't observe handler state changes.
* Use this when the emitter needs to act on mutations the handlers made to
* `data` — e.g. validation hooks where a listener can set `data.allow =
* false` to reject, or pre-commit pipelines where every stage must complete
* before the next step runs. Regular `emit` is fire-and-forget and can't
* observe handler state changes.
*
* Listeners run sequentially in the order they're registered so an
* earlier handler's mutation is visible to later ones. A listener that
* throws is logged (same as `emit`) and the chain continues.
* Listeners run sequentially in the order they're registered so an earlier
* handler's mutation is visible to later ones. A listener that throws is
* logged (same as `emit`) and the chain continues.
*/
async emitAndWait<T extends keyof EventMap>(
key: T,
@@ -130,18 +129,37 @@ export class EventClient extends PuterClient {
});
}
/**
* Whether anything would run if `key` were emitted, wildcards included.
*
* For emitters whose work is only worth doing when someone is listening —
* gathering the payload costs a round trip, say. Emitting into the void is
* otherwise perfectly cheap and doesn't need this.
*/
hasListeners<T extends keyof EventMap>(key: T): boolean {
const parts = key.split('.');
for (let i = 0; i < parts.length; i++) {
const matchKey = (
i === parts.length - 1
? key
: `${parts.slice(0, i + 1).join('.')}.*`
) as ListenKey;
if (this.#eventListeners[matchKey]?.length) return true;
if (extensionStore.events[matchKey]?.length) return true;
}
return false;
}
/**
* Subscribe to an event by exact key OR a wildcard prefix.
*
* Wildcards: a key ending in `.*` matches every event whose name
* starts with the prefix. `outer.*` matches `outer.gui.item.removed`,
* `outer.fs.write-hash`, and any other dot-extended descendant.
* Exact keys still match exactly. See `emit()` for the dispatch order.
*
* Callback receives the full `(key, data, meta)` tuple as passed
* to `emit()` — wildcard subscribers can branch on the triggering
* event name.
* Wildcards: a key ending in `.*` matches every event whose name starts
* with the prefix. `outer.*` matches `outer.gui.item.removed`,
* `outer.fs.write-hash`, and any other dot-extended descendant. Exact keys
* still match exactly. See `emit()` for the dispatch order.
*
* Callback receives the full `(key, data, meta)` tuple as passed to
* `emit()` — wildcard subscribers can branch on the triggering event name.
*/
on<P extends ListenKey>(
key: P,
+58
View File
@@ -20,6 +20,7 @@ import type {
Response as ExpressResponse,
} from 'express';
import { Actor } from '../../core';
import type { UsageInput } from '../../services/metering/types';
import { FSEntry } from '../../stores/fs/FSEntry';
// GUI write events spread an entry plus per-event metadata into `response`.
@@ -305,6 +306,38 @@ export type EventMap = {
};
'storage.quota.bonus': { userId: number; extra: number };
// ---- Metering ----
// Recurring charges are pure mechanism here: the metering service knows
// when to ask (once per user per month, the first time that month's usage
// record is touched) and how to record the answer, and an extension
// decides what the account owes. Listeners push onto `charges`; the
// service applies them as ordinary usage once every listener has run.
// Nothing listening → no charges and no claim is ever taken.
//
// Emitted after the claim is settled, so a listener that calls back into
// metering can't re-trigger it.
'metering.monthly.charges': {
/**
* User-scoped, with no app on it: the account is what recurs, and the
* app that happened to trigger the month's first call has nothing to do
* with what is owed. Price against what the user owns.
*/
actor: Actor;
/** The month being charged for, `YYYY-MM` in UTC. */
month: string;
/**
* Push what the user owes here. Every listener's charges are merged
* into one amount map and recorded as a single increment.
*/
charges: UsageInput[];
};
// ---- Workers ----
// Only a genuinely new worker. Redeploying an existing name updates its
// row instead, and never reaches here — so a listener that prices this
// is pricing workers that came into existence, not deploys.
'worker.create': { actor: Actor; workerName: string };
// ---- Outer / GUI broadcast ----
'outer.cacheUpdate': {
cacheKey: string[];
@@ -312,6 +345,15 @@ export type EventMap = {
ttlSeconds?: number;
};
'outer.fs.write-hash': { hash: string; uuid: string };
/**
* Cache keys the KV read cache must stop serving, because the entries
* behind them were just written somewhere else.
*
* `outer.*` rather than `outer.pubsub.*` on purpose: the cache lives in the
* Redis a cluster shares, so one node applying the invalidation covers the
* whole cluster — fanning it out to siblings would just repeat the write.
*/
'outer.kv.cacheInvalidated': { cacheKeys: string[] };
'outer.gui.item.added': GuiEvent;
'outer.gui.item.updated': GuiEvent;
'outer.gui.item.moved': GuiEvent;
@@ -402,6 +444,22 @@ export type EventMap = {
[K in `route.${string}`]: RouteLifecycleEvent;
} & {
[K in `pubsub.login.${string}`]: { authtoken: string };
} & {
/**
* A user's subscription now resolves to a different policy. Carried on the
* `outer.pubsub.*` channel so it reaches sibling nodes and peer clusters,
* not just the one that handled the change — every node caches the resolved
* policy, so a purchase is only live once they have all dropped theirs.
*/
'outer.pubsub.metering.subscription-changed': { userUuid: string };
/**
* A user's purchased credit balance changed. Separate from a policy change
* because it moves the other half of the same budget, and carried on the
* `outer.pubsub.*` channel for the same reason: every node caches whether
* an account has budget left, so a top-up only lifts enforcement once they
* have all dropped that answer.
*/
'outer.pubsub.metering.credits-changed': { userUuid: string };
};
/**
+1 -1
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
+12 -8
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -74,11 +74,11 @@ export type PreludeCheckStatus =
| 'transaction_mismatch';
/**
* Prelude Verify v2 client (https://docs.prelude.so/verify/v2). Sends and checks
* SMS one-time codes — Prelude generates, delivers, and validates the code, so
* we never store one ourselves. No-ops with a warning when no API key is
* configured (so dev environments don't crash); `isConfigured()` lets callers
* surface a clean "phone verification unavailable" error instead.
* Prelude Verify v2 client (https://docs.prelude.so/verify/v2). Sends and
* checks SMS one-time codes — Prelude generates, delivers, and validates the
* code, so we never store one ourselves. No-ops with a warning when no API key
* is configured (so dev environments don't crash); `isConfigured()` lets
* callers surface a clean "phone verification unavailable" error instead.
*/
export class PreludeClient extends PuterClient {
private apiKey: string | null = null;
@@ -113,7 +113,9 @@ export class PreludeClient extends PuterClient {
/**
* Whether SMS verification should be offered for a country. False when the
* country is unknown, has no SMS channel, or its rate exceeds the cost cap.
* country is unknown, has no SMS channel, or its rate exceeds the cost
* cap.
*
* @param iso ISO-3166 alpha-2 (e.g. 'US') — from the parsed phone number.
*/
isCountrySupported(iso: string | undefined): boolean {
@@ -174,7 +176,9 @@ export class PreludeClient extends PuterClient {
}
/**
* Check a code the user entered against the active verification for `target`.
* Check a code the user entered against the active verification for
* `target`.
*
* @returns `{ status }` — `'success'` means verified.
*/
async checkVerification(
+5 -3
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -17,11 +17,13 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/** Per-country Prelude SMS pricing, hardcoded from the Prelude global price
/**
* Per-country Prelude SMS pricing, hardcoded from the Prelude global price
* sheet. `sms` is the average cost in EUR, or null when SMS is unavailable
* there. Used to cap phone verification to affordable countries (see
* PreludeClient.isCountrySupported). Regenerate from the price sheet when
* Prelude updates rates. */
* Prelude updates rates.
*/
export interface CountryPrice {
name: string;
/** Prelude SMS average in EUR, or null when SMS is not available. */
@@ -0,0 +1,171 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterEach, describe, expect, it, vi } from 'vitest';
import type { IConfig } from '../../types';
import { RedisClient } from './RedisClient';
const config = (redis: Record<string, unknown>): IConfig =>
({ port: 0, extensions: [], redis }) as unknown as IConfig;
type ClusterOptions = {
redisOptions?: { tls?: unknown; connectTimeout?: number };
clusterRetryStrategy?: (attempts: number) => number;
};
const optionsOf = (client: RedisClient): ClusterOptions =>
(client as unknown as { options: ClusterOptions }).options;
describe('RedisClient — in-process mock', () => {
let client: RedisClient | null = null;
afterEach(async () => {
if (client) await client.onServerShutdown();
client = null;
});
it('falls back to the mock when no startup nodes are configured', async () => {
client = new RedisClient(config({}));
await client.set('greeting', 'hello');
await expect(client.get('greeting')).resolves.toBe('hello');
});
it('honours an explicit mock request even with startup nodes present', async () => {
client = new RedisClient(
config({
useMock: true,
startupNodes: [{ host: 'redis.internal', port: 6379 }],
}),
);
await client.set('k', 'v');
await expect(client.get('k')).resolves.toBe('v');
});
it('exposes the shutdown hook directly on the cluster instance', () => {
client = new RedisClient(config({}));
expect(typeof client.onServerShutdown).toBe('function');
});
it('disconnects when a clean quit fails', async () => {
const instance = new RedisClient(config({}));
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
vi.spyOn(instance, 'quit').mockRejectedValue(new Error('quit refused'));
const disconnect = vi.spyOn(instance, 'disconnect');
await instance.onServerShutdown();
expect(disconnect).toHaveBeenCalledTimes(1);
expect(warn).toHaveBeenCalledWith(
'[redis] failed to quit redis client cleanly',
expect.any(Error),
);
warn.mockRestore();
});
});
describe('RedisClient — real cluster wiring', () => {
let client: RedisClient | null = null;
afterEach(() => {
client?.disconnect();
client = null;
});
const startupNodes = [{ host: '127.0.0.1', port: 7001 }];
it('enables TLS by default and bounds the connect timeout', () => {
client = new RedisClient(config({ startupNodes }));
expect(optionsOf(client).redisOptions?.tls).toEqual({});
expect(optionsOf(client).redisOptions?.connectTimeout).toBe(10000);
});
it('lets a plain-TCP deployment opt out of TLS', () => {
client = new RedisClient(config({ startupNodes, tls: false }));
expect(optionsOf(client).redisOptions?.tls).toBeUndefined();
});
it('backs off on cluster retries up to a ceiling', () => {
client = new RedisClient(config({ startupNodes, tls: false }));
const retry = optionsOf(client).clusterRetryStrategy!;
expect(retry(0)).toBe(100);
expect(retry(5)).toBe(600);
expect(retry(100)).toBe(2000);
});
it('treats startup churn as a warning and everything else as an error', () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
const error = vi.spyOn(console, 'error').mockImplementation(() => {});
client = new RedisClient(config({ startupNodes, tls: false }));
client.emit('error', new Error('None of startup nodes is available'));
expect(warn).toHaveBeenCalledWith(
expect.stringContaining('startup issue while connecting'),
);
const fatal = new Error('WRONGPASS invalid credentials');
client.emit('error', fatal);
expect(error).toHaveBeenCalledWith('[redis] cluster error', fatal);
warn.mockRestore();
error.mockRestore();
});
it('applies the same triage to per-node errors', () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
const error = vi.spyOn(console, 'error').mockImplementation(() => {});
client = new RedisClient(config({ startupNodes, tls: false }));
const transient = new Error('boom');
transient.name = 'ClusterAllFailedError';
client.emit('node error', transient, '127.0.0.1:7001');
expect(warn).toHaveBeenCalledWith(
expect.stringContaining(
'startup issue for cluster node 127.0.0.1:7001',
),
);
const fatal = new Error('node exploded');
client.emit('node error', fatal, '127.0.0.1:7002');
expect(error).toHaveBeenCalledWith(
'[redis] cluster node error (127.0.0.1:7002)',
fatal,
);
// Non-Error payloads are stringified rather than crashing the handler.
client.emit('node error', 'None of startup nodes is available', 'n3');
expect(warn).toHaveBeenCalledWith(
expect.stringContaining('startup issue for cluster node n3'),
);
warn.mockRestore();
error.mockRestore();
});
it('logs the transport lifecycle once each', () => {
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
client = new RedisClient(config({ startupNodes, tls: false }));
client.emit('connect');
client.emit('ready');
expect(log).toHaveBeenCalledWith('[redis] cluster transport connected');
expect(log).toHaveBeenCalledWith('[redis] cluster ready');
log.mockRestore();
});
});
+7 -7
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -108,13 +108,13 @@ const buildCluster = (config: IConfig): Cluster => {
/**
* `RedisClient` IS the ioredis `Cluster` instance — consumers call
* `this.clients.redis.get(...)` / `.set(...)` directly rather than
* going through an inner `.client` field. Lifecycle methods
* (`onServerShutdown`) are attached onto the cluster instance itself.
* `this.clients.redis.get(...)` / `.set(...)` directly rather than going
* through an inner `.client` field. Lifecycle methods (`onServerShutdown`) are
* attached onto the cluster instance itself.
*
* Type-wise, `RedisClient` is `Cluster & WithLifecycle`; the registry-
* facing value below is a constructor that returns that shape. Mirrors
* the `DatabaseClientFactory` pattern.
* Type-wise, `RedisClient` is `Cluster & WithLifecycle`; the registry- facing
* value below is a constructor that returns that shape. Mirrors the
* `DatabaseClientFactory` pattern.
*/
export type RedisClient = Cluster & WithLifecycle;
+489
View File
@@ -0,0 +1,489 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { GetObjectCommand } from '@aws-sdk/client-s3';
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { existsSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import type { IConfig } from '../../types';
import { S3Client } from './S3Client';
const LOCAL_BUCKET = 'puter-local';
/**
* The multipart helper takes its command sender as a parameter, so the failure
* paths can be driven without a live endpoint.
*/
type S3Internals = {
awsConfig: { endpoint?: string };
migrateLegacyStorage(opts?: {
bucket?: string;
legacyPath?: string;
}): Promise<{ migratedFileCount: number; scannedEntryCount: number }>;
uploadMultipart(args: {
bucket: string;
client: { send: (command: unknown) => Promise<unknown> };
filePath: string;
fileSize: number;
key: string;
}): Promise<void>;
};
const internals = (client: S3Client) => client as unknown as S3Internals;
const localConfig = (): IConfig =>
({
port: 0,
extensions: [],
s3: { localConfig: { inMemory: true } },
}) as unknown as IConfig;
const startLocal = async (): Promise<S3Client> => {
const client = new S3Client(localConfig());
await client.onServerStart();
return client;
};
const objectBody = async (
client: S3Client,
key: string,
): Promise<string | undefined> => {
const result = await client
.get()
.send(new GetObjectCommand({ Bucket: LOCAL_BUCKET, Key: key }));
return result.Body?.transformToString();
};
describe('S3Client — local development endpoint', () => {
let client: S3Client;
beforeEach(async () => {
client = await startLocal();
});
afterEach(async () => {
await client.onServerShutdown();
});
it('points clients at the in-process object store', async () => {
await client
.get()
.send(
new GetObjectCommand({
Bucket: LOCAL_BUCKET,
Key: 'definitely-absent',
}),
)
.then(
() => {
throw new Error('expected a missing-key failure');
},
(error: { name?: string }) => {
expect(error.name).toBe('NoSuchKey');
},
);
});
it('reuses one client per region and mints a new one per region', () => {
const first = client.get('us-west-2');
expect(client.get('us-west-2')).toBe(first);
expect(client.get('eu-central-1')).not.toBe(first);
});
it('serves presigning from the same client when no public endpoint is set', () => {
expect(client.getForPresign('us-west-2')).toBe(client.get('us-west-2'));
});
it('keeps modest upload thresholds so the local store is not overwhelmed', () => {
expect(client.maxSingleUploadSize).toBe(10 * 1024 * 1024);
expect(client.partSize).toBe(5 * 1024 * 1024);
});
});
describe('S3Client — remote endpoint configuration', () => {
let local: S3Client;
let remote: S3Client | null = null;
beforeEach(async () => {
local = await startLocal();
});
afterEach(async () => {
if (remote) await remote.onServerShutdown();
remote = null;
await local.onServerShutdown();
});
const remoteConfig = (extra: Record<string, unknown>): IConfig =>
({
port: 0,
extensions: [],
s3: {
s3Config: {
endpoint: internals(local).awsConfig.endpoint,
accessKeyId: 'fakeAccessKeyId',
secretAccessKey: 'fakeSecretAccessKey',
region: 'us-west-2',
forcePathStyle: true,
...extra,
},
},
}) as unknown as IConfig;
it('talks to a configured S3-compatible endpoint', async () => {
remote = new S3Client(remoteConfig({}));
await remote.onServerStart();
await remote
.get()
.send(new GetObjectCommand({ Bucket: LOCAL_BUCKET, Key: 'nope' }))
.then(
() => {
throw new Error('expected a missing-key failure');
},
(error: { name?: string }) => {
expect(error.name).toBe('NoSuchKey');
},
);
});
it('signs browser-facing URLs against the public endpoint', async () => {
remote = new S3Client(
remoteConfig({ publicEndpoint: 'https://cdn.example.test' }),
);
await remote.onServerStart();
const presign = remote.getForPresign('us-west-2');
expect(presign).not.toBe(remote.get('us-west-2'));
// Cached per region like the regular client map.
expect(remote.getForPresign('us-west-2')).toBe(presign);
expect(remote.getForPresign('eu-central-1')).not.toBe(presign);
});
it('shares one client when the public endpoint matches the private one', async () => {
const endpoint = internals(local).awsConfig.endpoint;
remote = new S3Client(remoteConfig({ publicEndpoint: endpoint }));
await remote.onServerStart();
expect(remote.getForPresign('us-west-2')).toBe(remote.get('us-west-2'));
});
it('raises the upload thresholds when using the ambient credential chain', async () => {
remote = new S3Client({
port: 0,
extensions: [],
s3: { s3Config: { useCredentialChain: true } },
} as unknown as IConfig);
await remote.onServerStart();
expect(remote.partSize).toBe(64 * 1024 * 1024);
expect(remote.maxSingleUploadSize).toBe(128 * 1024 * 1024);
});
});
describe('S3Client — legacy storage migration', () => {
let client: S3Client;
let dir: string;
beforeEach(async () => {
client = await startLocal();
dir = mkdtempSync(join(tmpdir(), 'puter-s3-legacy-'));
});
afterEach(async () => {
rmSync(dir, { recursive: true, force: true });
await client.onServerShutdown();
});
it('does nothing when there is no legacy directory', async () => {
await expect(
internals(client).migrateLegacyStorage({
legacyPath: join(dir, 'absent'),
}),
).resolves.toEqual({ migratedFileCount: 0, scannedEntryCount: 0 });
});
it('uploads each legacy file, skips directories, then removes the tree', async () => {
const small = 'small-file-contents';
writeFileSync(join(dir, 'small.txt'), small);
writeFileSync(join(dir, 'other.txt'), 'other');
mkdirSync(join(dir, 'a-subdirectory'));
await expect(
internals(client).migrateLegacyStorage({ legacyPath: dir }),
).resolves.toEqual({ migratedFileCount: 2, scannedEntryCount: 3 });
await expect(objectBody(client, 'small.txt')).resolves.toBe(small);
await expect(objectBody(client, 'other.txt')).resolves.toBe('other');
expect(existsSync(dir)).toBe(false);
});
it('switches to a multipart upload for a file over the single-put limit', async () => {
// S3 requires every part but the last to be at least 5 MiB, so the
// fixture has to straddle that for real.
client.maxSingleUploadSize = 1024 * 1024;
client.partSize = 5 * 1024 * 1024;
const body = 'X'.repeat(6 * 1024 * 1024);
writeFileSync(join(dir, 'large.bin'), body);
await expect(
internals(client).migrateLegacyStorage({ legacyPath: dir }),
).resolves.toEqual({ migratedFileCount: 1, scannedEntryCount: 1 });
await expect(objectBody(client, 'large.bin')).resolves.toBe(body);
});
});
describe('S3Client — multipart failure handling', () => {
let client: S3Client;
let dir: string;
let filePath: string;
beforeEach(async () => {
client = await startLocal();
dir = mkdtempSync(join(tmpdir(), 'puter-s3-multipart-'));
filePath = join(dir, 'part-source.bin');
writeFileSync(filePath, 'Y'.repeat(30));
client.partSize = 10;
});
afterEach(async () => {
rmSync(dir, { recursive: true, force: true });
await client.onServerShutdown();
});
const commandName = (command: unknown) =>
(command as { constructor: { name: string } }).constructor.name;
it('fails when the store will not open a multipart upload', async () => {
const send = vi.fn(async () => ({}));
await expect(
internals(client).uploadMultipart({
bucket: LOCAL_BUCKET,
client: { send },
filePath,
fileSize: 30,
key: 'no-upload-id',
}),
).rejects.toThrow('Failed to start multipart upload');
});
it('aborts the upload when a part comes back without an ETag', async () => {
const sent: string[] = [];
const send = vi.fn(async (command: unknown) => {
const name = commandName(command);
sent.push(name);
if (name === 'CreateMultipartUploadCommand') {
return { UploadId: 'upload-1' };
}
if (name === 'UploadPartCommand') return {};
return {};
});
await expect(
internals(client).uploadMultipart({
bucket: LOCAL_BUCKET,
client: { send },
filePath,
fileSize: 30,
key: 'etag-less',
}),
).rejects.toMatchObject({ statusCode: 400, legacyCode: 'bad_request' });
expect(sent).toEqual([
'CreateMultipartUploadCommand',
'UploadPartCommand',
'AbortMultipartUploadCommand',
]);
});
it('sends one part per chunk and completes with every ETag', async () => {
const parts: { PartNumber: number; ContentLength: number }[] = [];
let completed: { ETag: string; PartNumber: number }[] | undefined;
const send = vi.fn(async (command: unknown) => {
const name = commandName(command);
const input = (command as { input: Record<string, unknown> }).input;
if (name === 'CreateMultipartUploadCommand') {
return { UploadId: 'upload-1' };
}
if (name === 'UploadPartCommand') {
parts.push({
PartNumber: input.PartNumber as number,
ContentLength: input.ContentLength as number,
});
return { ETag: `etag-${input.PartNumber}` };
}
if (name === 'CompleteMultipartUploadCommand') {
completed = (
input.MultipartUpload as {
Parts: { ETag: string; PartNumber: number }[];
}
).Parts;
}
return {};
});
client.partSize = 12;
await internals(client).uploadMultipart({
bucket: LOCAL_BUCKET,
client: { send },
filePath,
fileSize: 30,
key: 'chunked',
});
expect(parts).toEqual([
{ PartNumber: 1, ContentLength: 12 },
{ PartNumber: 2, ContentLength: 12 },
{ PartNumber: 3, ContentLength: 6 },
]);
expect(completed).toEqual([
{ ETag: 'etag-1', PartNumber: 1 },
{ ETag: 'etag-2', PartNumber: 2 },
{ ETag: 'etag-3', PartNumber: 3 },
]);
});
it('aborts and rethrows when completing the upload fails', async () => {
const sent: string[] = [];
const send = vi.fn(async (command: unknown) => {
const name = commandName(command);
sent.push(name);
if (name === 'CreateMultipartUploadCommand') {
return { UploadId: 'upload-1' };
}
if (name === 'UploadPartCommand') return { ETag: 'etag' };
if (name === 'CompleteMultipartUploadCommand') {
throw new Error('complete rejected');
}
return {};
});
await expect(
internals(client).uploadMultipart({
bucket: LOCAL_BUCKET,
client: { send },
filePath,
fileSize: 30,
key: 'incomplete',
}),
).rejects.toThrow('complete rejected');
expect(sent.at(-1)).toBe('AbortMultipartUploadCommand');
});
});
describe('S3Client — disk-backed local store', () => {
let cwd: string;
let dir: string;
let client: S3Client | null = null;
beforeEach(() => {
cwd = process.cwd();
dir = mkdtempSync(join(tmpdir(), 'puter-s3-disk-'));
});
afterEach(async () => {
if (client) await client.onServerShutdown();
client = null;
process.chdir(cwd);
rmSync(dir, { recursive: true, force: true });
});
it('persists to the configured directories and drains legacy storage', async () => {
// The legacy sweep resolves `storage/` against the working
// directory, so the whole fixture lives in a throwaway cwd.
process.chdir(dir);
mkdirSync(join(dir, 'storage'));
writeFileSync(join(dir, 'storage', 'legacy.txt'), 'from-disk');
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
client = new S3Client({
port: 0,
extensions: [],
s3: {
localConfig: {
host: '127.0.0.1',
port: 0,
dataDir: join(dir, 'fauxqs-data'),
s3StorageDir: join(dir, 'fauxqs-s3'),
},
},
} as unknown as IConfig);
await client.onServerStart();
expect(log).toHaveBeenCalledWith(
'[s3] migrated 1 file(s) from legacy storage',
);
await expect(objectBody(client, 'legacy.txt')).resolves.toBe(
'from-disk',
);
expect(existsSync(join(dir, 'storage'))).toBe(false);
log.mockRestore();
});
});
describe('S3Client — short reads during multipart', () => {
let client: S3Client;
let dir: string;
beforeEach(async () => {
client = await startLocal();
dir = mkdtempSync(join(tmpdir(), 'puter-s3-short-'));
});
afterEach(async () => {
rmSync(dir, { recursive: true, force: true });
await client.onServerShutdown();
});
it('stops uploading parts once the file runs out early', async () => {
const filePath = join(dir, 'truncated.bin');
writeFileSync(filePath, 'Z'.repeat(20));
client.partSize = 10;
const parts: number[] = [];
const send = vi.fn(async (command: unknown) => {
const name = (command as { constructor: { name: string } })
.constructor.name;
const input = (command as { input: Record<string, unknown> }).input;
if (name === 'CreateMultipartUploadCommand') {
return { UploadId: 'upload-1' };
}
if (name === 'UploadPartCommand') {
parts.push(input.ContentLength as number);
return { ETag: `etag-${input.PartNumber}` };
}
return {};
});
// The caller's size is stale — the file is shorter than advertised.
await internals(client).uploadMultipart({
bucket: LOCAL_BUCKET,
client: { send },
filePath,
fileSize: 50,
key: 'truncated',
});
expect(parts).toEqual([10, 10]);
});
});
+21 -7
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -21,6 +21,7 @@ import {
AbortMultipartUploadCommand,
CompleteMultipartUploadCommand,
CreateMultipartUploadCommand,
HeadBucketCommand,
PutObjectCommand,
S3Client as AwsS3Client,
type S3ClientConfig,
@@ -184,8 +185,8 @@ export class S3Client extends PuterClient {
// ------------------------------------------------------------------
/**
* Get (or create) an S3Client for the given region.
* Clients are cached per-region for connection reuse.
* Get (or create) an S3Client for the given region. Clients are cached
* per-region for connection reuse.
*/
get(
region = this.config.s3_region || this.config.region || 'us-west-2',
@@ -213,10 +214,10 @@ export class S3Client extends PuterClient {
/**
* Client used to generate browser-facing presigned URLs. When
* `s3Config.publicEndpoint` is set, this returns a client bound to
* that endpoint — its signatures resolve against the public host
* the browser will actually hit. When unset, falls back to the
* regular client (prod behavior: one public endpoint everywhere).
* `s3Config.publicEndpoint` is set, this returns a client bound to that
* endpoint — its signatures resolve against the public host the browser
* will actually hit. When unset, falls back to the regular client (prod
* behavior: one public endpoint everywhere).
*/
getForPresign(
region = this.config.s3_region || this.config.region || 'us-west-2',
@@ -244,6 +245,19 @@ export class S3Client extends PuterClient {
return client;
}
/**
* Cheapest round-trip that proves the object store answers for a bucket: no
* object data, no listing, just a HEAD. Throws on any failure (missing
* bucket, bad credentials, unreachable endpoint) so callers can treat it as
* a liveness probe. Reuses the pooled per-region client.
*/
async headBucket(
bucket = this.config.s3_bucket || LEGACY_STORAGE_BUCKET,
region?: string,
): Promise<void> {
await this.get(region).send(new HeadBucketCommand({ Bucket: bucket }));
}
// ------------------------------------------------------------------
// Legacy storage migration
// ------------------------------------------------------------------
+3 -3
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -45,8 +45,8 @@ export interface IExtensionClientInstances {
/**
* Optional ClickHouse analytics client. Absent by default — a production
* deployment registers it via an extension to speed up the app-stats path
* at scale (see {@link ClickhouseClient}). Always branch on its presence
* and fall back to SQL.
* at scale (see {@link ClickhouseClient}). Always branch on its presence and
* fall back to SQL.
*/
clickhouse?: ClickhouseClient;
}
+136 -39
View File
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -31,15 +31,99 @@ import DEFAULT_APP_ICON from './default-app-icon.js';
/**
* REST endpoints for app management.
*
* Delegates to AppDriver for the actual CRUD + permission logic —
* these routes are just thin shape adapters that translate REST
* conventions into driver calls.
* Delegates to AppDriver for the actual CRUD + permission logic — these routes
* are just thin shape adapters that translate REST conventions into driver
* calls.
*/
/**
* Desktop boot reads the app list and individual app records repeatedly, so the
* ceiling is set well above normal boot traffic and exists to catch a runaway
* client rather than to pace one.
*
* "Repeatedly" is the operative word: an app record is read on launch, on
* permission checks, and again by anything resolving an app by name, so these
* accumulate against whatever else a session is doing rather than arriving on
* their own. Sized for a session working hard, not for a person clicking.
*/
const APP_READ_LIMIT = {
scope: 'app-read',
limit: 1_800,
window: 60_000,
key: 'user',
};
/**
* Unauthenticated icon serving; no actor to key on, so the bucket is the
* address — and an address is a NAT, a campus or a carrier gateway that can
* hold hundreds of desktops. Each desktop boot pulls the taskbar's icons at
* once, so a single burst from one network is already thousands of requests.
* Responses are publicly cacheable and usually a redirect, so the ceiling is
* not protecting bandwidth; it is there so a client looping on a broken icon
* can't spin unbounded.
*/
const APP_ICON_LIMIT = {
scope: 'app-icon',
limit: 12_000,
window: 60_000,
key: 'ip',
};
export class AppController extends PuterController {
get appStore() {
return this.stores.app;
}
// In-flight background app-open writes. Tracked only so tests and
// shutdown can wait for them — the request path never does.
#pendingOpenWrites = new Set();
/**
* Record an app open. `app_opens` is analytics: it backs the recent-apps
* list and the open counters, and no response field is derived from it. The
* client posts this without awaiting and updates its own recent list
* optimistically, so holding a response open for a primary write only added
* latency to the launch that write is measuring.
*
* Failures are logged, never surfaced — a dropped stat must not turn into a
* failed app open.
*
* @param {string} appUid
* @param {number} userId
* @returns {Promise<void>} Settles when the write and event emit finish
*/
#recordAppOpen(appUid, userId) {
const ts = Math.floor(Date.now() / 1000);
const work = (async () => {
try {
await this.clients.db.write(
'INSERT INTO `app_opens` (`app_uid`, `user_id`, `ts`) VALUES (?, ?, ?)',
[appUid, userId, ts],
);
} catch (e) {
console.warn('[rao] insert failed:', e);
}
try {
this.clients.event?.emitAndWait(
'app.opened',
{ app_uid: appUid, user_id: userId, ts },
{},
);
} catch {
// event emission best-effort
}
})();
this.#pendingOpenWrites.add(work);
work.finally(() => this.#pendingOpenWrites.delete(work));
return work;
}
/** Await every in-flight app-open write. */
async drainPendingAppOpens() {
await Promise.allSettled([...this.#pendingOpenWrites]);
}
get appDriver() {
// Drivers are wired into the shared driversContainers export by
// PuterServer at boot. Controllers get them lazily via this getter
@@ -49,7 +133,9 @@ export class AppController extends PuterController {
return d;
}
registerRoutes(router) {
registerRoutes(
/** @type {import('../../core/http/PuterRouter').PuterRouter} */ router,
) {
// GET /apps — list apps owned by the current user
router.get(
'/apps',
@@ -57,6 +143,7 @@ export class AppController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: APP_READ_LIMIT,
},
async (req, res) => {
const apps = await this.appDriver.select({
@@ -71,8 +158,16 @@ export class AppController extends PuterController {
'/apps/nameAvailable',
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
requireAuth: true,
// Answers "does this name exist?" for any name, so it is a
// name-enumeration oracle however cheap it is to serve.
// Mirrors the `isNameAvailable` budget on AppDriver.
rateLimit: {
scope: 'app-name-available',
limit: 60,
window: 60_000,
key: 'user',
},
},
async (req, res) => {
const name = req.query?.name;
@@ -107,6 +202,7 @@ export class AppController extends PuterController {
{
subdomain: 'api',
requireAuth: true,
rateLimit: APP_READ_LIMIT,
},
async (req, res) => {
const actor = req.actor;
@@ -148,33 +244,10 @@ export class AppController extends PuterController {
legacyCode: 'not_found',
});
// Persist open record
try {
await this.clients.db.write(
'INSERT INTO `app_opens` (`app_uid`, `user_id`, `ts`) VALUES (?, ?, ?)',
[
app_uid,
req.actor.user.id,
Math.floor(Date.now() / 1000),
],
);
} catch (e) {
console.warn('[rao] insert failed:', e);
}
try {
this.clients.event?.emitAndWait(
'app.opened',
{
app_uid,
user_id: req.actor.user.id,
ts: Math.floor(Date.now() / 1000),
},
{},
);
} catch {
// event emission best-effort
}
// Validation and authorization are settled by this point, so
// the caller learns the outcome now and the stats write lands
// on its own. See `#recordAppOpen`.
this.#recordAppOpen(app_uid, req.actor.user.id);
res.json({});
},
@@ -188,6 +261,7 @@ export class AppController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: APP_READ_LIMIT,
},
async (req, res) => {
const raw = req.params.name;
@@ -214,7 +288,13 @@ export class AppController extends PuterController {
source: 'appsRoute',
args: req.query ?? {},
});
return { ...shaped, privateAccess };
return {
...shaped,
privateAccess:
shaped.privateAccess?.hasAccess === false
? shaped.privateAccess
: privateAccess,
};
}),
);
@@ -255,6 +335,7 @@ export class AppController extends PuterController {
{
subdomain: 'api',
requireAuth: true,
rateLimit: APP_READ_LIMIT,
},
async (req, res) => {
const appList = Array.isArray(req.body) ? req.body : [];
@@ -437,7 +518,11 @@ export class AppController extends PuterController {
}
setIconSecurityHeaders(res);
res.set('Content-Type', mime);
res.set('Cache-Control', 'public, max-age=60');
// Same freshness as the redirect path above — this is the
// same resource, just served inline because no CDN file
// exists yet. A 60s TTL made every app launch re-fetch the
// icon over a connection the launch itself is competing for.
res.set('Cache-Control', 'public, max-age=900');
res.send(Buffer.from(icon.slice(commaIdx + 1), 'base64'));
// Trigger background generation so next request hits the CDN
@@ -459,15 +544,27 @@ export class AppController extends PuterController {
// Icons are <img src> targets from the GUI (root) AND resolved via
// api_base_url in taskbar payloads. Register on both so either origin
// works without a cross-subdomain redirect.
router.get('/app-icon/:app_uid', { subdomain: ['api', ''] }, serveIcon);
router.get(
'/app-icon/:app_uid',
{ subdomain: ['api', ''], rateLimit: APP_ICON_LIMIT },
serveIcon,
);
router.get(
'/app-icon/:app_uid/:size',
{ subdomain: ['api', ''] },
{ subdomain: ['api', ''], rateLimit: APP_ICON_LIMIT },
serveIcon,
);
}
onServerStart() {}
onServerPrepareShutdown() {}
/**
* Let outstanding stats writes finish before the process goes away —
* otherwise a deploy silently drops every open recorded in the seconds
* before it.
*/
async onServerPrepareShutdown() {
await this.drainPendingAppOpens();
}
onServerShutdown() {}
}
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -91,6 +91,11 @@ const uniqueName = (prefix: string) =>
const uniqueIndexUrl = () =>
`https://example-${Math.random().toString(36).slice(2, 10)}.test/`;
// 1x1 transparent PNG. The `icon` write path validates the decoded payload,
// not just the declared MIME, so icon fixtures must be real images.
const MINIMAL_PNG_BASE64 =
'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII=';
const createApp = async (
actor: Actor,
overrides: Record<string, unknown> = {},
@@ -285,12 +290,7 @@ describe('AppController POST /rao', () => {
const { res } = makeRes();
await expect(
withActor(actor, () =>
callRoute(
'post',
'/rao',
makeReq({ body: {}, actor }),
res,
),
callRoute('post', '/rao', makeReq({ body: {}, actor }), res),
),
).rejects.toMatchObject({ statusCode: 400 });
});
@@ -328,6 +328,9 @@ describe('AppController POST /rao', () => {
);
expect(captured.body).toEqual({});
// The stats write is deliberately not awaited by the request.
await server.controllers.apps.drainPendingAppOpens();
const rows = (await server.clients.db.read(
'SELECT `app_uid`, `user_id` FROM `app_opens` WHERE `app_uid` = ? AND `user_id` = ?',
[app.uid, owner.userId],
@@ -335,6 +338,74 @@ describe('AppController POST /rao', () => {
expect(rows).toHaveLength(1);
});
// `app_opens` is analytics: the response carries nothing derived from it
// and the client never awaits the post. Holding the response open for a
// primary write put that write's latency inside the app launch it was
// measuring, on a connection the launch was competing for.
it('responds without waiting for the stats write', async () => {
const owner = await makeUser();
const app = await createApp(owner.actor);
let releaseWrite = () => {};
const writeSpy = vi
.spyOn(server.clients.db, 'write')
.mockImplementation(
() =>
new Promise((resolve) => {
releaseWrite = () => resolve(undefined as never);
}),
);
try {
const { res, captured } = makeRes();
await withActor(owner.actor, () =>
callRoute(
'post',
'/rao',
makeReq({ body: { app_uid: app.uid }, actor: owner.actor }),
res,
),
);
// The handler returned while the INSERT is still outstanding.
expect(captured.body).toEqual({});
expect(writeSpy).toHaveBeenCalled();
} finally {
releaseWrite();
await server.controllers.apps.drainPendingAppOpens();
writeSpy.mockRestore();
}
});
// A stats failure must never turn into a failed app open — and since the
// write is now unawaited, it must not surface as an unhandled rejection.
it('swallows a failing stats write without rejecting the request', async () => {
const owner = await makeUser();
const app = await createApp(owner.actor);
const writeSpy = vi
.spyOn(server.clients.db, 'write')
.mockRejectedValue(new Error('primary is down'));
try {
const { res, captured } = makeRes();
await withActor(owner.actor, () =>
callRoute(
'post',
'/rao',
makeReq({ body: { app_uid: app.uid }, actor: owner.actor }),
res,
),
);
expect(captured.body).toEqual({});
await expect(
server.controllers.apps.drainPendingAppOpens(),
).resolves.toBeUndefined();
} finally {
writeSpy.mockRestore();
}
});
it('falls back to actor.app.uid when the body omits app_uid', async () => {
const owner = await makeUser();
const app = await createApp(owner.actor);
@@ -354,6 +425,8 @@ describe('AppController POST /rao', () => {
);
expect(captured.body).toEqual({});
await server.controllers.apps.drainPendingAppOpens();
const rows = (await server.clients.db.read(
'SELECT `app_uid` FROM `app_opens` WHERE `app_uid` = ? AND `user_id` = ?',
[app.uid, owner.userId],
@@ -427,6 +500,73 @@ describe('AppController GET /apps/:name', () => {
expect((arr[0] as Record<string, unknown>).uid).toBe(app.uid);
expect(arr[1]).toBeNull();
});
// The driver is the authoritative producer of launch metadata: it runs
// the hosted-subdomain guard, which this route knows nothing about. The
// route's own entitlement re-check must never turn that denial into an
// affirmative verdict — the GUI launcher appends `puter.auth.token` to
// whatever `index_url` it is handed.
it('preserves the hosted-backing denial for the app owner', async () => {
const owner = await makeUser();
const sub = uniqueName('gone');
const row = await server.stores.subdomain.create({
userId: owner.userId,
subdomain: sub,
});
const app = await createApp(owner.actor, {
index_url: `https://${sub}.site.puter.localhost/`,
});
await server.stores.subdomain.deleteByUuid(
String((row as { uuid: string }).uuid),
{ userId: owner.userId },
);
const { res, captured } = makeRes();
await withActor(owner.actor, () =>
callRoute(
'get',
'/apps/:name',
makeReq({ params: { name: app.name }, actor: owner.actor }),
res,
),
);
const body = captured.body as Record<string, unknown>;
expect(body.privateAccess).toMatchObject({
hasAccess: false,
reason: 'hosted_backing_unavailable',
});
});
it('omits the stale index_url for a non-owner', async () => {
const owner = await makeUser();
const stranger = await makeUser();
const sub = uniqueName('gone');
const row = await server.stores.subdomain.create({
userId: owner.userId,
subdomain: sub,
});
const app = await createApp(owner.actor, {
index_url: `https://${sub}.site.puter.localhost/`,
});
await server.stores.subdomain.deleteByUuid(
String((row as { uuid: string }).uuid),
{ userId: owner.userId },
);
const { res, captured } = makeRes();
await withActor(stranger.actor, () =>
callRoute(
'get',
'/apps/:name',
makeReq({ params: { name: app.name }, actor: stranger.actor }),
res,
),
);
const body = captured.body as Record<string, unknown>;
expect(body.index_url).toBeUndefined();
expect(body.privateAccess).toMatchObject({ hasAccess: false });
});
});
// ── POST /query/app ─────────────────────────────────────────────────
@@ -649,7 +789,9 @@ describe('AppController GET /app-icon/:app_uid', () => {
it('decodes a data URL icon and serves the declared MIME', async () => {
const owner = await makeUser();
const png = Buffer.from('mock-png-bytes');
// Real PNG bytes: the write path sniffs the decoded payload and
// rejects anything that isn't the image type it claims to be.
const png = Buffer.from(MINIMAL_PNG_BASE64, 'base64');
const dataUrl = `data:image/png;base64,${png.toString('base64')}`;
const app = await createApp(owner.actor, { icon: dataUrl });
@@ -666,6 +808,25 @@ describe('AppController GET /app-icon/:app_uid', () => {
expect((captured.body as Buffer).equals(png)).toBe(true);
});
// Icons are fetched on every app launch, over a connection the launch is
// already competing for. The inline path must not be cached more briefly
// than the redirect path that serves the same resource.
it('caches an inline data-URL icon as long as the redirect path', async () => {
const owner = await makeUser();
const dataUrl = `data:image/png;base64,${MINIMAL_PNG_BASE64}`;
const app = await createApp(owner.actor, { icon: dataUrl });
const { res, captured } = makeRes();
await callRoute(
'get',
'/app-icon/:app_uid',
makeReq({ params: { app_uid: app.uid } }),
res,
);
expect(captured.headers['cache-control']).toBe('public, max-age=900');
});
it('falls back to the default icon when the data-URL MIME is not allowlisted', async () => {
const owner = await makeUser();
// text/html is NOT in the icon allowlist — must NOT be echoed back.
@@ -707,3 +868,175 @@ describe('AppController GET /app-icon/:app_uid', () => {
expect(captured.statusCode).toBe(200);
});
});
// ── /rao actor gating ───────────────────────────────────────────────
describe('AppController POST /rao actor gating', () => {
it('refuses to record an open for a scoped access token', async () => {
const { actor } = await makeUser();
const app = await createApp(actor);
// Shared / scoped credentials must not drive analytics counters.
const tokenActor = {
...actor,
accessToken: { uuid: uuidv4(), permissions: [] },
} as unknown as Actor;
const { res } = makeRes();
await expect(
withActor(tokenActor, () =>
callRoute(
'post',
'/rao',
makeReq({
body: { app_uid: app.uid },
actor: tokenActor,
}),
res,
),
),
).rejects.toMatchObject({ statusCode: 403, legacyCode: 'forbidden' });
});
it("refuses to let an app actor record another app's open", async () => {
const { actor } = await makeUser();
const own = await createApp(actor);
const other = await createApp(actor);
const appActor = {
...actor,
app: { uid: own.uid as string },
} as unknown as Actor;
const { res } = makeRes();
await expect(
withActor(appActor, () =>
callRoute(
'post',
'/rao',
makeReq({
body: { app_uid: other.uid },
actor: appActor,
}),
res,
),
),
).rejects.toMatchObject({ statusCode: 403, legacyCode: 'forbidden' });
});
});
// ── app-icon redirect + fallback paths ──────────────────────────────
describe('AppController GET /app-icon remote icons', () => {
// AppDriver.create validates the `icon` column, so these fixtures write
// it straight to the row. AppStore caches by uid — drop that entry or the
// handler reads the pre-update app back.
const setIcon = async (uid: string, icon: string) => {
await server.clients.db.write(
'UPDATE `apps` SET `icon` = ? WHERE `uid` = ?',
[icon, uid],
);
await server.stores.app.invalidateByUid(uid);
};
it('redirects to an icon hosted on a trusted host', async () => {
const owner = await makeUser();
const app = await createApp(owner.actor);
const { static_hosting_domain: hostingDomain } = (
server.controllers.apps as unknown as {
config: { static_hosting_domain: string };
}
).config;
const trusted = `https://puter-app-icons.${hostingDomain}/${app.uid}-128.png`;
await setIcon(String(app.uid), trusted);
const { res, captured } = makeRes();
await callRoute(
'get',
'/app-icon/:app_uid',
makeReq({ params: { app_uid: app.uid } }),
res,
);
expect(captured.redirectStatus).toBe(302);
expect(captured.redirectUrl).toBe(trusted);
expect(captured.headers['cache-control']).toBe('public, max-age=900');
});
it('never redirects to an icon URL on an untrusted host', async () => {
const owner = await makeUser();
const app = await createApp(owner.actor);
// An open redirect here would let an app row point browsers anywhere.
await setIcon(String(app.uid), 'https://evil.example.com/icon.png');
const { res, captured } = makeRes();
await callRoute(
'get',
'/app-icon/:app_uid',
makeReq({ params: { app_uid: app.uid } }),
res,
);
expect(captured.redirectUrl).toBeUndefined();
expect(captured.headers['content-type']).toContain('image/svg+xml');
});
it('serves the default icon for a malformed data URL with no comma', async () => {
const owner = await makeUser();
const app = await createApp(owner.actor);
await setIcon(String(app.uid), 'data:image/png;base64');
const { res, captured } = makeRes();
await callRoute(
'get',
'/app-icon/:app_uid',
makeReq({ params: { app_uid: app.uid } }),
res,
);
expect(captured.headers['content-type']).toContain('image/svg+xml');
});
it('serves the default icon for an unknown app uid', async () => {
const { res, captured } = makeRes();
await callRoute(
'get',
'/app-icon/:app_uid',
makeReq({ params: { app_uid: `app-${uuidv4()}` } }),
res,
);
expect(captured.headers['content-type']).toContain('image/svg+xml');
});
it('serves the default icon for a bare (non-URL, non-data) icon value', async () => {
const owner = await makeUser();
const app = await createApp(owner.actor);
await setIcon(String(app.uid), 'legacy-icon-name.png');
const { res, captured } = makeRes();
await callRoute(
'get',
'/app-icon/:app_uid/:size',
makeReq({ params: { app_uid: app.uid, size: '64' } }),
res,
);
expect(captured.headers['content-type']).toContain('image/svg+xml');
});
});
// ── app-icon rate limit ─────────────────────────────────────────────
describe('AppController app-icon rate limit', () => {
it('sizes the icon bucket for a whole network rather than one desktop', () => {
// Icons are unauthenticated `<img src>` targets, so the only key is
// the address — which one NAT shares across every desktop behind it,
// and each boot pulls the taskbar's icons in a burst.
for (const path of ['/app-icon/:app_uid', '/app-icon/:app_uid/:size']) {
const route = router.routes.find(
(r) => r.method === 'get' && r.path === path,
);
if (!route) throw new Error(`No GET ${path} route`);
expect(route.options.rateLimit).toEqual({
scope: 'app-icon',
limit: 12_000,
window: 60_000,
key: 'ip',
});
}
});
});
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -42,6 +42,7 @@ import type { TokenSource } from '../../core/http/types.js';
import { PuterServer } from '../../server.js';
import { FULL_API_ACCESS } from '../../services/permission/consts.js';
import { setupTestServer } from '../../testUtil.js';
import { FS_READ_LIMIT } from '../fs/limits.js';
// ── Test harness ────────────────────────────────────────────────────
@@ -955,6 +956,40 @@ describe('AuthController.handleLogin', () => {
expect(isCompleteLoginResponse(res.body)).toBe(true);
});
it('refuses an email the account has moved off', async () => {
const moverName = `lm_${Math.random().toString(36).slice(2, 10)}`;
const oldEmail = `${moverName}[email protected]`;
const newEmail = `${moverName}[email protected]`;
await controller.handleSignup(
makeReq({ username: moverName, email: oldEmail, password }),
makeRes(),
);
// Warm the by-email lookup the way a real login would.
await controller.handleLogin(
makeReq({ email: oldEmail, password }),
makeRes(),
);
const mover = await server.stores.user.getByUsername(moverName);
await server.stores.user.update(mover!.id, {
email: newEmail,
clean_email: newEmail,
});
await expect(
controller.handleLogin(
makeReq({ email: oldEmail, password }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 404 });
const res = makeRes();
await controller.handleLogin(
makeReq({ email: newEmail, password }),
res,
);
expect(isCompleteLoginResponse(res.body)).toBe(true);
});
it('returns 400 when neither username nor email is supplied', async () => {
await expect(
controller.handleLogin(makeReq({ password }), makeRes()),
@@ -1356,6 +1391,26 @@ describe('AuthController account-lifecycle route gating', () => {
expect(opts.requireUserActor).toBe(true);
});
// A fresh token is minted per protected mutation and nothing caches them,
// so issuance has to outrun the COMBINED rate of everything that spends
// one. The session-authenticated download path dominates — a multi-select
// download spends a token per file — with logout and the handful of
// session-management writes behind it.
it('GET /get-anticsrf-token clears the budgets that consume tokens', () => {
type Window = { limit: number; window: number };
const issuance = routeOptions('get', '/get-anticsrf-token')
.rateLimit as Window;
const logout = routeOptions('post', '/logout').rateLimit as Window;
expect(issuance.window).toBe(60_000);
expect(logout.window).toBe(60_000);
expect(FS_READ_LIMIT.window).toBe(60_000);
expect(issuance.limit).toBeGreaterThan(
FS_READ_LIMIT.limit + logout.limit,
);
});
it('requireUserActorGate rejects app-under-user and access-token actors', () => {
const gate = requireUserActorGate();
const run = (actor: Partial<Actor>) =>
@@ -2068,6 +2123,59 @@ describe('AuthController.handleGetUserAppToken + handleCheckApp', () => {
expect(decoded.app_uid).toBe(app.uid);
});
// This handler sits in the app-launch critical path. The permission
// grant, the token mint, and the AppData mkdir are mutually independent,
// so they must overlap rather than run as three serial round trips —
// awaiting any one of them before starting the others silently triples
// the latency with every test still passing.
it('runs the permission grant, token mint and AppData mkdir concurrently', async () => {
const order: string[] = [];
const defer = <T,>(label: string, value: T) => {
order.push(`${label}:start`);
return new Promise<T>((resolve) =>
setTimeout(() => {
order.push(`${label}:end`);
resolve(value);
}, 20),
);
};
const permSpy = vi
.spyOn(server.services.permission, 'grantUserAppPermission')
.mockImplementation(() => defer('grant', undefined as never));
const tokenSpy = vi
.spyOn(server.services.auth, 'getUserAppToken')
.mockImplementation(() => defer('token', 'signed.jwt.value'));
const mkdirSpy = vi
.spyOn(server.services.fs, 'mkdir')
.mockImplementation(() => defer('mkdir', undefined as never));
try {
await inCtx(actor, () =>
controller.handleGetUserAppToken(
makeReq({ app_uid: app.uid }, { actor }),
makeRes(),
),
);
} finally {
permSpy.mockRestore();
tokenSpy.mockRestore();
mkdirSpy.mockRestore();
}
// All three must be in flight before any of them settles.
const firstEnd = order.findIndex((e) => e.endsWith(':end'));
const starts = order.slice(0, firstEnd);
expect(starts).toHaveLength(3);
expect(starts).toEqual(
expect.arrayContaining([
'grant:start',
'token:start',
'mkdir:start',
]),
);
});
it('after get-user-app-token, check-app reports authenticated:true and returns a token', async () => {
// Ensure the flag is granted (re-run is idempotent).
await inCtx(actor, () =>
@@ -4077,6 +4185,30 @@ describe('AuthController user-protected mutations (validation paths)', () => {
).rejects.toMatchObject({ statusCode: 400 });
});
it('change-email: accepts an address that resolves back to the caller', async () => {
// `[email protected]` canonicalizes to the caller's own row, so the
// collision check has to exclude them — otherwise Puter reports your
// own address as already in use and there's no way to set it.
const local = `ch_${uniq()}`;
const { user, actor } = await makeUserAndActor();
await server.stores.user.update(user.id, {
email: `${local}@gmail.com`,
clean_email: `${local}@gmail.com`,
email_confirmed: 1,
});
const res = makeRes();
await controller.handleChangeEmail(
makeReq({ new_email: `${local}[email protected]` }, { actor }),
res,
);
expect(res.body).toEqual({});
const after = await server.stores.user.getById(user.id, {
force: true,
});
expect(after!.unconfirmed_change_email).toBe(`${local}[email protected]`);
});
it('change-email: stages the new email + token on success', async () => {
const { user, actor } = await makeUserAndActor();
const newEmail = `ch_${uniq()}@test.local`;
@@ -6328,237 +6460,6 @@ describe('AuthController.handleRevokeSession additional branches', () => {
});
});
// ── handleMigrateToken ──────────────────────────────────────────────
describe('AuthController.handleMigrateToken', () => {
const TEST_ORIGIN = 'https://migrate.test.local';
// PuterServer keeps config in a private field (#config), so we go
// through the controller — IController stores it as `protected
// config` which TS marks but JS doesn't enforce, and the controller
// is the actual consumer of #isMigrateTokenOriginAllowed anyway.
const controllerConfig = () =>
(controller as { config: Record<string, unknown> }).config;
// Mints a v1-shaped JWT signed under the test server's legacy
// secret. The body matches what migrateLegacyToken expects per
// `decoded.type`.
const mintV1Token = (payload: Record<string, unknown>): string => {
const legacy = controllerConfig().jwt_secret as string | undefined;
if (!legacy) throw new Error('test config missing jwt_secret');
return jwt.sign(payload, legacy);
};
beforeAll(() => {
// Make the origin allow-check pass for these tests. We mutate
// the live config because setupTestServer is shared across the
// file; the original value is undefined (default config has no
// `origin`) so we don't need to restore.
controllerConfig().origin = TEST_ORIGIN;
});
it('rejects when the Origin header is missing', async () => {
await expect(
controller.handleMigrateToken(makeReq({}), makeRes()),
).rejects.toMatchObject({ statusCode: 403 });
});
it('allows the exchange from an unlisted origin (apps live on arbitrary domains)', async () => {
// puter.js apps run on any third-party domain; the v1 bearer
// token is the credential, so the exchange itself is not
// origin-gated — only cookie issuance is (next test).
const { user } = await makeUserAndActor();
const v1 = mintV1Token({
type: 'access-token',
token_uid: uuidv4(),
user_uid: user.uuid,
});
const res = makeRes();
await controller.handleMigrateToken(
makeReq(
{},
{
headers: {
origin: 'https://some-app.example',
authorization: `Bearer ${v1}`,
},
},
),
res,
);
expect((res.body as { kind: string }).kind).toBe('access_token');
expect((res.body as { token: string }).token).toBeTruthy();
});
it('does NOT set the puter_token_v2 cookie for an app token from an untrusted origin', async () => {
// Cookie planting on the GUI origin is what the allowlist
// prevents: an attacker page may exchange a token it already
// holds, but must not be able to set a session cookie.
const { user } = await makeUserAndActor();
const appUid = `app-${uuidv4()}`;
const v1 = mintV1Token({
type: 'app-under-user',
user_uid: user.uuid,
app_uid: appUid,
});
const res = makeRes();
await controller.handleMigrateToken(
makeReq(
{},
{
headers: {
origin: 'https://some-app.example',
authorization: `Bearer ${v1}`,
},
},
),
res,
);
expect((res.body as { kind: string }).kind).toBe('app');
expect((res.body as { token: string }).token).toBeTruthy();
expect(res.cookies.puter_token_v2).toBeUndefined();
});
it('normalizes trailing slash on the request Origin (B4)', async () => {
// The Origin header per spec doesn't carry a trailing slash, but
// a misconfigured proxy or a deployment with config.origin
// ending in `/` would otherwise force every call to reject.
const { user } = await makeUserAndActor();
const v1 = mintV1Token({
type: 'access-token',
token_uid: uuidv4(),
user_uid: user.uuid,
});
const res = makeRes();
await controller.handleMigrateToken(
makeReq(
{},
{
headers: {
origin: `${TEST_ORIGIN}/`, // trailing slash
authorization: `Bearer ${v1}`,
},
},
),
res,
);
expect((res.body as { kind: string }).kind).toBe('access_token');
});
it('normalizes case on the request Origin (B4)', async () => {
const { user } = await makeUserAndActor();
const v1 = mintV1Token({
type: 'access-token',
token_uid: uuidv4(),
user_uid: user.uuid,
});
const res = makeRes();
await controller.handleMigrateToken(
makeReq(
{},
{
headers: {
origin: TEST_ORIGIN.toUpperCase(),
authorization: `Bearer ${v1}`,
},
},
),
res,
);
expect((res.body as { kind: string }).kind).toBe('access_token');
});
it('returns 409 reauth_required for v1 web/session tokens', async () => {
// Web tokens never migrate silently — they always go through
// the interactive reauth flow. The body code is what puter.js /
// GUI key on; the 409 status is what tells SDK code "this
// isn't a generic auth failure, route through reauth".
const { user } = await makeUserAndActor();
const v1 = mintV1Token({
type: 'session',
user_uid: user.uuid,
uuid: uuidv4(),
});
await expect(
controller.handleMigrateToken(
makeReq(
{},
{
headers: {
origin: TEST_ORIGIN,
authorization: `Bearer ${v1}`,
},
},
),
makeRes(),
),
).rejects.toMatchObject({
statusCode: 409,
code: 'reauth_required',
});
});
it('does NOT set the puter_token_v2 cookie when migrating an access token (B3)', async () => {
// Access tokens are programmatic — they ride in Authorization
// headers, not browser cookies. Setting a cookie here would
// confuse cookie-only middleware downstream.
const { user } = await makeUserAndActor();
const v1 = mintV1Token({
type: 'access-token',
token_uid: uuidv4(),
user_uid: user.uuid,
});
const res = makeRes();
await controller.handleMigrateToken(
makeReq(
{},
{
headers: {
origin: TEST_ORIGIN,
authorization: `Bearer ${v1}`,
},
},
),
res,
);
expect(res.cookies.puter_token_v2).toBeUndefined();
expect((res.body as { kind: string }).kind).toBe('access_token');
expect((res.body as { token: string }).token).toBeTruthy();
});
it('sets the puter_token_v2 cookie when migrating an app-under-user token (B3)', async () => {
// App tokens DO get a cookie companion — the app runs inside an
// iframe in the GUI, and the GUI's cookie-only middleware
// authenticates subsequent calls from the iframe via the cookie
// rather than the client having to plumb Authorization through
// every request.
const { user } = await makeUserAndActor();
const appUid = `app-${uuidv4()}`;
const v1 = mintV1Token({
type: 'app-under-user',
user_uid: user.uuid,
app_uid: appUid,
});
const res = makeRes();
await controller.handleMigrateToken(
makeReq(
{},
{
headers: {
origin: TEST_ORIGIN,
authorization: `Bearer ${v1}`,
},
},
),
res,
);
expect((res.body as { kind: string }).kind).toBe('app');
const cookie = res.cookies.puter_token_v2;
expect(cookie).toBeDefined();
expect(cookie.value).toBe((res.body as { token: string }).token);
expect(cookie.opts?.httpOnly).toBe(true);
});
});
// -- auth_id preservation on forced re-login --
@@ -6868,3 +6769,412 @@ describe('AuthController auth_id preservation on reauth', () => {
).rejects.toMatchObject({ statusCode: 429 });
});
});
// ── Popup sign-in relay (/login/wait + /login/set) ──────────────────
/**
* The relay stands in for the popup's `postMessage` hand-off on
* cross-origin-isolated openers, where COOP has severed `window.opener`.
* postMessage is audience-bound for free (it posts with `targetOrigin`);
* these tests pin the equivalent binding on the server-side path, since the
* session id is a link-borne value and not a secret.
*/
describe('AuthController.loginWait audience binding', () => {
const OPENER = 'https://opener.test';
/** Mint a real app-under-user token for `origin`, as the popup would. */
const mintAppToken = async (actor: Actor, origin: string) => {
const res = makeRes();
await inCtx(actor, () =>
controller.handleGetUserAppToken(makeReq({ origin }, { actor }), res),
);
return (res.body as { token: string }).token;
};
/**
* Start a wait, then relay `token` into it. The handler resolves the
* origin (async, DB-backed) before subscribing, so the emit is retried
* until the wait settles rather than fired after a fixed sleep.
*/
const waitWithRelay = async (
session: string,
headers: Record<string, unknown>,
token: string | null,
) => {
const res = makeRes();
const waiting = controller.loginWait(makeReq({ session }, { headers }), res);
const settled = waiting.then(
() => 'ok' as const,
(e: unknown) => e,
);
if (token !== null) {
let done = false;
settled.then(() => {
done = true;
});
for (let i = 0; i < 100 && !done; i++) {
await controller.loginSet(
makeReq({ session, auth_token: token }),
makeRes(),
);
await new Promise((r) => setTimeout(r, 10));
}
}
return { res, outcome: await settled };
};
it('returns the token when the caller Origin matches the app it was minted for', async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const { res, outcome } = await waitWithRelay(
uuidv4(),
{ origin: OPENER },
token,
);
expect(outcome).toBe('ok');
expect((res.body as { auth_token: string }).auth_token).toBe(token);
});
it('withholds a token minted for a different app from a mismatched Origin', async () => {
const { actor } = await makeUserAndActor();
// The attack: the popup was talked into minting for OPENER, but the
// party holding the session id is somewhere else entirely.
const token = await mintAppToken(actor, OPENER);
const { res, outcome } = await waitWithRelay(
uuidv4(),
{ origin: 'https://evil.test' },
token,
);
// Same 408 the empty path returns — a mismatched caller must not be
// able to tell "nothing arrived" from "something arrived for someone
// else".
expect(outcome).toMatchObject({ statusCode: 408 });
expect(res.body).toBeUndefined();
});
it('rejects a caller that sends no Origin header', async () => {
// curl and any server-side fetch land here. Without this the session
// id alone — which travels in a link — would be enough to collect.
await expect(
controller.loginWait(makeReq({ session: uuidv4() }, {}), makeRes()),
).rejects.toMatchObject({ statusCode: 403 });
});
it('rejects the opaque "null" origin', async () => {
// Sandboxed iframes and file:// documents both serialise to "null",
// so honouring it would make two unrelated opaque origins equal.
await expect(
controller.loginWait(
makeReq({ session: uuidv4() }, { headers: { origin: 'null' } }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 403 });
});
it('still rejects a malformed session id before looking at Origin', async () => {
await expect(
controller.loginWait(
makeReq(
{ session: 'not-a-uuid' },
{ headers: { origin: OPENER } },
),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('withholds a token that is not an app-under-user token', async () => {
// A session/GUI token relayed through here would sign the opener in
// as the user outright, not as the app.
const username = `relay_${uniq()}`;
const loginRes = makeRes();
await controller.handleSignup(
makeReq({
username,
email: `${username}@test.local`,
password: 'correct-horse-battery',
}),
loginRes,
);
const guiToken = (loginRes.body as { token: string }).token;
const { res, outcome } = await waitWithRelay(
uuidv4(),
{ origin: OPENER },
guiToken,
);
expect(outcome).toMatchObject({ statusCode: 408 });
expect(res.body).toBeUndefined();
});
it('withholds a token with a valid shape but a forged signature', async () => {
const { actor } = await makeUserAndActor();
const real = await mintAppToken(actor, OPENER);
const forged = jwt.sign(
jwt.decode(real) as object,
'not-the-server-secret',
{ keyid: 'v2' },
);
const { res, outcome } = await waitWithRelay(
uuidv4(),
{ origin: OPENER },
forged,
);
expect(outcome).toMatchObject({ statusCode: 408 });
expect(res.body).toBeUndefined();
});
});
// -- Batch grant / revoke + cross-app data grants -----------------------
describe('AuthController — app-data grants', () => {
let issuer: { id: number; username: string };
let issuerActor: Actor;
beforeAll(async () => {
const name = `ad_${Math.random().toString(36).slice(2, 10)}`;
await controller.handleSignup(
makeReq({
username: name,
email: `${name}@test.local`,
password: 'correct-horse-battery',
}),
makeRes(),
);
const row = await server.stores.user.getByUsername(name);
await server.stores.user.update(row!.id, { email_confirmed: 1 });
issuer = { id: row!.id, username: row!.username };
issuerActor = {
user: {
id: row!.id,
uuid: row!.uuid,
username: row!.username,
email: row!.email,
email_confirmed: true,
},
} as Actor;
});
const makeAppRow = async (fields: Record<string, unknown> = {}) =>
(await server.stores.app.create(
{
name: `ad-${uuidv4()}`,
title: 'AppDataTest',
index_url: 'https://example.test/index.html',
...fields,
},
{ ownerUserId: issuer.id },
)) as { id: number; uid: string };
const grantedPermissions = async (appUid: string): Promise<string[]> => {
const rows = (await server.clients.db.read(
'SELECT p.`permission` FROM `user_to_app_permissions` p ' +
'JOIN `apps` a ON a.`id` = p.`app_id` ' +
'WHERE p.`user_id` = ? AND a.`uid` = ?',
[issuer.id, appUid],
)) as Array<{ permission: string }>;
return rows.map((r) => r.permission);
};
const post = (
handler: 'handleGrantUserApp' | 'handleRevokeUserApp',
body: Record<string, unknown>,
) => {
const res = makeRes();
return inCtx(issuerActor, () =>
controller[handler](makeReq(body, { actor: issuerActor }), res),
).then(() => res);
};
it('grants every permission in a list in one request', async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
const permissions = [
`app-data:${target.uid}:kv:read`,
`app-data:${target.uid}:kv:delete`,
];
await post('handleGrantUserApp', {
app_uid: grantee.uid,
permissions,
});
expect(await grantedPermissions(grantee.uid)).toEqual(
expect.arrayContaining(permissions),
);
});
it('revokes every permission in a list in one request', async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
const permissions = [
`app-data:${target.uid}:kv:read`,
`app-data:${target.uid}:fs:read`,
];
await post('handleGrantUserApp', {
app_uid: grantee.uid,
permissions,
});
await post('handleRevokeUserApp', {
app_uid: grantee.uid,
permissions,
});
const remaining = await grantedPermissions(grantee.uid);
for (const permission of permissions) {
expect(remaining).not.toContain(permission);
}
});
it('rejects the scalar and list forms together', async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permission: `app-data:${target.uid}:kv:read`,
permissions: [`app-data:${target.uid}:kv:write`],
}),
).rejects.toMatchObject({ statusCode: 400 });
});
it('rejects an empty, oversized, or `*`-bearing list', async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
for (const permissions of [
[],
Array.from(
{ length: 17 },
(_x, i) => `app-data:${target.uid}:kv:read${i}`,
),
[`app-data:${target.uid}:kv:read`, '*'],
]) {
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permissions,
}),
).rejects.toMatchObject({ statusCode: 400 });
}
});
it('writes nothing when one entry in the list is invalid', async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permissions: [
`app-data:${target.uid}:kv:read`,
`app-data:app-does-not-exist:kv:read`,
],
}),
).rejects.toMatchObject({ statusCode: 404 });
// The valid entry must not have landed: validation runs over the whole
// list before any row is written.
expect(await grantedPermissions(grantee.uid)).not.toContain(
`app-data:${target.uid}:kv:read`,
);
});
it('writes nothing when an entry is too wide for the column it lands in', async () => {
// `#validateAppPermissionParams` allows 4096 chars but the column is 255,
// so this passes shape validation and fails inside the grant. Before the
// pre-flight, the first entry committed and the caller still got a 400 —
// and the dialog reads a 4xx as "nothing was written".
const grantee = await makeAppRow();
const target = await makeAppRow();
const good = `app-data:${target.uid}:kv:read`;
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permissions: [good, 'x'.repeat(300)],
}),
).rejects.toMatchObject({ statusCode: 400 });
expect(await grantedPermissions(grantee.uid)).not.toContain(good);
});
it('404s when the target app does not exist', async () => {
const grantee = await makeAppRow();
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permission: 'app-data:app-nope/:kv:read',
}),
).rejects.toMatchObject({ statusCode: 404 });
});
it('rejects a bare `app-data` with no target app', async () => {
const grantee = await makeAppRow();
for (const permission of ['app-data', 'app-data:', 'app-data::kv']) {
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permission,
}),
).rejects.toMatchObject({ statusCode: 400 });
}
});
it('refuses a grant when the target opted out of sharing', async () => {
const grantee = await makeAppRow();
const closed = await makeAppRow({
metadata: JSON.stringify({ share_app_data: false }),
});
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permission: `app-data:${closed.uid}:kv:read`,
}),
).rejects.toMatchObject({ statusCode: 403 });
});
it('still allows revoking a grant after the target opts out', async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
const permission = `app-data:${target.uid}:kv:read`;
await post('handleGrantUserApp', {
app_uid: grantee.uid,
permission,
});
await server.stores.app.update(target.id, {
metadata: JSON.stringify({ share_app_data: false }),
});
// A user must always be able to withdraw consent, whatever the target
// now says about sharing.
await post('handleRevokeUserApp', { app_uid: grantee.uid, permission });
expect(await grantedPermissions(grantee.uid)).not.toContain(permission);
});
it("creates the target's AppData directory for an fs grant", async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
const path = `/${issuer.username}/AppData/${target.uid}`;
expect(await server.stores.fsEntry.getEntryByPath(path)).toBeFalsy();
await post('handleGrantUserApp', {
app_uid: grantee.uid,
permission: `app-data:${target.uid}:fs:read`,
});
// Without this the grant is valid but every read 404s until the target
// app happens to run for the first time.
expect(await server.stores.fsEntry.getEntryByPath(path)).toBeTruthy();
});
it('leaves unrelated permissions untouched by the new validation', async () => {
const grantee = await makeAppRow();
const permission = 'service:unrelated:ii:read';
await post('handleGrantUserApp', {
app_uid: grantee.uid,
permission,
});
expect(await grantedPermissions(grantee.uid)).toContain(permission);
});
});
+467 -141
View File
@@ -1,21 +1,20 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify it under the
* terms of the GNU Affero General Public License as published by the Free
* Software Foundation, either version 3 of the License, or (at your option) any
* later version.
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
* details.
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import bcrypt from 'bcrypt';
@@ -28,6 +27,7 @@ import type { HttpErrorOptions } from '../../core/http/HttpError.js';
import { HttpError } from '../../core/http/HttpError.js';
import { antiCsrf } from '../../core/http/middleware/antiCsrf.js';
import { generateCaptcha } from '../../core/http/middleware/captcha.js';
import type { Actor } from '../../core/actor.js';
import { checkRateLimit } from '../../core/http/middleware/rateLimit.js';
import {
signStepUpToken,
@@ -61,11 +61,19 @@ import {
generateDefaultFsentries,
promoteToVerifiedGroup,
} from '../../util/userProvisioning.js';
import {
APP_DATA_PERMISSION_PREFIX,
appDataSharingAllowed,
parseAppDataPermission,
} from '../../services/permission/appDataScopes.js';
import { PuterController } from '../types.js';
const USERNAME_REGEX = /^\w{1,}$/;
const USERNAME_MAX_LENGTH = 45;
const FINGERPRINT_MAX_LENGTH = 128;
// One consent prompt covers a handful of scopes at most. The cap keeps a
// crafted request from turning a single grant call into a bulk write.
const MAX_PERMISSIONS_PER_REQUEST = 16;
const DISPATCH_ID_MAX_LENGTH = 128;
// Default SMS send attempts before the card fallback opens.
const DEFAULT_CARD_FALLBACK_ATTEMPTS = 2;
@@ -75,6 +83,94 @@ const DEFAULT_CARD_FALLBACK_ATTEMPTS = 2;
// crossed.
const SEND_PHONE_RATE_LIMIT = 10;
const SEND_PHONE_RATE_WINDOW_MS = 60 * 60_000;
// -- Post-login route limits -----------------------------------------
//
// The credential legs above (login, signup, recovery, confirmation) each
// carry their own limit. Everything a session can reach *after* signing
// in shares the four shapes below, keyed on the actor rather than the
// network — a per-account ceiling is the meaningful one once we know who
// is calling.
/**
* Mints or reconfigures a credential. Deliberately an hour-scale window: these
* are human actions taken a handful of times, and an unbounded rate turns one
* compromised session into a durable foothold.
*/
const CREDENTIAL_MINT_LIMIT = {
scope: 'auth-credential-mint',
limit: 20,
window: 60 * 60_000,
key: 'user',
} as const;
/**
* Second-factor configuration, including the verify leg. Shorter window than
* the mint limit because enabling 2FA legitimately involves a few attempts in a
* row, but unbounded verification is a TOTP brute force.
*/
const TWO_FACTOR_LIMIT = {
scope: 'auth-2fa-configure',
limit: 30,
window: 15 * 60_000,
key: 'user',
} as const;
/** Permission and membership writes. Never called in a loop by a client. */
const GRANT_LIMIT = {
scope: 'auth-grant',
limit: 60,
window: 60_000,
key: 'user',
} as const;
/**
* Read-only checks the GUI makes on nearly every interaction. The ceiling is
* high enough that only a runaway loop reaches it.
*/
const AUTH_CHECK_LIMIT = {
scope: 'auth-check',
limit: 300,
window: 60_000,
key: 'user',
} as const;
/**
* Anti-CSRF token issuance. Clients mint a fresh token per protected mutation
* and cache nothing, so this ceiling has to clear the SUM of the budgets that
* spend tokens — matching any single one of them guarantees the gate fires
* before the mutation it guards does.
*
* What spends them: the session-authenticated download path, one token per
* file, at the read budget of 600/min — a multi-selection download burns tokens
* exactly the way a bulk delete burns its own budget, so this tracks the bulk
* figure used for filesystem mutations; logout at 60/min; and the
* session-management writes (revoke, rename), which a person triggers a handful
* of times. Call it ~700/min of real demand, and leave enough on top that a
* bulk operation runs out of files before it runs out of tokens.
*/
const ANTI_CSRF_MINT_LIMIT = {
scope: 'anticsrf',
limit: 1200,
window: 60_000,
key: 'user',
} as const;
/** Settings-page reads — enumerating sessions, permissions, groups. */
const AUTH_LIST_LIMIT = {
scope: 'auth-list',
limit: 120,
window: 60_000,
key: 'user',
} as const;
/** Session plumbing: logout, GUI token, cookie sync. */
const SESSION_LIMIT = {
scope: 'auth-session',
limit: 60,
window: 60_000,
key: 'user',
} as const;
// Once the threshold is crossed the fallback stays open this long, so the
// user can finish the card flow without racing the attempt counter's expiry.
const CARD_FALLBACK_OPEN_TTL_SECONDS = 24 * 60 * 60;
@@ -133,6 +229,31 @@ export class AuthController extends PuterController {
legacyCode: 'bad_request',
});
}
// Browser-only gate. The session id is client-chosen and travels in a
// link, so it is not a secret — the `Origin` header is what actually
// says who is asking, and only a browser is prevented from lying about
// it. A caller with no `Origin` (curl, a server-side fetch) could
// otherwise collect a token minted for someone else's app just by
// knowing the id.
//
// `"null"` is rejected too: sandboxed iframes and `file://` documents
// serialise their opaque origin that way, and two *unrelated* opaque
// origins would compare equal to each other.
const reqOrigin = req.headers.origin;
if (!reqOrigin || reqOrigin === 'null') {
throw new HttpError(403, 'Origin not allowed', {
legacyCode: 'forbidden',
});
}
// The app identity this caller is allowed to collect a token for,
// derived from the browser-attested header rather than anything in
// the request body — so no client, honest or not, can influence the
// comparison made after the token arrives.
const expectedAppUid =
await this.services.auth.appUidFromOrigin(reqOrigin);
const { resolve, promise } = Promise.withResolvers<void>();
let token: string | null = null;
@@ -153,12 +274,56 @@ export class AuthController extends PuterController {
});
}
// Audience check. The postMessage hand-off this relay stands in for
// is origin-bound for free — it posts with `targetOrigin`, so a page
// can only ever receive a token minted for *itself*. Delivering
// server-side dropped that binding; this restores it. Without it a
// popup talked into minting for app X (see `trustsOpenerOriginParam`
// in the GUI) hands X's token to whoever holds the session id.
if (!this.#tokenIsForApp(token, expectedAppUid)) {
// Deliberately the same 408 the no-token path returns: a caller
// learns only that nothing arrived for them, not that a token
// for a different app went past.
throw new HttpError(408, 'Request timeout.', {
legacyCode: 'request_timeout',
});
}
res.json({
auth_token: token,
});
}
/**
* Whether a relayed token is an app-under-user token minted for
* `expectedAppUid`. Verifies the signature — an unverified decode would let
* a caller relay a token whose claims it wrote itself.
*/
#tokenIsForApp(token: string, expectedAppUid: string): boolean {
try {
const payload = this.services.token.verify<{
type?: string;
app_uid?: string;
}>('auth', token);
return (
payload.type === 'app-under-user' &&
!!payload.app_uid &&
payload.app_uid === expectedAppUid
);
} catch {
// Malformed, expired, or signed with a key we don't hold.
return false;
}
}
@Post('/login/set', {
subdomain: ['api'],
// Unauthenticated fan-out to every `/login/wait` listener on the
// session id. A legitimate popup posts here exactly once per sign-in,
// so a generous per-IP cap costs honest traffic nothing while denying
// an attacker unbounded attempts to land a token on a guessed id.
rateLimit: [
{ scope: 'login-set', limit: 60, window: 15 * 60_000, key: 'ip' },
],
})
async loginSet(req: Request, res: Response) {
const { session, auth_token } = req.body;
@@ -182,6 +347,11 @@ export class AuthController extends PuterController {
// -- Login -------------------------------------------------------
@Post('/login', {
// Returns a full session token in the response body. Reflected CORS
// would otherwise let any page trade a password for that token and
// read it — third-party sign-in goes through `puter.auth.signIn()`,
// whose popup runs on this origin and yields an app-scoped token.
guiOriginOnly: true,
captcha: true,
// Two limits: per-fingerprint keeps users behind a shared IP
// (offices, campuses) from throttling each other, while the
@@ -298,6 +468,8 @@ export class AuthController extends PuterController {
// -- Login: OTP verification -------------------------------------
@Post('/login/otp', {
// Second leg of `/login` — also completes into `#completeLogin`.
guiOriginOnly: true,
captcha: true,
rateLimit: [
{ scope: 'login-otp', limit: 15, window: 30 * 60_000 },
@@ -362,6 +534,8 @@ export class AuthController extends PuterController {
// -- Login: recovery code ----------------------------------------
@Post('/login/recovery-code', {
// Second leg of `/login` — also completes into `#completeLogin`.
guiOriginOnly: true,
captcha: true,
rateLimit: [
{ scope: 'login-recovery', limit: 10, window: 60 * 60_000 },
@@ -439,6 +613,9 @@ export class AuthController extends PuterController {
// -- Signup ------------------------------------------------------
@Post('/signup', {
// Completes into `#completeLogin`, so it hands back a session token
// exactly like `/login`. Same reasoning.
guiOriginOnly: true,
captcha: true,
rateLimit: [
{ scope: 'signup', limit: 10, window: 15 * 60_000 },
@@ -961,12 +1138,13 @@ export class AuthController extends PuterController {
requireUserActor: true,
allowUnconfirmed: true,
antiCsrf: true,
rateLimit: SESSION_LIMIT,
})
async handleLogout(req: Request, res: Response): Promise<void> {
// Clear the session cookie + the v2 migrate-token companion
// cookie (set after `/auth/migrate-token` for app-under-user
// iframes). authProbe reads `puter_token_v2` as a fallback, so
// a stale value would re-authenticate the next request.
// Clear the session cookie + `puter_token_v2`. Nothing issues the
// latter any more (it came from the retired token migration), but
// authProbe still reads it as a fallback, so a value left in a
// browser would re-authenticate the next request.
res.clearCookie(this.config.cookie_name ?? 'puter_token');
res.clearCookie('puter_token_v2');
// Drop any step-up elevation too, so it can't reactivate on a shared
@@ -2259,15 +2437,18 @@ export class AuthController extends PuterController {
}
await this.#validateEmail(new_email);
// Block if any confirmed account (password or OIDC) already
// Block if any OTHER confirmed account (password or OIDC) already
// owns that email. Match raw + canonical to collapse gmail
// aliases.
// aliases — which is also why the caller has to be excluded: an
// alias of your own current address resolves back to you, and
// "already in use" about yourself is nonsense.
const canonical = cleanEmail(new_email);
const existing =
(await this.stores.user.getByEmail(new_email)) ??
(await this.stores.user.getByCleanEmail(canonical));
if (
existing &&
existing.id !== req.actor!.user.id &&
(existing.email_confirmed || existing.password !== null)
) {
throw new HttpError(400, 'This email is already in use.', {
@@ -2621,7 +2802,25 @@ export class AuthController extends PuterController {
// -- Captcha generation -------------------------------------------
@Get('/api/captcha/generate', { subdomain: '*' })
@Get('/api/captcha/generate', {
subdomain: '*',
// Unauthenticated, renders an image per call, and is the gate
// protecting /login and /signup — so bulk pre-generation is
// directly useful to an attacker. Per-fingerprint for fairness on
// shared IPs, plus a per-IP backstop against header rotation.
//
// The fingerprint bucket is the one sized for a person: a handful of
// refreshes while getting a captcha right. The IP bucket is not — one
// address is a whole office, campus or carrier gateway, and everyone
// behind it is signing in through the same counter, so sizing it for
// a browser would deny the captcha to a network rather than to an
// attacker. It stays wide enough for that population and narrow
// enough that header rotation still runs out.
rateLimit: [
{ scope: 'captcha', limit: 30, window: 60_000 },
{ scope: 'captcha-ip', limit: 3_000, window: 60_000, key: 'ip' },
],
})
async handleCaptchaGenerate(_req: Request, res: Response): Promise<void> {
const difficulty =
(this.config as { captcha?: { difficulty?: string } }).captcha
@@ -2633,6 +2832,7 @@ export class AuthController extends PuterController {
// -- Anti-CSRF token generation ----------------------------------
@Get('/get-anticsrf-token', {
rateLimit: ANTI_CSRF_MINT_LIMIT,
// Anti-CSRF tokens are only consumed by `requireUserActor` routes,
// so issuance is scoped to the same actor kind for consistency.
requireUserActor: true,
@@ -2653,6 +2853,7 @@ export class AuthController extends PuterController {
@Post('/auth/grant-user-user', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGrantUserUser(req: Request, res: Response): Promise<void> {
const { target_username, permission, extra, meta } = req.body;
@@ -2745,13 +2946,114 @@ export class AuthController extends PuterController {
return app.uid;
}
/**
* Resolve the `permission` / `permissions` pair into the list to act on.
*
* One consent prompt can cover several scopes (read a store, write
* another), and a client looping the single form would have to invent its
* own partial-failure and rollback handling. Accepting the array keeps that
* in one request.
*/
#appPermissionList(body: {
permission?: unknown;
permissions?: unknown;
}): string[] {
const { permission, permissions } = body;
if (permissions !== undefined && permissions !== null) {
if (permission !== undefined && permission !== null) {
throw new HttpError(
400,
'Pass `permission` or `permissions`, not both',
{ legacyCode: 'bad_request' },
);
}
if (!Array.isArray(permissions) || permissions.length === 0) {
throw new HttpError(400, 'Invalid `permissions`', {
legacyCode: 'bad_request',
});
}
if (permissions.length > MAX_PERMISSIONS_PER_REQUEST) {
throw new HttpError(400, 'Too many `permissions`', {
legacyCode: 'bad_request',
});
}
for (const entry of permissions) {
this.#validateAppPermissionParams({ permission: entry });
// `*` means "revoke everything" in the scalar form only —
// inside a list it would silently widen a targeted request.
if (!entry || entry === '*') {
throw new HttpError(400, 'Invalid `permissions`', {
legacyCode: 'bad_request',
});
}
}
return [...new Set(permissions as string[])];
}
return typeof permission === 'string' && permission ? [permission] : [];
}
/**
* Gate a cross-app data grant: the target must exist, must not have opted
* out of sharing, and must be named. Also creates the target's AppData
* directory for an `fs` scope, since it is only created lazily when the app
* first runs — without this a valid grant would 404 until then.
*/
async #prepareAppDataGrant(
actor: Actor,
permission: string,
): Promise<void> {
const parsed = parseAppDataPermission(permission);
if (!parsed) {
// A bare `app-data` (or one with an empty target) would cover every
// app the user has by prefix implication, which no prompt can
// describe. Reject rather than treat it as an unrelated permission.
if (
permission === APP_DATA_PERMISSION_PREFIX ||
permission.startsWith(`${APP_DATA_PERMISSION_PREFIX}:`)
) {
throw new HttpError(
400,
'Invalid `app-data` permission: missing target app',
{ legacyCode: 'bad_request' },
);
}
return;
}
const target = await this.stores.app.getByUid(parsed.targetAppUid);
if (!target) {
throw new HttpError(
404,
`entity_not_found: app:${parsed.targetAppUid}`,
{ legacyCode: 'subject_does_not_exist' },
);
}
if (!appDataSharingAllowed(target)) {
throw new HttpError(
403,
'This app does not share its data with other apps',
{ legacyCode: 'forbidden' },
);
}
const username = actor.user?.username;
const userId = actor.user?.id;
if ((parsed.store === 'fs' || !parsed.store) && username && userId) {
await this.services.fs.mkdir(userId, {
path: `/${username}/AppData/${parsed.targetAppUid}`,
createMissingParents: true,
} as never);
}
}
@Post('/auth/grant-user-app', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGrantUserApp(req: Request, res: Response): Promise<void> {
let { app_uid } = req.body;
const { origin, permission, extra, meta } = req.body;
const { origin, permission, permissions, extra, meta } = req.body;
this.#validateAppPermissionParams({
app_uid,
origin,
@@ -2759,27 +3061,43 @@ export class AuthController extends PuterController {
extra,
meta,
});
const list = this.#appPermissionList({ permission, permissions });
if (origin) {
app_uid = await this.#registeredAppUidFromOrigin(origin);
}
if (!app_uid || !permission) {
if (!app_uid || list.length === 0) {
throw new HttpError(400, 'Missing `app_uid` or `permission`', {
legacyCode: 'bad_request',
});
}
await this.services.permission.grantUserAppPermission(
req.actor!,
app_uid,
permission,
extra ?? undefined,
meta ?? undefined,
);
// Validate every entry before writing any, so a bad one in the list
// cannot leave a partially-granted set behind: the dialog reads a 4xx as
// "nothing was written" and skips its withdrawal, so a partial commit
// leaves live access the user was told they refused. The rewrite running
// twice is cheaper than splitting the grant into prepare/commit.
for (const entry of list) {
await this.services.permission.assertUserAppPermissionWritable(
entry,
);
await this.#prepareAppDataGrant(req.actor!, entry);
}
for (const entry of list) {
await this.services.permission.grantUserAppPermission(
req.actor!,
app_uid,
entry,
extra ?? undefined,
meta ?? undefined,
);
}
res.json({});
}
@Post('/auth/grant-user-group', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGrantUserGroup(req: Request, res: Response): Promise<void> {
const { group_uid, permission, extra, meta } = req.body;
@@ -2808,6 +3126,7 @@ export class AuthController extends PuterController {
@Post('/auth/revoke-user-user', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleRevokeUserUser(req: Request, res: Response): Promise<void> {
const { target_username, permission, meta } = req.body;
@@ -2830,24 +3149,28 @@ export class AuthController extends PuterController {
@Post('/auth/revoke-user-app', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleRevokeUserApp(req: Request, res: Response): Promise<void> {
let { app_uid } = req.body;
const { origin, permission, meta } = req.body;
const { origin, permission, permissions, meta } = req.body;
this.#validateAppPermissionParams({
app_uid,
origin,
permission,
meta,
});
const list = this.#appPermissionList({ permission, permissions });
if (origin) {
app_uid = await this.#registeredAppUidFromOrigin(origin);
}
if (!app_uid || !permission) {
if (!app_uid || list.length === 0) {
throw new HttpError(400, 'Missing `app_uid` or `permission`', {
legacyCode: 'bad_request',
});
}
// Deliberately not gated by the target's sharing flag: a user must
// always be able to withdraw a grant, whatever the target now says.
if (permission === '*') {
await this.services.permission.revokeUserAppAll(
req.actor!,
@@ -2855,12 +3178,14 @@ export class AuthController extends PuterController {
meta ?? undefined,
);
} else {
await this.services.permission.revokeUserAppPermission(
req.actor!,
app_uid,
permission,
meta ?? undefined,
);
for (const entry of list) {
await this.services.permission.revokeUserAppPermission(
req.actor!,
app_uid,
entry,
meta ?? undefined,
);
}
}
res.json({});
}
@@ -2868,6 +3193,7 @@ export class AuthController extends PuterController {
@Post('/auth/revoke-user-group', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleRevokeUserGroup(req: Request, res: Response): Promise<void> {
const { group_uid, permission, meta } = req.body;
@@ -2887,7 +3213,11 @@ export class AuthController extends PuterController {
// -- Permission checks -------------------------------------------
@Post('/auth/check-permissions', { subdomain: 'api', requireAuth: true })
@Post('/auth/check-permissions', {
subdomain: 'api',
requireAuth: true,
rateLimit: AUTH_CHECK_LIMIT,
})
async handleCheckPermissions(req: Request, res: Response): Promise<void> {
const { permissions } = req.body;
if (!Array.isArray(permissions)) {
@@ -2915,7 +3245,11 @@ export class AuthController extends PuterController {
// -- Session management ------------------------------------------
@Get('/auth/list-sessions', { subdomain: 'api', requireUserActor: true })
@Get('/auth/list-sessions', {
subdomain: 'api',
requireUserActor: true,
rateLimit: AUTH_LIST_LIMIT,
})
async handleListSessions(req: Request, res: Response): Promise<void> {
const sessions = await this.services.auth.listSessions(req.actor!);
res.json(sessions);
@@ -2997,7 +3331,11 @@ export class AuthController extends PuterController {
// -- Dev app permissions -----------------------------------------
@Post('/auth/grant-dev-app', { subdomain: 'api', requireUserActor: true })
@Post('/auth/grant-dev-app', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGrantDevApp(req: Request, res: Response): Promise<void> {
let { app_uid } = req.body;
const { origin, permission, extra, meta } = req.body;
@@ -3030,6 +3368,7 @@ export class AuthController extends PuterController {
@Post('/auth/revoke-dev-app', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleRevokeDevApp(req: Request, res: Response): Promise<void> {
let { app_uid } = req.body;
@@ -3065,6 +3404,7 @@ export class AuthController extends PuterController {
@Get('/auth/list-permissions', {
subdomain: 'api',
requireUserActor: true,
rateLimit: AUTH_LIST_LIMIT,
})
async handleListPermissions(req: Request, res: Response): Promise<void> {
const userId = req.actor!.user.id;
@@ -3129,7 +3469,11 @@ export class AuthController extends PuterController {
// -- App origin resolution ---------------------------------------
@Post('/auth/app-uid-from-origin', { subdomain: 'api', requireAuth: true })
@Post('/auth/app-uid-from-origin', {
subdomain: 'api',
requireAuth: true,
rateLimit: AUTH_CHECK_LIMIT,
})
async handleAppUidFromOrigin(req: Request, res: Response): Promise<void> {
const origin = req.body?.origin || req.query?.origin;
if (!origin)
@@ -3145,6 +3489,9 @@ export class AuthController extends PuterController {
@Post('/auth/get-user-app-token', {
subdomain: 'api',
requireUserActor: true,
// Called once per app launch, and the GUI can legitimately launch
// several in quick succession.
rateLimit: { ...AUTH_CHECK_LIMIT, scope: 'app-token', limit: 120 },
})
async handleGetUserAppToken(req: Request, res: Response): Promise<void> {
let { app_uid } = req.body;
@@ -3168,15 +3515,31 @@ export class AuthController extends PuterController {
app = await this.stores.app.createFromOrigin(app_uid, origin, {
ownerUserId,
});
// An origin's uid is a deterministic uuidv5, so a deleted app
// reappears here under the identical uid. Withdraw any cross-app
// data grants left pointing at it before this new row can inherit
// consent the user gave its predecessor. Only *this* path can reuse
// a uid: `AppStore.create` mints a random uuid4, which no deleted
// app can ever hold again.
//
// Called directly rather than through `app.changed`: the token is
// issued below, so this has to be able to stop that, and
// `emitAndWait` swallows listener errors. Letting it throw is the
// point — a sweep that failed leaves the old grants live against an
// app whoever controls the origin now has just claimed.
await this.services.appPermission.withdrawAppDataGrants(
app_uid,
'uid reused by a new app',
);
}
if (!app) {
throw new HttpError(404, `App ${app_uid} does not exist`, {
legacyCode: 'not_found',
});
}
// Grant the app-is-authenticated flag
const userPermGrantPromise =
await this.services.permission.grantUserAppPermission(
this.services.permission.grantUserAppPermission(
req.actor!,
app_uid,
'flag:app-is-authenticated',
@@ -3184,7 +3547,7 @@ export class AuthController extends PuterController {
{},
);
const token = await this.services.auth.getUserAppToken(
const tokenPromise = this.services.auth.getUserAppToken(
req.actor!,
app_uid,
);
@@ -3211,7 +3574,11 @@ export class AuthController extends PuterController {
}
})();
await Promise.all([userPermGrantPromise, missingFSPathPromise]);
const [, token] = await Promise.all([
userPermGrantPromise,
tokenPromise,
missingFSPathPromise,
]);
try {
const a = app as {
@@ -3243,7 +3610,11 @@ export class AuthController extends PuterController {
res.json({ token, app_uid });
}
@Post('/auth/check-app', { subdomain: 'api', requireUserActor: true })
@Post('/auth/check-app', {
subdomain: 'api',
requireUserActor: true,
rateLimit: AUTH_CHECK_LIMIT,
})
async handleCheckApp(req: Request, res: Response): Promise<void> {
let { app_uid } = req.body;
const { origin } = req.body;
@@ -3279,87 +3650,10 @@ export class AuthController extends PuterController {
// -- Access tokens -----------------------------------------------
async handleMigrateToken(req: Request, res: Response): Promise<void> {
// 1. Browser-only gate. No Origin header → reject (server-side
// callers should re-auth properly). The exchange itself is open
// to any browser origin: puter.js apps live on arbitrary
// third-party domains, the v1 bearer token is the credential,
// and cookies are never read here — so a hostile page learns
// nothing it doesn't already hold. Origin *trust* only decides
// whether the `puter_token_v2` companion cookie is set below.
const reqOrigin = req.headers.origin;
if (!reqOrigin) {
throw new HttpError(403, 'Origin not allowed', {
legacyCode: 'forbidden',
});
}
// 2. Extract token. Header takes precedence so body-capture logs
// (if any) never see the credential.
const authHeader = req.headers.authorization;
const headerToken =
typeof authHeader === 'string' && authHeader.startsWith('Bearer ')
? authHeader.slice('Bearer '.length).trim()
: null;
const bodyToken =
typeof req.body?.token === 'string' ? req.body.token.trim() : null;
const v1Token = headerToken || bodyToken;
if (!v1Token) {
throw new HttpError(400, 'Missing token', {
legacyCode: 'bad_request',
});
}
const result = await this.services.auth.migrateLegacyToken(v1Token, {
ip: req.ip,
userAgent:
typeof req.headers['user-agent'] === 'string'
? req.headers['user-agent']
: undefined,
});
// `puter_token_v2` is the cookie companion to v2 app tokens —
// it lets the GUI's cookie-only middleware authenticate
// subsequent same-origin calls without the client forwarding
// Authorization headers. Issuing it is gated on the trusted
// origin allowlist: an arbitrary attacker page must not be able
// to plant a session cookie on the GUI origin (login CSRF).
// Untrusted origins still get the token in the JSON body — the
// cookie is only consumed same-origin (see authProbe) so they
// lose nothing. Access tokens are programmatic (no browser
// cookie surface) so we deliberately skip them here.
if (
result.kind === 'app' &&
this.#isMigrateTokenOriginAllowed(reqOrigin)
) {
res.cookie('puter_token_v2', result.token, {
...sessionCookieFlags(this.config),
httpOnly: true,
});
}
res.json(result);
}
#isMigrateTokenOriginAllowed(origin: string): boolean {
// Origin headers and config values both reach us in inconsistent
// shapes (trailing slash, mixed case from misconfigured deploys,
// stray whitespace from JSON config edits). Normalize both sides
// before equality so a `config.origin` with a trailing slash
// doesn't reject every same-origin browser call.
const normalize = (raw: string | undefined): string =>
(raw ?? '').trim().replace(/\/+$/, '').toLowerCase();
const incoming = normalize(origin);
if (!incoming) return false;
if (incoming === normalize(this.config.origin)) return true;
const allowlist = (
this.config as { allow_migrate_token_origins?: string[] }
).allow_migrate_token_origins;
if (!Array.isArray(allowlist)) return false;
return allowlist.some((entry) => normalize(entry) === incoming);
}
@Post('/auth/create-access-token', {
subdomain: 'api',
requireAuth: true,
rateLimit: CREDENTIAL_MINT_LIMIT,
})
async handleCreateAccessToken(req: Request, res: Response): Promise<void> {
const { permissions, expiresIn, label } = req.body;
@@ -3428,6 +3722,7 @@ export class AuthController extends PuterController {
@Post('/auth/configure-2fa/:action', {
subdomain: 'api',
requireUserActor: true,
rateLimit: TWO_FACTOR_LIMIT,
})
async handleConfigure2fa(req: Request, res: Response): Promise<void> {
const action = req.params.action;
@@ -3558,7 +3853,11 @@ export class AuthController extends PuterController {
// -- Developer profile -------------------------------------------
@Get('/get-dev-profile', { subdomain: 'api', requireUserActor: true })
@Get('/get-dev-profile', {
subdomain: 'api',
requireUserActor: true,
rateLimit: AUTH_LIST_LIMIT,
})
async handleGetDevProfile(req: Request, res: Response): Promise<void> {
const user = await this.stores.user.getById(req.actor!.user.id!, {
force: true,
@@ -3588,7 +3887,13 @@ export class AuthController extends PuterController {
// -- Group management --------------------------------------------
@Post('/group/create', { subdomain: 'api', requireUserActor: true })
@Post('/group/create', {
subdomain: 'api',
requireUserActor: true,
// Creates a persistent row per call with no quota behind it, so it
// sits on the hour-scale budget rather than the grant one.
rateLimit: { ...CREDENTIAL_MINT_LIMIT, scope: 'group-create' },
})
async handleGroupCreate(req: Request, res: Response): Promise<void> {
const extra = req.body.extra ?? {};
const metadata = req.body.metadata ?? {};
@@ -3609,7 +3914,11 @@ export class AuthController extends PuterController {
res.json({ uid });
}
@Post('/group/add-users', { subdomain: 'api', requireUserActor: true })
@Post('/group/add-users', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGroupAddUsers(req: Request, res: Response): Promise<void> {
const { uid, users } = req.body ?? {};
if (!uid)
@@ -3641,7 +3950,11 @@ export class AuthController extends PuterController {
res.json({});
}
@Post('/group/remove-users', { subdomain: 'api', requireUserActor: true })
@Post('/group/remove-users', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGroupRemoveUsers(req: Request, res: Response): Promise<void> {
const { uid, users } = req.body ?? {};
if (!uid)
@@ -3673,7 +3986,11 @@ export class AuthController extends PuterController {
res.json({});
}
@Get('/group/list', { subdomain: 'api', requireUserActor: true })
@Get('/group/list', {
subdomain: 'api',
requireUserActor: true,
rateLimit: AUTH_LIST_LIMIT,
})
async handleGroupList(req: Request, res: Response): Promise<void> {
const userId = req.actor!.user.id!;
const [owned, member] = await Promise.all([
@@ -3686,7 +4003,22 @@ export class AuthController extends PuterController {
});
}
@Get('/group/public-groups', { subdomain: 'api' })
@Get('/group/public-groups', {
subdomain: 'api',
// The only unauthenticated route in the group set, so IP is the
// only key available — and that makes the bucket an aggregate:
// one office, campus or carrier gateway is a single key for
// everybody behind it, and each of them reads this once while
// bootstrapping. Sized for that population of real people rather
// than one browser, and no wider: this sits next to the sign-in
// surface, so it stays a real bound on enumeration.
rateLimit: {
scope: 'public-groups',
limit: 1_200,
window: 60_000,
key: 'ip',
},
})
async handleGroupPublicGroups(_req: Request, res: Response): Promise<void> {
res.json({
user: this.config.default_user_group ?? null,
@@ -3699,6 +4031,7 @@ export class AuthController extends PuterController {
@Get('/get-gui-token', {
requireUserActor: true,
allowUnconfirmed: true,
rateLimit: SESSION_LIMIT,
})
async handleGetGuiToken(req: Request, res: Response): Promise<void> {
if (!req.actor?.session?.uid)
@@ -3718,6 +4051,12 @@ export class AuthController extends PuterController {
}
@Get('/session/sync-cookie', {
rateLimit: SESSION_LIMIT,
// Installs the session cookie. Only page script on our own origin
// should be able to ask for that (the `tokenSource` check below is the
// companion rule: the token has to come from an Authorization header,
// not a URL).
guiOriginOnly: true,
requireUserActor: true,
allowUnconfirmed: true,
})
@@ -4082,19 +4421,6 @@ export class AuthController extends PuterController {
},
(req, res) => this.handleRenameSession(req, res),
);
router.post(
'/auth/migrate-token',
{
rateLimit: {
scope: 'migrate-token',
limit: 20,
window: 15 * 60_000,
key: 'ip',
},
},
(req, res) => this.handleMigrateToken(req, res),
);
}
// -- Private helpers ----------------------------------------------
@@ -0,0 +1,103 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Wiring test for `guiOriginOnly`.
*
* `originGate.test.ts` covers what the gate decides. This covers that the
* routes which hand a session credential back to the caller actually opt into
* it — a correct gate wired to nothing protects nothing, and the failure is
* invisible (the route keeps working, just for everybody).
*/
import { describe, expect, it } from 'vitest';
import { AuthController } from './AuthController.js';
import {
ROUTES_METADATA_KEY,
type CollectedRoute,
} from '../../core/http/types.js';
// The decorators register on the prototype via `addInitializer`, which only
// runs once an instance exists. We never start this one — constructing it is
// enough to populate the route metadata.
const collectRoutes = (): CollectedRoute[] => {
const proto = AuthController.prototype as unknown as Record<
string,
CollectedRoute[] | undefined
>;
if (!proto[ROUTES_METADATA_KEY]) {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
new (AuthController as any)({}, {}, {}, {}, {});
}
return proto[ROUTES_METADATA_KEY] ?? [];
};
/**
* Every route that answers with a usable session credential. Adding one here
* without `guiOriginOnly` is the mistake this test exists to catch: reflected
* CORS would let any page read the credential out of the response.
*/
const CREDENTIAL_ROUTES: Array<[method: string, path: string]> = [
['post', '/login'],
['post', '/login/otp'],
['post', '/login/recovery-code'],
['post', '/signup'],
['get', '/session/sync-cookie'],
];
describe('guiOriginOnly wiring on AuthController', () => {
const routes = collectRoutes();
it('registers the routes under test at all (guards against a rename)', () => {
for (const [method, path] of CREDENTIAL_ROUTES) {
const found = routes.find(
(r) => r.method === method && r.path === path,
);
expect(found, `${method.toUpperCase()} ${path} not registered`)
.toBeDefined();
}
});
it.each(CREDENTIAL_ROUTES)(
'gates %s %s to the GUI origin',
(method, path) => {
const route = routes.find(
(r) => r.method === method && r.path === path,
);
expect(route?.options.guiOriginOnly).toBe(true);
},
);
// The popup relay is the third-party sign-in path: `puter.auth.signIn()`
// polls `/login/wait` from whatever origin the app is served from, and
// `/login/set` is posted by the popup. Gating either to our own origin
// would break every third-party app, and neither needs it — the token
// they move is app-scoped, not a session.
it.each([
['post', '/login/wait'],
['post', '/login/set'],
])('leaves %s %s open cross-origin', (method, path) => {
const route = routes.find(
(r) => r.method === method && r.path === path,
);
expect(route, `${method.toUpperCase()} ${path} not registered`)
.toBeDefined();
expect(route?.options.guiOriginOnly).toBeUndefined();
});
});
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -27,20 +27,19 @@ import { PuterController } from '../types.js';
*
* The route is intentionally a thin shell: parse the four custom
* `X-Broadcast-*` headers, hand the raw body + parsed body off to
* `BroadcastService.verifyAndEmit()`, and translate its structured
* result into HTTP. All the cryptography, replay protection, and
* event-bus dispatch live in the service so they're reusable from
* tests / direct callers.
* `BroadcastService.verifyAndEmit()`, and translate its structured result into
* HTTP. All the cryptography, replay protection, and event-bus dispatch live in
* the service so they're reusable from tests / direct callers.
*
* Mounted with `subdomain: '*'` (any host) because peers reach the
* webhook through the ALB DNS, not the public `api.<domain>` subdomain,
* so the host header can be an internal ALB hostname rather than
* `api.<domain>` or `<domain>`. Authentication is via the HMAC +
* peer-id + nonce triple, not the host.
* Mounted with `subdomain: '*'` (any host) because peers reach the webhook
* through the ALB DNS, not the public `api.<domain>` subdomain, so the host
* header can be an internal ALB hostname rather than `api.<domain>` or
* `<domain>`. Authentication is via the HMAC + peer-id + nonce triple, not the
* host.
*
* `req.rawBody` is captured by the global JSON parser and is what the
* HMAC verifies against — do NOT switch this route to a custom body
* parser without preserving the raw bytes.
* `req.rawBody` is captured by the global JSON parser and is what the HMAC
* verifies against — do NOT switch this route to a custom body parser without
* preserving the raw bytes.
*/
@Controller('/broadcast')
export class BroadcastController extends PuterController {
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -28,9 +28,23 @@ const ALLOWED_SORT_ORDER = ['asc', 'desc'];
* Desktop/UI preference routes.
*
* Two categories:
* - User-level: desktop background, taskbar items (UserStore)
* - Folder-level: layout, sort_by/sort_order (fsentries table)
*
* - User-level: desktop background, taskbar items (UserStore)
* - Folder-level: layout, sort_by/sort_order (fsentries table)
*/
/**
* Desktop preference writes — background, taskbar, layout, sort order. All four
* persist to the user row on every call, and the GUI fires them on direct user
* action, so a per-minute ceiling well above human speed is enough to catch a
* stuck client.
*/
const PREFERENCE_WRITE_LIMIT = {
scope: 'desktop-preference',
limit: 120,
window: 60_000,
key: 'user',
};
export class DesktopController extends PuterController {
constructor(config, clients, stores, services) {
super(config, clients, stores, services);
@@ -52,6 +66,7 @@ export class DesktopController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: PREFERENCE_WRITE_LIMIT,
},
async (req, res) => {
const { url, color, fit } = req.body ?? {};
@@ -107,6 +122,7 @@ export class DesktopController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: PREFERENCE_WRITE_LIMIT,
},
async (req, res) => {
const { items } = req.body ?? {};
@@ -133,6 +149,7 @@ export class DesktopController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: PREFERENCE_WRITE_LIMIT,
},
async (req, res) => {
const { item_uid, item_path, layout } = req.body ?? {};
@@ -160,6 +177,7 @@ export class DesktopController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: PREFERENCE_WRITE_LIMIT,
},
async (req, res) => {
const { item_uid, item_path, sort_by, sort_order } =
@@ -195,16 +213,15 @@ export class DesktopController extends PuterController {
// -- Helpers ------------------------------------------------------
/**
* Update columns on an actor-owned fsentry. Accepts either `item_uid`
* (fast path) or `item_path` (path lookups in FSEntryStore now have
* a recursive-CTE fallback for legacy rows with a NULL `path` column,
* so this works for old accounts too).
* Update columns on an actor-owned fsentry. Accepts either `item_uid` (fast
* path) or `item_path` (path lookups in FSEntryStore now have a
* recursive-CTE fallback for legacy rows with a NULL `path` column, so this
* works for old accounts too).
*
* Ownership: `entry.user_id === actor.user.id`. Kept as added
* validation against bad paths — the previous "drop user_id entirely"
* theory turned out to be wrong (the actual legacy issue was NULL
* paths, not user_id drift), so this filter doesn't lock out old
* accounts in practice.
* Ownership: `entry.user_id === actor.user.id`. Kept as added validation
* against bad paths — the previous "drop user_id entirely" theory turned
* out to be wrong (the actual legacy issue was NULL paths, not user_id
* drift), so this filter doesn't lock out old accounts in practice.
*/
async #updateFSEntry(actor, { item_uid, item_path }, patch) {
if (!item_uid && !item_path) {
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -124,9 +124,9 @@ const makeSyntheticDriver = () => ({
const buildController = (driver: ReturnType<typeof makeSyntheticDriver>) => {
// The controller reads `this.services?.permission` only when an actor
// is on the request; otherwise the services bag is unused. The
// alarm client is invoked on the rate-limit / concurrency rejection
// paths, so stub a no-op `create`.
// is on the request; otherwise the services bag is unused. The rejection
// paths no longer alarm, but other paths still reach the client, so keep
// a no-op `create` stubbed.
const clients = { alarm: { create: () => {} } };
return new DriverController(
{} as any,
@@ -0,0 +1,437 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Upstream-error translation and rate-limit rejection in `/drivers/call`.
*
* When a driver's upstream provider fails, the caller must see a stable Puter
* error code rather than a raw vendor payload. Each SDK reports its status
* differently (`status`, `response.status`, an AWS `$metadata` block, or only a
* message), so the controller sniffs all four — this suite pins the mapping for
* every shape and asserts the resulting `legacyCode` / `statusCode`, plus the
* `upstreamStatus` / `upstreamCode` diagnostic fields.
*
* The synthetic driver stands in for a provider-backed one: it is the input to
* the translation under test, and it lets a single controller instance cover
* every failure shape without real provider credentials.
*/
/* eslint-disable @typescript-eslint/no-explicit-any */
import { Readable, Writable } from 'node:stream';
import type { Request, RequestHandler, Response } from 'express';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import type { DriverMethodLifecycleEvent } from '../../clients/event/types.js';
import { runWithContext } from '../../core/context.js';
import { configureRateLimit } from '../../core/http/middleware/rateLimit.js';
import { DriverController } from './DriverController.js';
// -- Harness ---------------------------------------------------------
const captureCallHandler = (controller: DriverController): RequestHandler => {
let handler: RequestHandler | undefined;
const fakeRouter = {
post: (path: string, _opts: unknown, h: RequestHandler) => {
if (path === '/call') handler = h;
return fakeRouter;
},
get: () => fakeRouter,
use: () => fakeRouter,
};
controller.registerRoutes(fakeRouter as any);
if (!handler) throw new Error('failed to capture POST /call handler');
return handler;
};
// A real Writable so `result.stream.pipe(res)` behaves like the express
// response it stands in for.
class MockRes extends Writable {
statusCode = 200;
body: unknown;
headers: Record<string, string> = {};
chunks: Buffer[] = [];
override _write(
chunk: Buffer,
_enc: BufferEncoding,
cb: (e?: Error) => void,
) {
this.chunks.push(Buffer.from(chunk));
cb();
}
status(code: number) {
this.statusCode = code;
return this;
}
json(body: unknown) {
this.body = body;
return this;
}
setHeader(k: string, v: string) {
this.headers[k.toLowerCase()] = v;
return this;
}
}
const makeReq = (body: Record<string, unknown>): Request =>
({
body,
headers: {},
query: {},
ip: '127.0.0.1',
socket: { remoteAddress: '127.0.0.1' },
}) as unknown as Request;
interface BuildOptions {
run?: () => unknown;
rateLimit?: unknown;
/**
* Rate-limit buckets are keyed by iface+method, so give tests that exercise
* limits their own namespace.
*/
iface?: string;
}
/**
* Build a controller around one synthetic driver. No actor is attached, so the
* permission scan is skipped and the call reaches the driver method.
*/
const build = (opts: BuildOptions = {}) => {
const events: DriverMethodLifecycleEvent[] = [];
const eventClient = {
emitAndWait: vi.fn(async () => {}),
emit: vi.fn((_key: string, payload: unknown) => {
events.push(payload as DriverMethodLifecycleEvent);
}),
on: vi.fn(),
};
const alarms: Array<{ id: string; severity: string }> = [];
const iface = opts.iface ?? 'test-iface';
const driver = {
driverInterface: iface,
driverName: 'test-driver',
isDefault: true,
...(opts.rateLimit ? { rateLimit: opts.rateLimit } : {}),
run:
opts.run ??
(() => {
throw new Error('no run configured');
}),
};
const controller = new DriverController(
{} as any,
{
event: eventClient,
alarm: {
create: (id: string, _t: string, _f: unknown, sev: string) => {
alarms.push({ id, severity: sev });
},
},
} as any,
{} as any,
{} as any,
{ testDriver: driver } as any,
);
return {
handler: captureCallHandler(controller),
events,
alarms,
driver,
iface,
};
};
const callWith = async (run: () => unknown) => {
const { handler, events } = build({ run });
const res = new MockRes();
const err = await runWithContext({}, () =>
handler(
makeReq({ interface: 'test-iface', method: 'run' }),
res as unknown as Response,
() => {},
),
).then(
() => null,
(e: unknown) => e,
);
return { err, res, events };
};
const throwing = (payload: unknown) => () => {
throw payload;
};
// -- Upstream status extraction --------------------------------------
describe('DriverController upstream error translation', () => {
it('maps an upstream 429 to a Puter 429 with upstream_rate_limited', async () => {
const { err } = await callWith(
throwing(
Object.assign(new Error('slow down'), {
status: 429,
code: 'rate_limit_exceeded',
}),
),
);
expect(err).toMatchObject({
statusCode: 429,
legacyCode: 'upstream_rate_limited',
message: 'slow down',
fields: {
upstreamStatus: 429,
upstreamCode: 'rate_limit_exceeded',
},
});
});
it('maps upstream 401 and 403 to a 500 upstream_auth_failed — never leaking auth state to the caller', async () => {
for (const status of [401, 403]) {
const { err } = await callWith(
throwing(Object.assign(new Error('bad key'), { status })),
);
expect(err).toMatchObject({
statusCode: 500,
legacyCode: 'upstream_auth_failed',
fields: { upstreamStatus: status },
});
}
});
it('maps any upstream 5xx to a 400 upstream_provider_unavailable with a generic message', async () => {
const { err } = await callWith(
throwing(
Object.assign(new Error('internal provider stack trace'), {
status: 503,
}),
),
);
expect(err).toMatchObject({
statusCode: 400,
legacyCode: 'upstream_provider_unavailable',
message: 'AI provider unavailable',
fields: { upstreamStatus: 503 },
});
});
it('maps a generic upstream 4xx to a 400 upstream_bad_request', async () => {
const { err } = await callWith(
throwing(
Object.assign(new Error('unsupported parameter'), {
status: 422,
}),
),
);
expect(err).toMatchObject({
statusCode: 400,
legacyCode: 'upstream_bad_request',
message: 'unsupported parameter',
fields: { upstreamStatus: 422 },
});
});
it('prefers the nested error.message and error.code over the top-level ones', async () => {
const { err } = await callWith(
throwing({
status: 400,
message: 'outer',
code: 'outer_code',
error: { message: 'inner detail', code: 'inner_code' },
}),
);
expect(err).toMatchObject({
legacyCode: 'upstream_bad_request',
message: 'inner detail',
fields: { upstreamStatus: 400, upstreamCode: 'inner_code' },
});
});
it('reads the status from statusCode when `status` is absent', async () => {
const { err } = await callWith(throwing({ statusCode: 429 }));
expect(err).toMatchObject({
statusCode: 429,
legacyCode: 'upstream_rate_limited',
});
});
it('reads the status from a nested response object (axios-style)', async () => {
const { err } = await callWith(
throwing({ response: { status: 429 }, message: 'axios rejected' }),
);
expect(err).toMatchObject({
statusCode: 429,
legacyCode: 'upstream_rate_limited',
fields: { upstreamStatus: 429 },
});
});
it('reads the status from an AWS $metadata block', async () => {
const { err } = await callWith(
throwing({
$metadata: { httpStatusCode: 400 },
message: 'ValidationException',
}),
);
expect(err).toMatchObject({
statusCode: 400,
legacyCode: 'upstream_bad_request',
fields: { upstreamStatus: 400 },
});
});
it('sniffs a status out of the message when nothing else carries one', async () => {
const { err } = await callWith(
throwing(new Error('Request failed with status code 422')),
);
expect(err).toMatchObject({
statusCode: 400,
legacyCode: 'upstream_bad_request',
fields: { upstreamStatus: 422 },
});
});
it('does not treat a bare 4xx-looking number in the message as a status', async () => {
const raw = new Error('the answer contained 404 rows');
const { err } = await callWith(throwing(raw));
// No status could be derived, so the original error passes through
// untranslated rather than being mislabelled.
expect(err).toBe(raw);
});
it('passes an HttpError from the driver straight through', async () => {
const { HttpError } = await import('../../core/http/HttpError.js');
const raw = new HttpError(404, 'no such key', {
legacyCode: 'not_found',
});
const { err } = await callWith(throwing(raw));
expect(err).toBe(raw);
});
it('passes non-object throwables through untouched', async () => {
const { err } = await callWith(throwing('a bare string'));
expect(err).toBe('a bare string');
});
it('passes an error with a sub-400 status through untranslated', async () => {
const raw = { status: 302, message: 'redirected' };
const { err } = await callWith(throwing(raw));
expect(err).toBe(raw);
});
it('emits the error lifecycle event with the original (untranslated) error', async () => {
const raw = Object.assign(new Error('provider down'), { status: 500 });
const { events, err } = await callWith(throwing(raw));
const errorEvent = events.find(
(e) => (e as { phase?: string }).phase === 'error',
) as unknown as Record<string, unknown>;
expect(errorEvent).toBeDefined();
expect(errorEvent.iface).toBe('test-iface');
expect(errorEvent.method).toBe('run');
expect(errorEvent.driver).toBe('test-driver');
expect(errorEvent.error).toBe(raw);
expect(typeof errorEvent.durationMs).toBe('number');
// The caller still sees the translated error.
expect(err).toMatchObject({
legacyCode: 'upstream_provider_unavailable',
});
});
});
// -- Stream results --------------------------------------------------
describe('DriverController stream responses', () => {
it('sets Transfer-Encoding: chunked for a chunked stream result', async () => {
const { handler } = build({
run: () => ({
dataType: 'stream',
content_type: 'audio/mpeg',
chunked: true,
stream: Readable.from(['a', 'b']),
}),
});
const res = new MockRes();
await runWithContext({}, () =>
handler(
makeReq({ interface: 'test-iface', method: 'run' }),
res as unknown as Response,
() => {},
),
);
expect(res.headers['content-type']).toBe('audio/mpeg');
expect(res.headers['transfer-encoding']).toBe('chunked');
// A piped stream never produces a JSON body.
expect(res.body).toBeUndefined();
});
it('omits Transfer-Encoding for a non-chunked stream result', async () => {
const { handler } = build({
run: () => ({
dataType: 'stream',
content_type: 'audio/mpeg',
stream: Readable.from(['a']),
}),
});
const res = new MockRes();
await runWithContext({}, () =>
handler(
makeReq({ interface: 'test-iface', method: 'run' }),
res as unknown as Response,
() => {},
),
);
expect(res.headers['content-type']).toBe('audio/mpeg');
expect('transfer-encoding' in res.headers).toBe(false);
});
});
// -- Rate limiting ---------------------------------------------------
describe('DriverController per-method rate limiting', () => {
beforeEach(() => {
configureRateLimit({ disabled: false } as never);
});
it('answers 429 without alarming once the per-method budget is spent', async () => {
const { handler, alarms, iface } = build({
run: () => ({ ok: true }),
rateLimit: { default: { limit: 1, window: 60_000 } },
iface: 'rate-limited-iface',
});
const call = () =>
runWithContext({}, () =>
handler(
makeReq({ interface: iface, method: 'run' }),
new MockRes() as unknown as Response,
() => {},
),
);
await call();
await expect(call()).rejects.toMatchObject({
statusCode: 429,
legacyCode: 'too_many_requests',
});
// Spending your own budget is the limit working as designed, so it
// must not raise anything — the 429 is the whole signal.
expect(alarms).toEqual([]);
});
});
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -158,11 +158,14 @@ interface MockRes {
sentBody: string | undefined;
contentType: string | undefined;
pipedFrom: Readable | undefined;
listeners: Record<string, Array<() => void>>;
status(code: number): MockRes;
json(body: unknown): MockRes;
setHeader(key: string, value: string): MockRes;
type(t: string): MockRes;
send(body: string): MockRes;
once(event: string, fn: () => void): MockRes;
emit(event: string): void;
}
const makeRes = (): MockRes => {
const res: MockRes = {
@@ -172,6 +175,19 @@ const makeRes = (): MockRes => {
sentBody: undefined,
contentType: undefined,
pipedFrom: undefined,
// `#handleCall` releases a driver's concurrency slot on `finish` /
// `close`, so the stub has to behave like an emitter for any driver
// that declares a `concurrent` policy.
listeners: {},
once(event: string, fn: () => void) {
(this.listeners[event] ??= []).push(fn);
return this;
},
emit(event: string) {
const fns = this.listeners[event] ?? [];
this.listeners[event] = [];
for (const fn of fns) fn();
},
status(code: number) {
this.statusCode = code;
return this;
@@ -1,23 +1,23 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify it under the
* terms of the GNU Affero General Public License as published by the Free
* Software Foundation, either version 3 of the License, or (at your option) any
* later version.
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
* details.
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { metrics } from '@opentelemetry/api';
import type { Request, Response } from 'express';
import { actorUid } from '../../core/actor.js';
import { Context } from '../../core/context.js';
@@ -45,6 +45,38 @@ import { PuterController } from '../types.js';
type DriverInstance = WithLifecycle & Record<string, unknown>;
/**
* Coarse envelope over the whole `/call` surface, so that spreading calls
* across many interfaces can't dodge every individual bucket. Per-driver limits
* are what actually shape traffic.
*
* "Coarse" is a constraint, not a description: for this to be an envelope it
* has to sit _above_ every per-driver budget, or it silently becomes the real
* limit for the widest ones and overrides the tier policy they declare.
* `driverPolicies.test.ts` asserts that ordering against every registered
* driver, so raising a driver's budget past this number fails there rather than
* in production. The headroom above the widest driver (notifications, at
* 3000/30s) is what leaves room for one caller to be busy on two interfaces at
* once.
*/
export const DRIVERS_CALL_LIMIT = {
scope: 'drivers-call',
limit: 8000,
window: 60_000,
key: 'user' as const,
};
// Every driver call is timed here already, for the lifecycle events below.
// Recording the same number as a histogram makes the per-interface latency
// distribution available downstream; which interfaces are worth keeping is a
// collector-side decision, not one made here, so this deliberately records
// everything and lets the export pipeline drop what it doesn't want.
const meter = metrics.getMeter('puter-backend');
const driverCallDuration = meter.createHistogram('driver.call.duration', {
description: 'Wall time of a driver method call',
unit: 'ms',
});
const extractUpstreamStatus = (e: {
status?: number;
statusCode?: number;
@@ -169,12 +201,26 @@ export class DriverController extends PuterController {
registerRoutes(router: PuterRouter): void {
router.post(
'/call',
{ subdomain: 'api', requireAuth: true },
{
subdomain: 'api',
requireAuth: true,
rateLimit: DRIVERS_CALL_LIMIT,
},
this.#handleCall,
);
router.get(
'/list-interfaces',
{ subdomain: 'api', requireAuth: true },
{
subdomain: 'api',
requireAuth: true,
// Static introspection output, read once at boot.
rateLimit: {
scope: 'drivers-list-interfaces',
limit: 60,
window: 60_000,
key: 'user',
},
},
this.#handleListInterfaces,
);
}
@@ -294,19 +340,8 @@ export class DriverController extends PuterController {
if (
!(await checkDriverRateLimit(req, ifaceName, method, rateLimitSpec))
) {
// De-dupe on (iface, method) so a hot loop across many users
// aggregates as occurrences on a single low-severity alarm
// instead of fanning out one per user.
this.clients.alarm.create(
`driver_rate_limit_hit:${ifaceName}:${method}`,
`Driver rate limit hit on ${ifaceName}:${method}`,
{
iface: ifaceName,
method,
userUuid: req.actor?.user?.uuid,
},
'info',
);
// Deliberately unalarmed: a caller spending its own budget is
// the limit working, not an incident. The 429 is the signal.
throw new HttpError(429, 'Too many requests.', {
legacyCode: 'too_many_requests',
});
@@ -329,16 +364,8 @@ export class DriverController extends PuterController {
concurrentSpec,
);
if (!handle.ok) {
this.clients.alarm.create(
`driver_concurrent_limit_hit:${ifaceName}:${method}`,
`Driver concurrency limit hit on ${ifaceName}:${method}`,
{
iface: ifaceName,
method,
userUuid: req.actor?.user?.uuid,
},
'info',
);
// Unalarmed for the same reason as the rate-limit rejection
// above: hitting a declared cap is the cap doing its job.
throw new HttpError(429, 'Too many concurrent requests.', {
legacyCode: 'too_many_requests',
});
@@ -429,6 +456,11 @@ export class DriverController extends PuterController {
() => (fn as (...x: unknown[]) => any).call(driver, args),
);
} catch (e) {
driverCallDuration.record(Date.now() - startedAt, {
driver: ifaceName,
'driver.method': method,
outcome: 'error',
});
this.clients.event?.emit(
`driver.${ifaceName}.${method}.error`,
{
@@ -446,6 +478,15 @@ export class DriverController extends PuterController {
);
throw translateProviderError(e);
}
// Same window the span and the lifecycle events measure: for streamed
// results this is stream start, not stream drain. Worth remembering
// when reading AI latency — it is time-to-first-token, not total.
driverCallDuration.record(Date.now() - startedAt, {
driver: ifaceName,
'driver.method': method,
outcome: 'ok',
});
this.clients.event?.emit(
`driver.${ifaceName}.${method}.after`,
{
@@ -0,0 +1,594 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, RequestHandler, Response } from 'express';
import {
afterAll,
afterEach,
beforeAll,
describe,
expect,
it,
vi,
} from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import type { Actor } from '../../core/actor.js';
import { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterServer } from '../../server.js';
import { AppFeedbackService } from '../../services/feedback/AppFeedbackService.js';
import { setupTestServer } from '../../testUtil.js';
// Boots one real PuterServer (in-memory sqlite + mocked externals) and
// registers AppFeedbackController's decorated routes onto a fresh
// PuterRouter. Tests drive the captured handlers with stub req/res; the
// stores/services underneath are the live wired ones, so rows land in the
// real `app_feedback` table.
let server: PuterServer;
let router: PuterRouter;
beforeAll(async () => {
server = await setupTestServer();
router = new PuterRouter();
server.controllers.appFeedback.registerRoutes(router);
});
afterAll(async () => {
await server?.shutdown();
});
afterEach(() => {
vi.restoreAllMocks();
});
const makeUser = async (): Promise<{ actor: Actor; userId: number }> => {
const username = `fdbk-${Math.random().toString(36).slice(2, 10)}`;
const created = await server.stores.user.create({
username,
uuid: uuidv4(),
password: null,
email: `${username}@test.local`,
free_storage: 100 * 1024 * 1024,
requires_email_confirmation: false,
});
const refreshed = (await server.stores.user.getById(created.id))!;
return {
userId: refreshed.id,
actor: {
user: {
id: refreshed.id,
uuid: refreshed.uuid,
username: refreshed.username,
email: refreshed.email ?? null,
email_confirmed: true,
} as Actor['user'],
},
};
};
const makeApp = async (
ownerUserId: number,
opts: { feedbackEnabled?: boolean; indexUrl?: string; name?: string } = {},
) => {
const name =
opts.name ?? `fdbk-app-${Math.random().toString(36).slice(2, 10)}`;
return await server.stores.app.create(
{
name,
title: `Feedback Test ${name}`,
index_url: opts.indexUrl ?? `https://${name}.example.com`,
...(opts.feedbackEnabled ? { feedback_enabled: 1 } : {}),
},
{ ownerUserId },
);
};
// Feedback is only offered when the deployment can deliver it (email
// transport configured); most tests want that baseline without asserting
// anything about the mail itself.
const mockEmailConfigured = () =>
vi.spyOn(server.clients.email, 'isConfigured', 'get').mockReturnValue(
true,
);
const confirmOwnerEmail = async (userId: number) => {
await server.clients.db.write(
'UPDATE `user` SET `email_confirmed` = ? WHERE `id` = ?',
[server.clients.db.booleanValue(true), userId],
);
const user = await server.stores.user.getById(userId);
if (user) await server.stores.user.invalidate(user);
};
const makeReq = (init: {
body?: unknown;
actor?: Actor;
query?: Record<string, unknown>;
}): Request => {
return {
body: init.body ?? {},
query: init.query ?? {},
headers: {},
actor: init.actor,
} as unknown as Request;
};
const makeRes = () => {
const captured: { statusCode: number; body: unknown } = {
statusCode: 200,
body: undefined,
};
const res = {
json: vi.fn((value: unknown) => {
captured.body = value;
return res;
}),
status: vi.fn((code: number) => {
captured.statusCode = code;
return res;
}),
};
return { res: res as unknown as Response, captured };
};
const findRoute = (method: string, path: string) => {
const route = router.routes.find(
(r) => r.method === method && r.path === path,
);
if (!route) throw new Error(`No ${method.toUpperCase()} ${path} route`);
return route;
};
const callRoute = async (
method: string,
path: string,
req: Request,
res: Response,
) => {
const handler: RequestHandler = findRoute(method, path).handler;
await handler(req, res, () => {
throw new Error('handler called next() unexpectedly');
});
};
const submit = (actor: Actor, body: unknown) => {
const { res, captured } = makeRes();
return callRoute('post', '/', makeReq({ body, actor }), res).then(
() => captured,
);
};
// ── Route gates ─────────────────────────────────────────────────────
describe('AppFeedbackController route options', () => {
it('rejects app actors and cross-origin pages on submit', () => {
const { options } = findRoute('post', '/');
// requireUserActor is what makes feedback impossible to submit
// programmatically with an app token; guiOriginOnly keeps
// cross-origin browser pages out even with a leaked user token.
expect(options.requireUserActor).toBe(true);
expect(options.guiOriginOnly).toBe(true);
});
it('stacks a per-user budget with a per-IP backstop', () => {
const { options } = findRoute('post', '/');
const limits = options.rateLimit;
expect(Array.isArray(limits)).toBe(true);
const keys = (limits as Array<{ key?: unknown }>).map((l) => l.key);
expect(keys).toContain('user');
expect(keys).toContain('ip');
});
it('requires a user actor on the target pre-flight too', () => {
const { options } = findRoute('get', '/target');
expect(options.requireUserActor).toBe(true);
});
});
// ── GET /app-feedback/target ────────────────────────────────────────
describe('AppFeedbackController GET /target', () => {
it('throws 400 when neither or both of app/origin are given', async () => {
const { actor } = await makeUser();
for (const query of [
{},
{ app: 'x', origin: 'https://x.example.com' },
]) {
const { res } = makeRes();
await expect(
callRoute('get', '/target', makeReq({ query, actor }), res),
).rejects.toMatchObject({ statusCode: 400 });
}
});
it('reports enabled:false for an unknown app', async () => {
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({ query: { app: 'no-such-app-xyz' }, actor }),
res,
);
expect(captured.body).toEqual({ enabled: false, app: null });
});
it('reports enabled:false for an app that has not opted in', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId);
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({ query: { app: app.name }, actor }),
res,
);
expect(captured.body).toMatchObject({ enabled: false });
});
it('reports enabled:true with canonical title/name for an opted-in app', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({ query: { app: app.uid }, actor }),
res,
);
expect(captured.body).toEqual({
enabled: true,
app: { name: app.name, title: app.title },
});
});
it('resolves an opted-in app whose name starts with "app-"', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const name = `app-fdbk-${Math.random().toString(36).slice(2, 10)}`;
const app = await makeApp(ownerId, { feedbackEnabled: true, name });
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({ query: { app: name }, actor }),
res,
);
expect(captured.body).toEqual({
enabled: true,
app: { name: app.name, title: app.title },
});
});
it('resolves an origin to the app whose index_url it matches', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const origin = new URL(app.index_url).origin;
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({ query: { origin }, actor }),
res,
);
expect(captured.body).toEqual({
enabled: true,
app: { name: app.name, title: app.title },
});
});
it('reports enabled:false when the email transport is unconfigured', async () => {
// No mockEmailConfigured(): this is the self-hosted no-SMTP default.
// Feedback that can never be delivered must not be solicited.
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({ query: { app: app.name }, actor }),
res,
);
expect(captured.body).toMatchObject({ enabled: false });
});
it('reports enabled:false for an origin with no registered app', async () => {
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({
query: { origin: 'https://nobody-registered.example.com' },
actor,
}),
res,
);
expect(captured.body).toEqual({ enabled: false, app: null });
});
});
// ── POST /app-feedback ──────────────────────────────────────────────
describe('AppFeedbackController POST /', () => {
it('throws 400 when message is missing or not a string', async () => {
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor } = await makeUser();
for (const message of [undefined, 12345, '']) {
await expect(
submit(actor, { app: app.name, message }),
).rejects.toMatchObject({ statusCode: 400 });
}
});
it('throws 400 when both app and origin are given', async () => {
const { actor } = await makeUser();
await expect(
submit(actor, {
app: 'x',
origin: 'https://x.example.com',
message: 'hi',
}),
).rejects.toMatchObject({ statusCode: 400 });
});
it('throws 400 when the origin exceeds the stored column size', async () => {
// source_origin is VARCHAR(2048) on MySQL/Postgres; a longer origin
// must be rejected up front, not fail (or silently truncate) at the
// INSERT after passing every other validation.
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const origin = `${new URL(app.index_url).origin}/${'x'.repeat(2500)}`;
const { actor } = await makeUser();
await expect(
submit(actor, { origin, message: 'hi' }),
).rejects.toMatchObject({ statusCode: 400 });
});
it('throws 403 feedback_not_enabled when the app has not opted in', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId);
const { actor } = await makeUser();
await expect(
submit(actor, { app: app.name, message: 'hi there' }),
).rejects.toMatchObject({
statusCode: 403,
legacyCode: 'feedback_not_enabled',
});
});
it('throws 403 for an unknown app and an unknown origin alike', async () => {
const { actor } = await makeUser();
await expect(
submit(actor, { app: 'no-such-app-xyz', message: 'hi' }),
).rejects.toMatchObject({ statusCode: 403 });
await expect(
submit(actor, {
origin: 'https://nobody-registered.example.com',
message: 'hi',
}),
).rejects.toMatchObject({ statusCode: 403 });
});
it('throws 403 feedback_not_enabled when the email transport is unconfigured', async () => {
// No mockEmailConfigured(): the opted-in app must still refuse — a
// stored row nothing can read, sold to the sender as delivered, is
// worse than an honest refusal.
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor } = await makeUser();
await expect(
submit(actor, { app: app.name, message: 'into the void' }),
).rejects.toMatchObject({
statusCode: 403,
legacyCode: 'feedback_not_enabled',
});
});
it('throws 400 when the message exceeds the length limit', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor } = await makeUser();
await expect(
submit(actor, {
app: app.name,
message: 'x'.repeat(
AppFeedbackService.MESSAGE_MAX_LENGTH + 1,
),
}),
).rejects.toMatchObject({ statusCode: 400 });
});
it('stores a normalized row and responds with an empty object', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor, userId } = await makeUser();
const captured = await submit(actor, {
app: app.uid,
message: ' Great\r\napp! ',
context: 'app',
});
expect(captured.body).toEqual({});
const rows = (await server.clients.db.read(
'SELECT * FROM `app_feedback` WHERE `user_id` = ?',
[userId],
)) as Array<Record<string, unknown>>;
expect(rows).toHaveLength(1);
expect(rows[0]).toMatchObject({
app_uid: app.uid,
message: 'Great\napp!',
source_env: 'app',
source_origin: null,
});
// Engine-agnostic reads: pg returns BIGINT as string and BOOLEAN as
// boolean, sqlite returns numbers for both.
expect(Number(rows[0].app_id)).toBe(app.id);
expect(Boolean(rows[0].email_sent)).toBe(false);
expect(typeof rows[0].uid).toBe('string');
expect(Number.isFinite(Number(rows[0].created_at))).toBe(true);
});
it('records the attested origin on web submissions', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const origin = new URL(app.index_url).origin;
const { actor, userId } = await makeUser();
await submit(actor, { origin, message: 'from the web', context: 'web' });
const rows = (await server.clients.db.read(
'SELECT `source_env`, `source_origin` FROM `app_feedback` WHERE `user_id` = ?',
[userId],
)) as Array<Record<string, unknown>>;
expect(rows).toHaveLength(1);
expect(rows[0]).toMatchObject({
source_env: 'web',
source_origin: origin,
});
});
it('enforces the per-user-per-app daily cap with 429', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor, userId } = await makeUser();
const since = Math.floor(Date.now() / 1000);
for (let i = 0; i < AppFeedbackService.PER_USER_APP_DAILY_LIMIT; i++) {
await server.stores.appFeedback.create({
appId: app.id,
appUid: app.uid,
userId,
message: `seed ${i}`,
});
}
expect(
await server.stores.appFeedback.countByUserAndAppSince(
userId,
app.id,
since - 60,
),
).toBe(AppFeedbackService.PER_USER_APP_DAILY_LIMIT);
await expect(
submit(actor, { app: app.name, message: 'one too many' }),
).rejects.toMatchObject({
statusCode: 429,
legacyCode: 'too_many_requests',
});
});
it('rolls back the stored row when a concurrent burst breaches the cap', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor, userId } = await makeUser();
for (let i = 0; i < AppFeedbackService.PER_USER_APP_DAILY_LIMIT; i++) {
await server.stores.appFeedback.create({
appId: app.id,
appUid: app.uid,
userId,
message: `seed ${i}`,
});
}
// Simulate the losing side of the check-then-insert race: the
// pre-insert check reads a stale under-cap count; the post-insert
// recount (real implementation) sees the truth.
vi.spyOn(
server.stores.appFeedback,
'countByUserAndAppSince',
).mockResolvedValueOnce(0);
await expect(
submit(actor, { app: app.name, message: 'raced past the cap' }),
).rejects.toMatchObject({
statusCode: 429,
legacyCode: 'too_many_requests',
});
const rows = (await server.clients.db.read(
'SELECT COUNT(*) AS n FROM `app_feedback` WHERE `user_id` = ?',
[userId],
)) as Array<{ n: unknown }>;
expect(Number(rows[0]?.n)).toBe(
AppFeedbackService.PER_USER_APP_DAILY_LIMIT,
);
});
it('enforces the per-user daily cap across apps with 429', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const target = await makeApp(ownerId, { feedbackEnabled: true });
const other = await makeApp(ownerId, { feedbackEnabled: true });
const { actor, userId } = await makeUser();
for (let i = 0; i < AppFeedbackService.PER_USER_DAILY_LIMIT; i++) {
await server.stores.appFeedback.create({
appId: other.id,
appUid: other.uid,
userId,
message: `seed ${i}`,
});
}
await expect(
submit(actor, { app: target.name, message: 'over the limit' }),
).rejects.toMatchObject({ statusCode: 429 });
});
});
// ── Owner email delivery ────────────────────────────────────────────
// Which submissions get emailed, and what the mail contains, is service
// logic covered in AppFeedbackService.test.ts. What the controller owes the
// caller is that mail trouble never becomes the sender's problem.
describe('AppFeedbackController owner email', () => {
it('a failing email send never fails the request', async () => {
mockEmailConfigured();
vi.spyOn(server.clients.email, 'send').mockRejectedValue(
new Error('smtp down'),
);
const { userId: ownerId } = await makeUser();
await confirmOwnerEmail(ownerId);
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor, userId } = await makeUser();
const captured = await submit(actor, {
app: app.name,
message: 'still stored',
});
expect(captured.body).toEqual({});
const rows = (await server.clients.db.read(
'SELECT `email_sent` FROM `app_feedback` WHERE `user_id` = ?',
[userId],
)) as Array<{ email_sent: unknown }>;
expect(rows).toHaveLength(1);
expect(Boolean(rows[0]?.email_sent)).toBe(false);
});
});
@@ -0,0 +1,169 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, Response } from 'express';
import { Controller, Get, Post } from '../../core/http/decorators.js';
import { HttpError } from '../../core/http/HttpError.js';
import { AppFeedbackService } from '../../services/feedback/AppFeedbackService.js';
import { PuterController } from '../types.js';
/**
* Endpoints behind the "send feedback to this app's developer" dialog
* (`puter.ui.showFeedbackDialog()`). Only the GUI (desktop dialog or the
* puter.com popup) calls these; apps cannot — both routes reject app actors,
* which is what makes feedback impossible to submit programmatically on a
* user's behalf.
*
* The target may be named either by `app` (uid or name — the desktop knows
* which app asked) or by `origin` (external site — the popup passes its
* browser-attested opener origin). Exactly one must be provided.
*/
/** Sanity cap on the raw body field; the service enforces the real limit. */
const RAW_MESSAGE_CAP = 50_000;
// Upper bound on the `app`/`origin` target params. Must not exceed the
// `source_origin` column (VARCHAR(2048) on MySQL/Postgres): the raw origin is
// stored verbatim, and a longer value would fail the INSERT after passing
// every validation — or be silently truncated on non-strict MySQL.
const TARGET_PARAM_MAX_LENGTH = 2048;
const readTargetParam = (value: unknown): string | undefined => {
return typeof value === 'string' &&
value.length > 0 &&
value.length <= TARGET_PARAM_MAX_LENGTH
? value
: undefined;
};
@Controller('/app-feedback')
export class AppFeedbackController extends PuterController {
/**
* GET /app-feedback/target — pre-flight for the dialog: whether the target
* app accepts feedback, plus its canonical title/name for display. Reveals
* nothing that `puter.apps.get` doesn't already.
*/
@Get('/target', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: {
scope: 'app-feedback-target',
limit: 60,
window: 60_000,
key: 'user',
},
})
async target(req: Request, res: Response): Promise<void> {
const app = readTargetParam(req.query.app);
const origin = readTargetParam(req.query.origin);
if (!app === !origin) {
throw new HttpError(
400,
'Exactly one of `app` and `origin` is required',
{ legacyCode: 'bad_request' },
);
}
const service = this.services.appFeedback as AppFeedbackService;
res.json(await service.getTarget({ app, origin }));
}
/**
* POST /app-feedback — store one feedback message and email the app's
* developer. Strict limits: the route limits below are the cheap first
* line; AppFeedbackService enforces durable per-user/per-app daily caps
* from the database (the route limiter fails open, the DB caps don't).
*/
@Post('/', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
// Submissions only ever originate from our own GUI pages (desktop
// dialog / popup). Cross-origin browser pages get stopped here even
// if they somehow hold a user token; non-browser clients still pass
// and are handled by the caps.
guiOriginOnly: true,
rateLimit: [
{
scope: 'app-feedback-user',
limit: 5,
window: 30 * 60_000,
key: 'user',
},
// IP backstop so freshly minted accounts can't stack per-user
// budgets from one machine.
{
scope: 'app-feedback-ip',
limit: 30,
window: 24 * 60 * 60_000,
key: 'ip',
},
],
})
async submit(req: Request, res: Response): Promise<void> {
const body = req.body ?? {};
const app = readTargetParam(body.app);
const origin = readTargetParam(body.origin);
if (!app === !origin) {
throw new HttpError(
400,
'Exactly one of `app` and `origin` is required',
{ legacyCode: 'bad_request' },
);
}
const message = body.message;
if (typeof message !== 'string' || message.length === 0) {
throw new HttpError(400, '`message` is required', {
legacyCode: 'bad_request',
});
}
if (message.length > RAW_MESSAGE_CAP) {
throw new HttpError(
400,
`\`message\` is too long (max ${AppFeedbackService.MESSAGE_MAX_LENGTH} characters)`,
{ legacyCode: 'bad_request' },
);
}
const sourceEnv =
body.context === 'app' || body.context === 'web'
? body.context
: undefined;
const userId = req.actor?.user?.id;
if (!userId) {
throw new HttpError(401, 'Unauthorized', {
legacyCode: 'unauthorized',
});
}
const service = this.services.appFeedback as AppFeedbackService;
await service.submit({
userId,
app,
origin,
message,
sourceEnv,
sourceOrigin: origin ?? null,
});
res.json({});
}
}
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
+24 -22
View File
@@ -1,28 +1,27 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify it under the
* terms of the GNU Affero General Public License as published by the Free
* Software Foundation, either version 3 of the License, or (at your option) any
* later version.
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
* details.
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, Response } from 'express';
import type { Readable } from 'node:stream';
import { v4 as uuidv4 } from 'uuid';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import type { Actor } from '../../core/actor.js';
import { makeActor, type Actor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
@@ -928,7 +927,7 @@ describe('FSController.searchEntries', () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appUid = `app-search-${uuidv4()}`;
const appActor: Actor = { ...userActor, app: { uid: appUid } };
const appActor = makeActor({ ...userActor, app: { uid: appUid } });
const needle = `appneedle-${Math.random().toString(36).slice(2, 8)}`;
// User-owned entry outside AppData — must NOT appear for the app.
@@ -979,7 +978,7 @@ describe('FSController.searchEntries', () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appUid = `app-search-${uuidv4()}`;
const appActor: Actor = { ...userActor, app: { uid: appUid } };
const appActor = makeActor({ ...userActor, app: { uid: appUid } });
const needle = `appneedle-${Math.random().toString(36).slice(2, 8)}`;
// Only seed outside AppData — the app must not be able to find it.
@@ -1128,7 +1127,7 @@ describe('FSController.mkdirEntry', () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appUid = `app-mkdir-${uuidv4()}`;
const appActor: Actor = { ...userActor, app: { uid: appUid } };
const appActor = makeActor({ ...userActor, app: { uid: appUid } });
const target = `/${username}/AppData/${appUid}`;
await withActor(userActor, () =>
@@ -1890,10 +1889,10 @@ describe('FSController.readdirEntries recursive', () => {
it('masks denials for app-under-user actors as a 404 (legacy parity)', async () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appActor: Actor = {
const appActor = makeActor({
...userActor,
app: { uid: `app-readdir-${uuidv4()}` },
};
});
// The user's Documents is outside the app's AppData subtree, so the
// app can't list it. Legacy `/readdir` masks this as a 404
// subject_does_not_exist rather than leaking a 403.
@@ -2373,15 +2372,18 @@ describe('FSController.statEntry additional branches', () => {
// ── #getReportedCosts ───────────────────────────────────────────────
describe('FSController.getReportedCosts', () => {
it('mirrors every FS_COSTS entry as a per-byte line item', async () => {
const { FS_COSTS } = await import('./costs.js');
it('mirrors every storage-operation price as a per-operation line item', async () => {
const { STORAGE_OP_COSTS } =
await import('../../services/metering/costs.js');
const reported = controller.getReportedCosts();
expect(reported.length).toBe(Object.keys(FS_COSTS).length);
for (const [usageType, ucentsPerUnit] of Object.entries(FS_COSTS)) {
expect(reported.length).toBe(Object.keys(STORAGE_OP_COSTS).length);
for (const [usageType, ucentsPerUnit] of Object.entries(
STORAGE_OP_COSTS,
)) {
expect(reported).toContainEqual({
usageType,
ucentsPerUnit,
unit: 'byte',
unit: 'operation',
source: 'controller:fs',
});
}
+188 -69
View File
@@ -1,21 +1,20 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify it under the
* terms of the GNU Affero General Public License as published by the Free
* Software Foundation, either version 3 of the License, or (at your option) any
* later version.
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
* details.
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import Busboy from 'busboy';
@@ -23,7 +22,6 @@ import type { Request, Response } from 'express';
import { posix as pathPosix } from 'node:path';
import { pipeline } from 'node:stream/promises';
import type { Actor } from '../../core/actor.js';
import { effectiveActorApp } from '../../core/actor.js';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { Controller, Get, Post } from '../../core/http/decorators.js';
@@ -40,7 +38,19 @@ import {
} from '../../util/concurrency.js';
import { applyInlineContentSecurity } from '../../util/inlineContentSecurity.js';
import { PuterController } from '../types.js';
import { FS_COSTS } from './costs.js';
import { STORAGE_OP_COSTS } from '../../services/metering/costs.js';
import {
FS_MULTIPART_LIMIT,
FS_MUTATE_LIMIT,
FS_READ_CONCURRENT,
FS_READ_LIMIT,
FS_READDIR_LIMIT,
FS_SEARCH_CONCURRENT,
FS_SEARCH_LIMIT,
FS_STAT_LIMIT,
FS_WRITE_CONCURRENT,
FS_WRITE_LIMIT,
} from './limits.js';
import {
assertAccess as assertLegacyAccess,
fsEntryMimeType,
@@ -107,16 +117,26 @@ const DEFAULT_BATCH_WRITE_SIDE_EFFECT_CONCURRENCY = 8;
@Controller('/fs')
export class FSController extends PuterController {
// Object-store requests are reported here because the filesystem is what
// makes them. Bytes leaving the server are not: they are metered for every
// response, so `MeteringService` prices them.
override getReportedCosts() {
return Object.entries(FS_COSTS).map(([usageType, ucentsPerUnit]) => ({
usageType,
ucentsPerUnit,
unit: 'byte',
source: 'controller:fs',
}));
return Object.entries(STORAGE_OP_COSTS).map(
([usageType, ucentsPerUnit]) => ({
usageType,
ucentsPerUnit,
unit: 'operation',
source: 'controller:fs',
}),
);
}
@Post('/startWrite', { subdomain: 'api', requireVerified: true })
@Post('/startWrite', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
})
async startWrite(
req: Request<RouteParams, null, SignedWriteRequest>,
res: Response<ClientSignedWriteResponse>,
@@ -174,7 +194,12 @@ export class FSController extends PuterController {
);
}
@Post('/startBatchWrite', { subdomain: 'api', requireVerified: true })
@Post('/startBatchWrite', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
})
async startBatchWrites(
req: Request<RouteParams, null, SignedWriteRequest[]>,
res: Response<ClientSignedWriteResponse[]>,
@@ -274,7 +299,12 @@ export class FSController extends PuterController {
);
}
@Post('/completeWrite', { subdomain: 'api', requireVerified: true })
@Post('/completeWrite', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
})
async completeWrite(
req: Request<RouteParams, null, CompleteWriteRequest>,
res: Response<ClientCompleteWriteResponse>,
@@ -305,7 +335,12 @@ export class FSController extends PuterController {
);
}
@Post('/completeBatchWrite', { subdomain: 'api', requireVerified: true })
@Post('/completeBatchWrite', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
})
async completeBatchWrites(
req: Request<RouteParams, null, CompleteWriteRequest[]>,
res: Response<ClientCompleteWriteResponse[]>,
@@ -348,7 +383,11 @@ export class FSController extends PuterController {
);
}
@Post('/abortWrite', { subdomain: 'api', requireVerified: true })
@Post('/abortWrite', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MULTIPART_LIMIT,
})
async abortWrite(
req: Request<RouteParams, null, AbortWriteRequest>,
res: Response<{ ok: true }>,
@@ -364,7 +403,12 @@ export class FSController extends PuterController {
res.json({ ok: true });
}
@Post('/signMultipartParts', { subdomain: 'api', requireVerified: true })
@Post('/signMultipartParts', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
})
async signMultipartParts(
req: Request<RouteParams, null, SignMultipartPartsRequest>,
res: Response<ClientSignMultipartPartsResponse>,
@@ -377,7 +421,13 @@ export class FSController extends PuterController {
res.json(this.#withoutStorageInternals(response));
}
@Post('/write', { subdomain: 'api', requireVerified: true })
@Post('/write', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_WRITE_LIMIT,
concurrent: FS_WRITE_CONCURRENT,
})
async write(
req: Request<RouteParams, null, WriteRequest>,
res: Response<ClientWriteResponse>,
@@ -423,7 +473,13 @@ export class FSController extends PuterController {
res.json(this.#withRequiredClientFsEntry(updatedResponse));
}
@Post('/batchWrite', { subdomain: 'api', requireVerified: true })
@Post('/batchWrite', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_WRITE_LIMIT,
concurrent: FS_WRITE_CONCURRENT,
})
async batchWrites(
req: Request<RouteParams, null, WriteRequest[]>,
res: Response<ClientWriteResponse[]>,
@@ -439,6 +495,11 @@ export class FSController extends PuterController {
let parseFailure: Error | null = null;
const uploadPromises: Promise<UploadedBatchWriteItem | null>[] = [];
const uploadedIndexes = new Set<number>();
// Manifest indexes are client-chosen and may skip values (an
// ignored `.DS_Store`, or a sparse numbering). `prepareBatchWrites`
// re-indexes by array position, so the two spaces have to be
// bridged explicitly or file parts get paired with the wrong item.
const preparedIndexByManifestIndex = new Map<number, number>();
let fileOrderIndex = 0;
const failParse = (error: unknown) => {
@@ -549,6 +610,12 @@ export class FSController extends PuterController {
{ pathAlreadyNormalized: true },
);
activeManifestItems.forEach((item, position) => {
preparedIndexByManifestIndex.set(
item.index,
position,
);
});
preparedBatch =
await this.services.fs.prepareBatchWrites(
userId,
@@ -621,9 +688,13 @@ export class FSController extends PuterController {
}
uploadedIndexes.add(itemIndex);
const preparedIndex =
preparedIndexByManifestIndex.get(itemIndex);
const preparedItem =
preparedBatch.itemsByIndex.get(itemIndex);
if (!preparedItem) {
preparedIndex === undefined
? undefined
: preparedBatch.itemsByIndex.get(preparedIndex);
if (preparedIndex === undefined || !preparedItem) {
throw new HttpError(
400,
`Batch write metadata was not found for index ${itemIndex}`,
@@ -641,7 +712,7 @@ export class FSController extends PuterController {
return await this.services.fs.uploadPreparedBatchItem({
preparedBatch,
itemIndex,
itemIndex: preparedIndex,
fileContent: stream,
uploadTracker,
});
@@ -653,6 +724,12 @@ export class FSController extends PuterController {
}
})();
uploadPromises.push(uploadPromise);
// Failures are collected with `Promise.allSettled` once
// parsing finishes, which is many ticks away — attach an
// inert handler now so an immediate rejection (e.g. a file
// part arriving before the manifest) isn't reported as an
// unhandled rejection in the meantime.
void uploadPromise.catch(() => undefined);
});
const parsingComplete = new Promise<void>((resolve, reject) => {
@@ -662,6 +739,18 @@ export class FSController extends PuterController {
req.pipe(busboy);
await parsingComplete;
// A manifest that failed to parse never sets
// `manifestPreparationPromise`, so surface the real reason
// (invalid JSON, bad item index, duplicate index → 409) before
// falling back to "no manifest was sent at all". Scoped to that
// case on purpose: with no prepared batch nothing can have been
// uploaded, so there is nothing to clean up. A parse failure that
// arrives *after* a manifest was accepted has to fall through to
// the cleanup path below, which sweeps objects already in storage.
if (parseFailure && !manifestPreparationPromise) {
await Promise.allSettled(uploadPromises);
throw parseFailure;
}
if (!manifestPreparationPromise) {
await Promise.allSettled(uploadPromises);
throw new HttpError(400, 'Batch write manifest is required', {
@@ -857,7 +946,11 @@ export class FSController extends PuterController {
// -- Read-side routes ------------------------------------------------
@Post('/stat', { subdomain: 'api', requireVerified: true })
@Post('/stat', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_STAT_LIMIT,
})
async statEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -971,12 +1064,20 @@ export class FSController extends PuterController {
* without a JSON body. Every value arrives as a string, so parameter
* parsing goes through the same coercion helpers the POST path uses.
*/
@Get('/readdir', { subdomain: 'api', requireVerified: true })
@Get('/readdir', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_READDIR_LIMIT,
})
async readdirEntriesViaGet(req: Request, res: Response) {
return this.readdirEntries(req, res);
}
@Post('/readdir', { subdomain: 'api', requireVerified: true })
@Post('/readdir', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_READDIR_LIMIT,
})
async readdirEntries(req: Request, res: Response) {
const actor = this.#requireActor(req);
// GET carries its parameters in the query string; POST in the body.
@@ -1172,7 +1273,12 @@ export class FSController extends PuterController {
}
}
@Post('/search', { subdomain: 'api', requireVerified: true })
@Post('/search', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_SEARCH_LIMIT,
concurrent: FS_SEARCH_CONCURRENT,
})
async searchEntries(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -1198,7 +1304,13 @@ export class FSController extends PuterController {
res.json(results);
}
@Get('/read', { subdomain: 'api', requireVerified: true })
@Get('/read', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_READ_LIMIT,
concurrent: FS_READ_CONCURRENT,
})
async readEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const query = this.#toObjectRecord(req.query);
@@ -1242,15 +1354,6 @@ export class FSController extends PuterController {
);
res.status(range ? 206 : 200);
const metering = this.services.metering as
| {
batchIncrementUsages?: (
actor: unknown,
entries: unknown[],
) => void;
}
| undefined;
try {
await pipeline(download.body, res);
} catch {
@@ -1258,28 +1361,15 @@ export class FSController extends PuterController {
// tore down both ends. Response is partially sent; nothing to do.
return;
}
// Meter egress only on successful completion.
if (metering?.batchIncrementUsages && download.contentLength) {
try {
const bytes = download.contentLength;
metering.batchIncrementUsages(actor, [
{
usageType: 'filesystem:egress:bytes',
usageAmount: bytes,
costOverride:
FS_COSTS['filesystem:egress:bytes'] * bytes,
},
]);
} catch {
// ignore — metering is non-critical.
}
}
}
// -- Mutation routes ------------------------------------------------
@Post('/mkdir', { subdomain: 'api', requireVerified: true })
@Post('/mkdir', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
async mkdirEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -1320,7 +1410,11 @@ export class FSController extends PuterController {
res.json(this.#toClientEntry(entry));
}
@Post('/touch', { subdomain: 'api', requireVerified: true })
@Post('/touch', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
async touchEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -1356,7 +1450,11 @@ export class FSController extends PuterController {
res.json(this.#toClientEntry(entry));
}
@Post('/rename', { subdomain: 'api', requireVerified: true })
@Post('/rename', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
async renameEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const body = this.#toObjectRecord(req.body);
@@ -1374,7 +1472,11 @@ export class FSController extends PuterController {
res.json(this.#toClientEntry(renamed));
}
@Post('/delete', { subdomain: 'api', requireVerified: true })
@Post('/delete', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
async deleteEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -1392,7 +1494,11 @@ export class FSController extends PuterController {
res.json({ ok: true });
}
@Post('/move', { subdomain: 'api', requireVerified: true })
@Post('/move', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
async moveEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -1420,7 +1526,12 @@ export class FSController extends PuterController {
res.json(this.#toClientEntry(moved));
}
@Post('/copy', { subdomain: 'api', requireVerified: true })
@Post('/copy', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_MUTATE_LIMIT,
})
async copyEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -1435,6 +1546,7 @@ export class FSController extends PuterController {
await this.#assertAccess(actor, source.path, 'read');
await this.#assertAccess(actor, destinationParent.path, 'write');
const storageAllowanceMax = this.#getStorageAllowanceMaxOverride(req);
const copy = await this.services.fs.copy(userId, {
source,
destinationParent,
@@ -1443,12 +1555,19 @@ export class FSController extends PuterController {
overwrite: this.#toBoolean(body.overwrite) ?? false,
dedupeName:
this.#toBoolean(body.dedupe_name ?? body.change_name) ?? true,
...(storageAllowanceMax !== undefined
? { storageAllowanceMax }
: {}),
});
this.#emitGuiItemAdded(copy);
res.json(this.#toClientEntry(copy));
}
@Post('/mkshortcut', { subdomain: 'api', requireVerified: true })
@Post('/mkshortcut', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
async mkshortcutEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -2404,7 +2523,7 @@ export class FSController extends PuterController {
// search results to this path so app actors can't see entries outside
// their AppData via `/fs/search`.
#appDataScopeForActor(actor: Actor): string | undefined {
const app = effectiveActorApp(actor);
const app = actor.effectiveApp;
if (!app) return undefined;
const username = actor.user?.username;
if (typeof username !== 'string' || username.length === 0)
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -21,7 +21,7 @@ import type { Request, Response } from 'express';
import { Readable } from 'node:stream';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import type { Actor } from '../../core/actor.js';
import { makeActor, type Actor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterServer } from '../../server.js';
@@ -250,7 +250,7 @@ describe('LegacyFSController.mkdir', () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appUid = `app-legacy-mkdir-${uuidv4()}`;
const appActor: Actor = { ...userActor, app: { uid: appUid } };
const appActor = makeActor({ ...userActor, app: { uid: appUid } });
const parent = `/${username}/AppData`;
await withActor(userActor, () =>
@@ -928,6 +928,86 @@ describe('LegacyFSController.copy', () => {
// The entry must still exist — the ghost handler must NOT have run.
expect(await server.stores.fsEntry.getEntryByPath(src)).not.toBeNull();
});
it('surfaces a name collision, then reports and removes the replaced entry on overwrite', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const src = `/${username}/Documents/dup.txt`;
const existing = `/${username}/Pictures/dup.txt`;
for (const p of [src, existing]) {
await withActor(actor, () =>
controller.touch(
makeReq({ body: { path: p }, actor }),
makeRes().res,
),
);
}
// Without overwrite: the v1 conflict contract the GUI's
// replace/skip prompts key on.
await expect(
withActor(actor, () =>
controller.copy(
makeReq({
body: {
source: src,
destination: `/${username}/Pictures`,
},
actor,
}),
makeRes().res,
),
),
).rejects.toMatchObject({
statusCode: 409,
legacyCode: 'item_with_same_name_exists',
fields: { entry_name: 'dup.txt' },
});
const replaced =
(await server.stores.fsEntry.getEntryByPath(existing))!;
// With overwrite: the replaced entry rides along in the response
// (so the caller can drop its row) and item.removed tells every
// other client to do the same — without it they keep a ghost row
// until the directory is re-listed.
const emitSpy = vi.spyOn(server.clients.event, 'emit');
let body: Array<{
copied: { path: string };
overwritten?: { id: string };
}>;
let removedCall: (typeof emitSpy.mock.calls)[number] | undefined;
try {
const { res, captured } = makeRes();
await withActor(actor, () =>
controller.copy(
makeReq({
body: {
source: src,
destination: `/${username}/Pictures`,
overwrite: true,
},
actor,
}),
res,
),
);
body = captured.body as typeof body;
removedCall = emitSpy.mock.calls.find(
([eventName]) => eventName === 'outer.gui.item.removed',
);
} finally {
emitSpy.mockRestore();
}
expect(body[0].copied.path).toBe(existing);
expect(body[0].overwritten?.id).toBe(replaced.uuid);
expect(removedCall).toBeTruthy();
const removedPayload = removedCall?.[1] as {
response?: { uid?: string };
};
expect(removedPayload.response?.uid).toBe(replaced.uuid);
});
});
// ── move ────────────────────────────────────────────────────────────
@@ -1004,6 +1084,88 @@ describe('LegacyFSController.move', () => {
const body = captured.body as { moved: { path: string } };
expect(body.moved.path).toBe(`/${username}/Pictures/bar`);
});
it('surfaces a name collision, then reports and removes the replaced entry on overwrite', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const src = `/${username}/Documents/clash.txt`;
const existing = `/${username}/Pictures/clash.txt`;
for (const p of [src, existing]) {
await withActor(actor, () =>
controller.touch(
makeReq({ body: { path: p }, actor }),
makeRes().res,
),
);
}
// Without overwrite: the v1 conflict contract the GUI's
// replace/skip prompts key on.
await expect(
withActor(actor, () =>
controller.move(
makeReq({
body: {
source: src,
destination: `/${username}/Pictures`,
},
actor,
}),
makeRes().res,
),
),
).rejects.toMatchObject({
statusCode: 409,
legacyCode: 'item_with_same_name_exists',
fields: { entry_name: 'clash.txt' },
});
const replaced =
(await server.stores.fsEntry.getEntryByPath(existing))!;
// With overwrite: the replaced entry rides along in the response
// (so the caller can drop its row) and item.removed tells every
// other client to do the same — without it they keep a ghost row
// until the directory is re-listed.
const emitSpy = vi.spyOn(server.clients.event, 'emit');
let body: {
moved: { path: string };
old_path: string;
overwritten?: { id: string };
};
let removedCall: (typeof emitSpy.mock.calls)[number] | undefined;
try {
const { res, captured } = makeRes();
await withActor(actor, () =>
controller.move(
makeReq({
body: {
source: src,
destination: `/${username}/Pictures`,
overwrite: true,
},
actor,
}),
res,
),
);
body = captured.body as typeof body;
removedCall = emitSpy.mock.calls.find(
([eventName]) => eventName === 'outer.gui.item.removed',
);
} finally {
emitSpy.mockRestore();
}
expect(body.old_path).toBe(src);
expect(body.moved.path).toBe(existing);
expect(body.overwritten?.id).toBe(replaced.uuid);
expect(removedCall).toBeTruthy();
const removedPayload = removedCall?.[1] as {
response?: { uid?: string };
};
expect(removedPayload.response?.uid).toBe(replaced.uuid);
});
});
// ── search ──────────────────────────────────────────────────────────
@@ -1050,7 +1212,7 @@ describe('LegacyFSController.search', () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appUid = `app-legacy-search-${uuidv4()}`;
const appActor: Actor = { ...userActor, app: { uid: appUid } };
const appActor = makeActor({ ...userActor, app: { uid: appUid } });
const needle = `appneedle-${Math.random().toString(36).slice(2, 8)}`;
await withActor(userActor, () =>
@@ -1255,10 +1417,10 @@ describe('LegacyFSController.sign', () => {
},
{ ownerUserId: userActor.user!.id! },
);
const attackerActor: Actor = {
const attackerActor = makeActor({
...userActor,
app: { uid: `attacker-${uuidv4()}` },
};
});
const { res } = makeRes();
await expect(
@@ -1292,7 +1454,7 @@ describe('LegacyFSController.sign', () => {
},
{ ownerUserId: userActor.user!.id! },
);
const appActor: Actor = { ...userActor, app: { uid: ownApp.uid } };
const appActor = makeActor({ ...userActor, app: { uid: ownApp.uid } });
const { res, captured } = makeRes();
await withActor(appActor, () =>
@@ -1572,6 +1734,185 @@ describe('LegacyFSController.openItem (write_url stripping)', () => {
});
});
// ── openItem: the grant is a user-authority action ──────────────────
//
// /open_item writes a user→app ACL row. An app actor reaching it could hand
// itself write on any file it can read — read-only access to a folder would
// silently widen into persistent write on the files inside it, with no second
// consent prompt. The user has to be the one opening the item, and the row it
// leaves can never exceed what that user proved on the entry.
describe('LegacyFSController.openItem (grant scope)', () => {
const makeApp = async (ownerUserId: number) => {
const name = `opener-${uuidv4()}`;
return (await server.stores.app.create(
{
name,
title: 'Opener test app',
index_url: `https://${name}.test/`,
},
{ ownerUserId },
)) as { id: number; uid: string };
};
it('is registered behind the user-session gate', () => {
const router = new PuterRouter();
(
controller as unknown as {
registerRoutes: (r: PuterRouter) => void;
}
).registerRoutes(router);
const route = router.routes.find(
(r) => r.method === 'post' && r.path === '/open_item',
);
expect(route?.options).toMatchObject({
requireUserActor: true,
allowFullAccessToken: true,
});
});
it('rejects an app actor, leaving no grant behind', async () => {
const { actor: userActor, userId } = await makeUser();
const username = userActor.user!.username!;
const target = `/${username}/Documents/app-opened.txt`;
await withActor(userActor, () =>
controller.touch(
makeReq({ body: { path: target }, actor: userActor }),
makeRes().res,
),
);
const entry = await server.stores.fsEntry.getEntryByPath(target);
// The app holds the read the user approved, and is its own default
// opener — the shape that would let it widen that read into write.
const app = await makeApp(userId);
await withActor(userActor, () =>
server.services.permission.grantUserAppPermission(
userActor,
app.uid,
`fs:${entry!.uuid}:read`,
),
);
const appActor = makeActor({
...userActor,
app: { uid: app.uid, id: app.id },
});
const spy = vi
.spyOn(server.services.suggestedApps, 'getSuggestedApps')
.mockResolvedValue([{ uuid: app.uid }] as never);
try {
await expect(
withActor(appActor, () =>
controller.openItem(
makeReq({
body: { uid: entry!.uuid },
actor: appActor,
}),
makeRes().res,
),
),
).rejects.toMatchObject({ statusCode: 403 });
} finally {
spy.mockRestore();
}
expect(
await server.stores.permission.hasUserAppPerm(
userId,
app.id,
`fs:${entry!.uuid}:write`,
),
).toBe(false);
});
it('grants the suggested app write when the caller has write', async () => {
const { actor, userId } = await makeUser();
const target = `/${actor.user!.username}/Documents/owned-open.txt`;
await withActor(actor, () =>
controller.touch(
makeReq({ body: { path: target }, actor }),
makeRes().res,
),
);
const entry = await server.stores.fsEntry.getEntryByPath(target);
const app = await makeApp(userId);
const spy = vi
.spyOn(server.services.suggestedApps, 'getSuggestedApps')
.mockResolvedValue([{ uuid: app.uid }] as never);
try {
await withActor(actor, () =>
controller.openItem(
makeReq({ body: { path: target }, actor }),
makeRes().res,
),
);
} finally {
spy.mockRestore();
}
expect(
await server.stores.permission.hasUserAppPerm(
userId,
app.id,
`fs:${entry!.uuid}:write`,
),
).toBe(true);
});
it('grants only read when the caller is a read-only sharee', async () => {
const victim = await makeUser();
const sharee = await makeUser();
const target = `/${victim.actor.user!.username}/Documents/ro-open.txt`;
await withActor(victim.actor, () =>
controller.touch(
makeReq({ body: { path: target }, actor: victim.actor }),
makeRes().res,
),
);
const entry = await server.stores.fsEntry.getEntryByPath(target);
await server.services.permission.grantUserUserPermission(
victim.actor,
sharee.actor.user!.username!,
`fs:${entry!.uuid}:read`,
{},
);
const app = await makeApp(sharee.userId);
const spy = vi
.spyOn(server.services.suggestedApps, 'getSuggestedApps')
.mockResolvedValue([{ uuid: app.uid }] as never);
try {
await withActor(sharee.actor, () =>
controller.openItem(
makeReq({
body: { uid: entry!.uuid },
actor: sharee.actor,
}),
makeRes().res,
),
);
} finally {
spy.mockRestore();
}
expect(
await server.stores.permission.hasUserAppPerm(
sharee.userId,
app.id,
`fs:${entry!.uuid}:write`,
),
).toBe(false);
expect(
await server.stores.permission.hasUserAppPerm(
sharee.userId,
app.id,
`fs:${entry!.uuid}:read`,
),
).toBe(true);
});
});
// ── requestAppRootDir ───────────────────────────────────────────────
describe('LegacyFSController.requestAppRootDir', () => {
@@ -1602,10 +1943,10 @@ describe('LegacyFSController.requestAppRootDir', () => {
it('rejects with 403 when the actor.app.uid differs from the requested app_uid', async () => {
const { actor } = await makeUser();
const { res } = makeRes();
const appActor = {
const appActor = makeActor({
...actor,
app: { uid: 'app-mismatch' },
} as unknown as Actor;
});
await expect(
withActor(appActor, () =>
controller.requestAppRootDir(
@@ -1623,10 +1964,10 @@ describe('LegacyFSController.requestAppRootDir', () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const appUid = 'app-self';
const appActor = {
const appActor = makeActor({
...actor,
app: { uid: appUid },
} as unknown as Actor;
});
const { res, captured } = makeRes();
await withActor(appActor, () =>
controller.requestAppRootDir(
@@ -2299,10 +2640,10 @@ describe('LegacyFSController.sign app sandbox + write downgrade', () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
// Build an app-under-user actor whose AppData root is the test app.
const appActor = {
const appActor = makeActor({
...actor,
app: { uid: 'sandbox-app' },
} as unknown as Actor;
});
// Create a file *outside* /Documents (anywhere outside AppData/<uid>).
const target = `/${username}/Documents/forbidden.txt`;
@@ -2566,10 +2907,10 @@ describe('LegacyFSController.suggestApps', () => {
it('refuses to look up entries an app actor cannot see', async () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appActor: Actor = {
const appActor = makeActor({
...userActor,
app: { uid: `app-suggest-${uuidv4()}` },
};
});
const target = `/${username}/Documents/probe.txt`;
await withActor(userActor, () =>
controller.touch(
@@ -2637,10 +2978,10 @@ describe('LegacyFSController.suggestApps', () => {
describe('LegacyFSController.readdirSubdomains', () => {
it('returns an empty array for app-under-user actors', async () => {
const { actor: userActor, userId } = await makeUser();
const appActor: Actor = {
const appActor = makeActor({
...userActor,
app: { uid: `app-subd-${uuidv4()}` },
};
});
await server.clients.db.write(
'INSERT INTO `subdomains` (`uuid`, `subdomain`, `user_id`) VALUES (?, ?, ?)',
[uuidv4(), `sd-${uuidv4().slice(0, 8)}`, userId],
@@ -2679,10 +3020,10 @@ describe('LegacyFSController.updateFsentryThumbnail', () => {
it('rejects an app actor probing entries outside AppData with 404', async () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appActor: Actor = {
const appActor = makeActor({
...userActor,
app: { uid: `app-thumb-${uuidv4()}` },
};
});
const target = `/${username}/Documents/thumbme.txt`;
await withActor(userActor, () =>
controller.touch(
@@ -2743,6 +3084,56 @@ describe('LegacyFSController.updateFsentryThumbnail', () => {
const body = captured.body as { thumbnail: string };
expect(typeof body.thumbnail).toBe('string');
});
// The write ACL here covers the entry being annotated, not whatever the
// thumbnail string points at. A storage pointer stored verbatim is later
// presigned (and deleted) by the thumbnails extension using the server's
// own credentials, so the owner of one file could name another user's
// object — an fs object's key is its fsentry uuid — and have the server
// read or destroy it. Only inline image data is accepted.
it.each([
[
'an s3:// pointer',
's3://puter-local/00000000-0000-4000-8000-000000000000',
],
['an https URL', 'https://cdn.example.com/x.png'],
['a bare object key', 'thumbnails/whatever'],
])(
'rejects %s instead of storing it verbatim',
async (_label, thumbnail) => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const target = `/${username}/Documents/thumbme3-${uuidv4()}.txt`;
await withActor(actor, () =>
controller.touch(
makeReq({
body: { path: target, set_modified_to_now: true },
actor,
}),
makeRes().res,
),
);
const entry = await server.stores.fsEntry.getEntryByPath(target);
const { res } = makeRes();
await expect(
withActor(actor, () =>
controller.updateFsentryThumbnail(
makeReq({
body: { uid: entry!.uuid, thumbnail },
actor,
}),
res,
),
),
).rejects.toMatchObject({ statusCode: 400 });
const after = await server.stores.fsEntry.getEntryByUuid(
entry!.uuid,
);
expect(after?.thumbnail ?? null).toBeNull();
},
);
});
// ── GET /get-launch-apps ────────────────────────────────────────────
+377 -164
View File
@@ -1,38 +1,41 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify it under the
* terms of the GNU Affero General Public License as published by the Free
* Software Foundation, either version 3 of the License, or (at your option) any
* later version.
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
* details.
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import Busboy from 'busboy';
import type { NextFunction, Request, RequestHandler, Response } from 'express';
import { contentType as contentTypeFromMime } from 'mime-types';
import { posix as pathPosix } from 'node:path';
import type { Actor } from '../../core/actor.js';
import { effectiveActorApp, isAccessTokenActor } from '../../core/actor.js';
import {
assertResolvedActor,
isAccessTokenActor,
makeActor,
} from '../../core/actor.js';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { RouteOptions } from '../../core/http/index.js';
import {
assertNotSuspended,
assertVerifiedAccount,
} from '../../core/http/middleware/gates.js';
import { RouteOptions } from '../../core/http/index.js';
import type { PuterRouter } from '../../core/http/PuterRouter.js';
import type { ACLService } from '../../services/acl/ACLService.js';
import { assertActorHasCredits } from '../../services/metering/enforcement.js';
import type { SignedFile } from '../../util/fileSigning.js';
import { verifySignature } from '../../util/fileSigning.js';
import {
@@ -41,7 +44,24 @@ import {
} from '../../util/hostedAppBacking.js';
import { applyInlineContentSecurity } from '../../util/inlineContentSecurity.js';
import { PuterController } from '../types.js';
import { FS_COSTS } from './costs.js';
import {
FS_BATCH_CONCURRENT,
FS_BATCH_LIMIT,
FS_DF_LIMIT,
FS_HELPER_LIMIT,
FS_MUTATE_LIMIT,
FS_POLL_LIMIT,
FS_READ_CONCURRENT,
FS_READ_LIMIT,
FS_READDIR_LIMIT,
FS_SEARCH_CONCURRENT,
FS_SEARCH_LIMIT,
FS_SIGN_LIMIT,
FS_SIGNED_CONCURRENT,
FS_SIGNED_READ_LIMIT,
FS_SIGNED_WRITE_LIMIT,
FS_STAT_LIMIT,
} from './limits.js';
import {
asRecord,
assertAccess,
@@ -96,6 +116,14 @@ export class LegacyFSController extends PuterController {
subdomain: 'api',
requireVerified: true,
} as RouteOptions;
// Operations that move file content or make object-store requests on
// the caller's behalf, and so are refused to an account with nothing
// left of its budget. The metadata routes above deliberately aren't:
// an account that has run out still has to be able to look at what it
// has and delete it. The signature-authorised routes aren't either —
// they carry no session to answer for, and `/sign` (which does) is
// where the URL that reaches them is minted.
const spends = { ...apiOptions, requireCredits: true } as RouteOptions;
// Signed-URL routes: the handler validates the URL signature itself,
// so no auth gate is applied (matches v1, which mounted these routers
// with no middleware).
@@ -104,40 +132,118 @@ export class LegacyFSController extends PuterController {
} as RouteOptions;
// Core filesystem_api routes — direct handlers over the FS service.
router.post('/stat', apiOptions, this.stat);
router.post('/readdir', apiOptions, this.readdir);
router.post('/mkdir', apiOptions, this.mkdir);
router.post('/copy', apiOptions, this.copy);
router.post('/move', apiOptions, this.move);
router.post('/delete', apiOptions, this.delete);
router.post('/rename', apiOptions, this.rename);
router.post('/touch', apiOptions, this.touch);
router.post('/search', apiOptions, this.search);
router.get('/read', apiOptions, this.read);
// Limits come from `./limits` and carry an explicit `scope`, so these
// draw from the same per-user budget as their v2 counterparts rather
// than handing a caller a second allowance for the same operation.
const mutate = { ...apiOptions, rateLimit: FS_MUTATE_LIMIT };
router.post(
'/stat',
{ ...apiOptions, rateLimit: FS_STAT_LIMIT },
this.stat,
);
router.post(
'/readdir',
{ ...apiOptions, rateLimit: FS_READDIR_LIMIT },
this.readdir,
);
router.post('/mkdir', mutate, this.mkdir);
router.post('/copy', { ...mutate, requireCredits: true }, this.copy);
router.post('/move', mutate, this.move);
router.post('/delete', mutate, this.delete);
router.post('/rename', mutate, this.rename);
router.post('/touch', mutate, this.touch);
router.post(
'/search',
{
...apiOptions,
rateLimit: FS_SEARCH_LIMIT,
concurrent: FS_SEARCH_CONCURRENT,
},
this.search,
);
router.get(
'/read',
{
...spends,
rateLimit: FS_READ_LIMIT,
concurrent: FS_READ_CONCURRENT,
},
this.read,
);
router.get(
'/token-read',
{
subdomain: 'api',
requireVerified: false,
allowAccessToken: true,
// An access token may or may not carry a user, so this shares
// the network-keyed budget the other signed routes use.
rateLimit: FS_SIGNED_READ_LIMIT,
},
this.tokenRead,
);
router.post('/batch', apiOptions, this.batch);
router.post(
'/batch',
{
...spends,
rateLimit: FS_BATCH_LIMIT,
concurrent: FS_BATCH_CONCURRENT,
},
this.batch,
);
// Signed-URL + meta routes.
router.post('/sign', apiOptions, this.sign);
router.post('/writeFile', signedOptions, this.writeFile);
router.get('/file', signedOptions, this.file);
router.all('/df', apiOptions, this.df);
router.post('/open_item', apiOptions, this.openItem);
router.post(
'/sign',
// Gated even though it moves nothing itself: the URL it returns
// outlives the request and is served by a route with no session to
// check, so this is the last point at which the account is known.
{ ...spends, rateLimit: FS_SIGN_LIMIT },
this.sign,
);
router.post(
'/writeFile',
{
...signedOptions,
rateLimit: FS_SIGNED_WRITE_LIMIT,
concurrent: FS_SIGNED_CONCURRENT,
},
this.writeFile,
);
router.get(
'/file',
{
...signedOptions,
rateLimit: FS_SIGNED_READ_LIMIT,
concurrent: FS_SIGNED_CONCURRENT,
},
this.file,
);
router.all('/df', { ...apiOptions, rateLimit: FS_DF_LIMIT }, this.df);
router.post(
'/open_item',
{
...apiOptions,
// Opening an item grants an app access to a file on the user's
// behalf, so only the user's own credential may drive it — an
// app calling it for itself would be widening its own ACL.
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: FS_HELPER_LIMIT,
},
this.openItem,
);
router.post(
'/auth/request-app-root-dir',
apiOptions,
{ ...apiOptions, rateLimit: FS_SIGN_LIMIT },
this.requestAppRootDir,
);
router.post('/auth/check-app-acl', apiOptions, this.checkAppAcl);
router.post(
'/auth/check-app-acl',
{ ...apiOptions, rateLimit: FS_SIGN_LIMIT },
this.checkAppAcl,
);
// `/down` — session-auth'd file download. Unlike `/file` (signed URL)
// this accepts a path on the user's behalf and streams as attachment.
@@ -154,7 +260,10 @@ export class LegacyFSController extends PuterController {
// (CSRF can't forge a header-credentialed request).
allowFullAccessToken: true,
requireVerified: true,
requireCredits: true,
antiCsrf: true,
rateLimit: FS_READ_LIMIT,
concurrent: FS_READ_CONCURRENT,
},
this.down,
);
@@ -165,93 +274,102 @@ export class LegacyFSController extends PuterController {
});
});
router.get('/get-launch-apps', apiOptions, async (req, res) => {
const recommendedSvc = this.services.recommendedApps as unknown as
| { getRecommendedApps?: () => Promise<unknown[]> }
| undefined;
const recommended = recommendedSvc?.getRecommendedApps
? await recommendedSvc.getRecommendedApps()
: [];
router.get(
'/get-launch-apps',
{ ...apiOptions, rateLimit: FS_HELPER_LIMIT },
async (req, res) => {
const recommendedSvc = this.services
.recommendedApps as unknown as
| { getRecommendedApps?: () => Promise<unknown[]> }
| undefined;
const recommended = recommendedSvc?.getRecommendedApps
? await recommendedSvc.getRecommendedApps()
: [];
let recent: unknown[] = [];
const userId = req.actor?.user?.id;
if (userId) {
const recentUids =
(await (
let recent: unknown[] = [];
const userId = req.actor?.user?.id;
if (userId) {
const recentUids =
(await (
this.stores.app as unknown as {
getRecentAppOpens?: (
id: number,
opts?: { limit?: number },
) => Promise<string[]>;
}
).getRecentAppOpens?.(userId, { limit: 10 })) ?? [];
// One batched read for the rows, then the backing checks
// concurrently. Serially awaiting a lookup per uid put ~2
// round trips of latency on every desktop boot.
const appsByUid = await (
this.stores.app as unknown as {
getRecentAppOpens?: (
id: number,
opts?: { limit?: number },
) => Promise<string[]>;
getByUids: (
uids: string[],
) => Promise<Map<string, Record<string, unknown>>>;
}
).getRecentAppOpens?.(userId, { limit: 10 })) ?? [];
// One batched read for the rows, then the backing checks
// concurrently. Serially awaiting a lookup per uid put ~2
// round trips of latency on every desktop boot.
const appsByUid = await (
this.stores.app as unknown as {
getByUids: (
uids: string[],
) => Promise<Map<string, Record<string, unknown>>>;
}
).getByUids(recentUids);
).getByUids(recentUids);
// `recentUids` is ordered most-recent-first; preserve it.
const orderedApps = recentUids
.map((uid) => appsByUid.get(uid))
.filter((app): app is Record<string, unknown> =>
Boolean(app),
// `recentUids` is ordered most-recent-first; preserve it.
const orderedApps = recentUids
.map((uid) => appsByUid.get(uid))
.filter((app): app is Record<string, unknown> =>
Boolean(app),
);
// Don't hand out an index_url whose puter-hosted backing is
// gone or reclaimed. The taskbar launches recents by name (so
// AppDriver's guard applies), but this list is a
// launch-metadata producer like any other — a future consumer
// reading index_url straight off it shouldn't inherit a stale
// origin.
const backingGoneFlags = await Promise.all(
orderedApps.map((app) =>
hostedIndexUrlBackingIsUnavailable({
app,
subdomainStore: this.stores.subdomain,
config: this.config,
}).catch(() => true),
),
);
// Don't hand out an index_url whose puter-hosted backing is
// gone or reclaimed. The taskbar launches recents by name (so
// AppDriver's guard applies), but this list is a
// launch-metadata producer like any other — a future consumer
// reading index_url straight off it shouldn't inherit a stale
// origin.
const backingGoneFlags = await Promise.all(
orderedApps.map((app) =>
hostedIndexUrlBackingIsUnavailable({
app,
subdomainStore: this.stores.subdomain,
config: this.config,
}).catch(() => true),
),
);
recent = orderedApps.map((app, index) => {
const backingGone = backingGoneFlags[index];
return {
uuid: app.uid,
name: app.name,
title: app.title,
icon: app.icon ?? null,
godmode: Boolean(app.godmode),
maximize_on_start: Boolean(app.maximize_on_start),
index_url: backingGone ? null : app.index_url,
...(backingGone
? { privateAccess: buildHostedBackingDenial() }
: {}),
// An app with no owner isn't owned by a Puter user —
// it's an "external" (origin-bootstrapped) app.
external:
app.owner_user_id == null ||
app.owner_user_id === '',
};
});
}
recent = orderedApps.map((app, index) => {
const backingGone = backingGoneFlags[index];
return {
uuid: app.uid,
name: app.name,
title: app.title,
icon: app.icon ?? null,
godmode: Boolean(app.godmode),
maximize_on_start: Boolean(app.maximize_on_start),
index_url: backingGone ? null : app.index_url,
...(backingGone
? { privateAccess: buildHostedBackingDenial() }
: {}),
// An app with no owner isn't owned by a Puter user —
// it's an "external" (origin-bootstrapped) app.
external:
app.owner_user_id == null ||
app.owner_user_id === '',
};
});
}
res.json({ recommended, recent });
},
);
res.json({ recommended, recent });
});
router.post('/suggest_apps', apiOptions, this.suggestApps);
router.post(
'/suggest_apps',
{ ...apiOptions, rateLimit: FS_HELPER_LIMIT },
this.suggestApps,
);
// puter-js polls this to decide whether to purge its in-memory FS
// cache. SocketService bumps a per-user Redis key on every
// `outer.gui.item.*` mutation — read it back here.
router.get(
'/cache/last-change-timestamp',
apiOptions,
{ ...apiOptions, rateLimit: FS_POLL_LIMIT },
async (req, res) => {
const userId = req.actor?.user?.id;
if (!userId) {
@@ -272,10 +390,14 @@ export class LegacyFSController extends PuterController {
},
);
router.post('/readdir-subdomains', apiOptions, this.readdirSubdomains);
router.post(
'/readdir-subdomains',
{ ...apiOptions, rateLimit: FS_HELPER_LIMIT },
this.readdirSubdomains,
);
router.post(
'/update-fsentry-thumbnail',
apiOptions,
{ ...apiOptions, rateLimit: FS_HELPER_LIMIT },
this.updateFsentryThumbnail,
);
@@ -590,26 +712,61 @@ export class LegacyFSController extends PuterController {
'write',
);
// The v1 wire contract reports the entry an overwrite replaced
// (`overwritten`) so clients can drop its stale row/icon. The copy
// deletes that entry, so resolve it beforehand.
const overwriteRequested = getBoolean(body, 'overwrite') ?? false;
let overwrittenEntry = null;
if (overwriteRequested) {
const targetName = getString(body, 'new_name') ?? source.name;
const targetPath =
destinationParent.path === '/'
? `/${targetName}`
: `${destinationParent.path}/${targetName}`;
overwrittenEntry =
await this.stores.fsEntry.getEntryByPath(targetPath);
}
const copy = await this.services.fs.copy(userId, {
source,
destinationParent,
newName: getString(body, 'new_name'),
overwrite: getBoolean(body, 'overwrite') ?? false,
overwrite: overwriteRequested,
dedupeName: getBoolean(body, 'dedupe_name', 'change_name') ?? false,
});
await this.#emitGuiEvent('outer.gui.item.added', copy);
// Without this, every other client keeps a ghost row for the
// replaced entry until the directory is re-listed.
if (overwrittenEntry) {
await this.#emitGuiEvent(
'outer.gui.item.removed',
overwrittenEntry,
);
}
// Legacy response shape: `[{copied: fsentry, overwritten?}]`.
// Array is historical — originally supported bulk copy.
const copied = await toLegacyEntry(this.clients.event, copy, {
const legacyEntryOpts = {
fsEntryStore: this.stores.fsEntry,
userStore: this.stores.user as unknown as {
getById: (
id: number,
) => Promise<Record<string, unknown> | null>;
},
});
res.json([{ copied }]);
};
const copied = await toLegacyEntry(
this.clients.event,
copy,
legacyEntryOpts,
);
const overwritten = overwrittenEntry
? await toLegacyEntry(
this.clients.event,
overwrittenEntry,
legacyEntryOpts,
)
: undefined;
res.json([{ copied, ...(overwritten ? { overwritten } : {}) }]);
};
move = async (req: Request, res: Response): Promise<void> => {
@@ -641,36 +798,77 @@ export class LegacyFSController extends PuterController {
'write',
);
// The v1 wire contract reports the entry an overwrite replaced
// (`overwritten`) so clients can drop its stale row/icon. The move
// deletes that entry, so resolve it beforehand.
const overwriteRequested = getBoolean(body, 'overwrite') ?? false;
let overwrittenEntry = null;
if (overwriteRequested) {
const targetName = getString(body, 'new_name') ?? source.name;
const targetPath =
destinationParent.path === '/'
? `/${targetName}`
: `${destinationParent.path}/${targetName}`;
const existing =
await this.stores.fsEntry.getEntryByPath(targetPath);
// Moving an entry onto its own path is not an overwrite.
if (existing && existing.uuid !== source.uuid) {
overwrittenEntry = existing;
}
}
const moved = await this.services.fs.move(userId, {
source,
destinationParent,
newName: getString(body, 'new_name'),
overwrite: getBoolean(body, 'overwrite') ?? false,
overwrite: overwriteRequested,
dedupeName: getBoolean(body, 'dedupe_name', 'change_name') ?? false,
// Trash/restore rides on this: GUI sends
// `{ original_name, original_path, trashed_ts }` when moving into
// Trash, and `null`/`{}` when restoring. See
// `src/gui/src/helpers.js` → `window.move_items`.
newMetadata: (body.new_metadata ?? undefined) as
| Record<string, unknown>
| null
| undefined,
Record<string, unknown> | null | undefined,
});
const oldPath = source.path;
await this.#emitGuiEvent('outer.gui.item.moved', moved, {
old_path: oldPath,
});
// Without this, every other client keeps a ghost row for the
// replaced entry until the directory is re-listed.
if (overwrittenEntry) {
await this.#emitGuiEvent(
'outer.gui.item.removed',
overwrittenEntry,
);
}
// Legacy response shape: `{moved: fsentry, old_path}`.
const movedEntry = await toLegacyEntry(this.clients.event, moved, {
// Legacy response shape: `{moved: fsentry, old_path, overwritten?}`.
const legacyEntryOpts = {
fsEntryStore: this.stores.fsEntry,
userStore: this.stores.user as unknown as {
getById: (
id: number,
) => Promise<Record<string, unknown> | null>;
},
};
const movedEntry = await toLegacyEntry(
this.clients.event,
moved,
legacyEntryOpts,
);
const overwritten = overwrittenEntry
? await toLegacyEntry(
this.clients.event,
overwrittenEntry,
legacyEntryOpts,
)
: undefined;
res.json({
moved: movedEntry,
old_path: oldPath,
...(overwritten ? { overwritten } : {}),
});
res.json({ moved: movedEntry, old_path: oldPath });
};
delete = async (req: Request, res: Response): Promise<void> => {
@@ -834,7 +1032,7 @@ export class LegacyFSController extends PuterController {
const actor = this.#requireActor(req);
// Subdomain enumeration is a user-level concern; app actors would
// otherwise see root_dir uids pointing outside their AppData scope.
if (effectiveActorApp(actor)) {
if (actor.effectiveApp) {
res.json([]);
return;
}
@@ -863,6 +1061,14 @@ export class LegacyFSController extends PuterController {
throw new HttpError(400, 'Missing `thumbnail`', {
legacyCode: 'bad_request',
});
// Only inline image data. Clients generate the thumbnail themselves
// and the thumbnails extension is what turns it into a storage
// pointer; accepting a pointer here would let a caller name an object
// the server would then sign reads of, and delete, on their behalf.
if (!thumbnail.startsWith('data:'))
throw new HttpError(400, '`thumbnail` must be a data: URL', {
legacyCode: 'bad_request',
});
const entry = await this.stores.fsEntry.getEntryByUuid(uid);
if (!entry || !entry.path)
@@ -903,7 +1109,7 @@ export class LegacyFSController extends PuterController {
// App-under-user actors only see entries within their AppData root;
// user actors are unscoped. Mirrors the ACL short-circuit in
// ACLService.check.
const app = effectiveActorApp(actor);
const app = actor.effectiveApp;
const username = actor.user?.username;
const pathScope =
app && typeof username === 'string' && username.length > 0
@@ -988,32 +1194,6 @@ export class LegacyFSController extends PuterController {
);
res.status(range ? 206 : 200);
// Best-effort egress metering.
const metering = this.services.metering as
| {
batchIncrementUsages?: (
actor: unknown,
entries: unknown[],
) => void;
}
| undefined;
if (metering?.batchIncrementUsages && download.contentLength) {
download.body.once('end', () => {
try {
const bytes = download.contentLength!;
metering.batchIncrementUsages!(actor, [
{
usageType: 'filesystem:egress:bytes',
usageAmount: bytes,
costOverride:
FS_COSTS['filesystem:egress:bytes'] * bytes,
},
]);
} catch {
// ignore — non-critical.
}
});
}
download.body.on('error', (err) => {
res.destroy(err);
});
@@ -1043,9 +1223,20 @@ export class LegacyFSController extends PuterController {
assertNotSuspended(actor!.user);
assertVerifiedAccount(actor!.user);
req.actor = actor!;
req.actor = assertResolvedActor(actor!);
Context.set('actor', actor);
// And the budget gate the other read routes declare with
// `requireCredits`. The global auth probe only looks for `auth_token`,
// so `?token=` leaves `req.actor` unset for the whole gate chain and
// the declarative form would wave every request through — this streams
// file content like `/read` does, so it is refused on the same terms.
await assertActorHasCredits(
this.services.metering,
req.actor,
this.config,
);
// Forward back to regular read after setting actor
return this.read(req, res, undefined, { realMime: true });
};
@@ -1083,8 +1274,7 @@ export class LegacyFSController extends PuterController {
}
type SignedOrEmpty =
| (SignedFile & { path?: string })
| Record<string, never>;
(SignedFile & { path?: string }) | Record<string, never>;
const result: { signatures: SignedOrEmpty[]; token?: string } = {
signatures: [],
};
@@ -1433,6 +1623,23 @@ export class LegacyFSController extends PuterController {
});
}
// Name who this response's bytes are billed to. A signature authorises
// access to a file; it says nothing about who is asking, so an
// unidentified caller is billed to the account whose file it is —
// otherwise a signed URL is a way to serve content for free. A caller
// who did identify themselves pays for what they fetch, as on a hosted
// site.
if (owner?.uuid) {
req.egressActor = req.actor ?? {
user: {
uuid: owner.uuid,
id: owner.id,
username: owner.username,
suspended: !!(owner as { suspended?: unknown }).suspended,
},
};
}
// Directory: return a signed listing of direct children.
// The caller only proved read access, so strip write_url from
// each child to prevent privilege escalation via /writeFile.
@@ -1500,16 +1707,25 @@ export class LegacyFSController extends PuterController {
};
/**
* POST /open_item — resolve an entry, grant the default suggested app write
* POST /open_item — resolve an entry, grant the default suggested app
* access to it, and return a signed URL + user-app token so the launched
* app can read/write the file via its app-under-user token.
*
* Matches v1 semantics: permission is always granted as `write` — the
* underlying user's permission check still caps the effective access
* (grantUserAppPermission doesn't escalate user privileges).
* This is a user-authority action: it writes a user→app ACL row. Two things
* keep an app from opening an item to widen its own access — the caller
* must be the user themselves (the route gate, re-checked here because
* extensions can reach handlers directly), and the grant is capped at the
* access the caller actually proved on the entry.
*/
openItem = async (req: Request, res: Response): Promise<void> => {
const actor = this.#requireActor(req);
if ((actor as { app?: unknown }).app) {
throw new HttpError(
403,
'This endpoint is only available to user sessions',
{ legacyCode: 'forbidden' },
);
}
const body = asRecord(req.body);
const entry = await resolveV1Selector(this.stores.fsEntry, body);
@@ -1521,8 +1737,8 @@ export class LegacyFSController extends PuterController {
'read',
);
// Downgrade the envelope when the caller only proved read.
// `/writeFile`'s ACL re-check would still block the write, but
// Downgrade the envelope and the grant when the caller only proved
// read. `/writeFile`'s ACL re-check would still block the write, but
// returning `write_url` to a read-only caller is the same
// privilege-leak shape that `/sign` and `/readdir` strip.
const writeOk = await this.services.acl.check(
@@ -1550,7 +1766,7 @@ export class LegacyFSController extends PuterController {
await this.services.permission.grantUserAppPermission(
actor,
defaultAppUid,
`fs:${entry.uuid}:write`,
`fs:${entry.uuid}:${writeOk ? 'write' : 'read'}`,
{},
{ reason: 'open_item' },
);
@@ -1624,10 +1840,7 @@ export class LegacyFSController extends PuterController {
const subjectRef = body.subject;
const appRef = body.app;
const mode = (getString(body, 'mode') ?? 'read') as
| 'see'
| 'list'
| 'read'
| 'write';
'see' | 'list' | 'read' | 'write';
if (!subjectRef || !appRef)
throw new HttpError(400, '`subject` and `app` are required', {
legacyCode: 'bad_request',
@@ -1649,10 +1862,10 @@ export class LegacyFSController extends PuterController {
});
// Build an actor-under-user shape for the check.
const actorForApp = {
user: (req.actor as { user?: unknown }).user,
const actorForApp = makeActor({
user: req.actor!.user,
app: { uid: (app as { uid: string }).uid },
} as unknown as Actor;
});
const descriptor = {
path: subject.path,
resolveAncestors: () =>
-30
View File
@@ -1,30 +0,0 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { toMicroCents } from '../../services/metering/utils.js';
// Microcents per byte. Egress roughly matches S3 data-transfer-out
// (~$0.12/GiB); cached egress is CloudFront-backed (~$0.10/GiB).
// Ingress and deletes are currently free.
export const FS_COSTS = {
'filesystem:ingress:bytes': 0,
'filesystem:delete:bytes': 0,
'filesystem:egress:bytes': toMicroCents(0.12 / 1024 / 1024 / 1024),
'filesystem:cached-egress:bytes': toMicroCents(0.1 / 1024 / 1024 / 1024),
} as const;
@@ -1,4 +1,4 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
@@ -18,13 +18,24 @@
*/
import { describe, expect, it } from 'vitest';
import { loadLegacyAssociatedApps } from './legacyFsHelpers.js';
import {
asRecord,
fsEntryMimeType,
getBoolean,
getString,
loadLegacyAssociatedApps,
signEntryThumbnail,
signingConfigFromAppConfig,
toLegacyEntry,
} from './legacyFsHelpers.js';
import type { FSEntry } from '../../stores/fs/FSEntry.js';
const entryWithApp = (associatedAppId: number): FSEntry =>
({ associatedAppId }) as unknown as FSEntry;
const appRow = (overrides: Record<string, unknown>): Record<string, unknown> => ({
const appRow = (
overrides: Record<string, unknown>,
): Record<string, unknown> => ({
id: 1,
uid: 'app-1',
owner_user_id: 99,
@@ -100,3 +111,336 @@ describe('loadLegacyAssociatedApps associated_app redaction', () => {
expect(app.godmode).toBe(0);
});
});
describe('asRecord', () => {
it.each([
['null', null],
['undefined', undefined],
['a number', 42],
['a string', 'nope'],
['an array', [1, 2]],
])('returns an empty record for %s', (_label, value) => {
expect(asRecord(value)).toEqual({});
});
it('passes an object through unchanged', () => {
const source = { a: 1 };
expect(asRecord(source)).toBe(source);
});
});
describe('getString', () => {
it('returns the first non-empty string among the candidate keys', () => {
expect(getString({ a: '', b: 'second' }, 'a', 'b')).toBe('second');
});
it('ignores non-string values', () => {
expect(getString({ a: 5, b: true }, 'a', 'b')).toBeUndefined();
});
it('returns undefined when no key matches', () => {
expect(getString({}, 'missing')).toBeUndefined();
});
});
describe('getBoolean', () => {
it.each([
[true, true],
[false, false],
[1, true],
[0, false],
['1', true],
['TRUE', true],
[' yes ', true],
['on', true],
['0', false],
['false', false],
['no', false],
['off', false],
])('coerces %o to %s', (input, expected) => {
expect(getBoolean({ flag: input }, 'flag')).toBe(expected);
});
it.each([[2], ['maybe'], [null], [{}]])(
'returns undefined for the uncoercible value %o',
(input) => {
expect(getBoolean({ flag: input }, 'flag')).toBeUndefined();
},
);
it('falls through to a later alias when the first key is absent', () => {
expect(
getBoolean({ change_name: true }, 'dedupe_name', 'change_name'),
).toBe(true);
});
});
describe('fsEntryMimeType', () => {
it('reports directories as "folder"', () => {
expect(fsEntryMimeType({ isDir: true, name: 'Documents' })).toBe(
'folder',
);
});
it('derives a MIME type with charset from the file name', () => {
expect(fsEntryMimeType({ isDir: false, name: 'a.png' })).toBe(
'image/png',
);
expect(fsEntryMimeType({ isDir: false, name: 'a.txt' })).toContain(
'text/plain',
);
});
it('returns null for an extensionless name', () => {
expect(fsEntryMimeType({ isDir: false, name: 'LICENSE' })).toBeNull();
});
});
describe('signEntryThumbnail', () => {
it('returns the input untouched when there is no event client', async () => {
expect(
await signEntryThumbnail(undefined, 'uuid-1', 's3://bucket/key'),
).toBe('s3://bucket/key');
});
it('normalizes a missing thumbnail to null', async () => {
expect(await signEntryThumbnail(undefined, 'uuid-1', null)).toBeNull();
});
it('returns the URL the listener rewrote onto the payload', async () => {
const eventClient = {
emitAndWait: async (
_key: string,
payload: { thumbnail: string },
) => {
payload.thumbnail = 'https://signed.test/thumb.png';
},
} as never;
expect(
await signEntryThumbnail(eventClient, 'uuid-1', 's3://bucket/key'),
).toBe('https://signed.test/thumb.png');
});
it('returns null when the listener blanks the thumbnail', async () => {
const eventClient = {
emitAndWait: async (
_key: string,
payload: { thumbnail: string },
) => {
payload.thumbnail = '';
},
} as never;
expect(
await signEntryThumbnail(eventClient, 'uuid-1', 's3://bucket/key'),
).toBeNull();
});
it('keeps the original value when the listener throws', async () => {
const eventClient = {
emitAndWait: async () => {
throw new Error('extension down');
},
} as never;
expect(
await signEntryThumbnail(eventClient, 'uuid-1', 's3://bucket/key'),
).toBe('s3://bucket/key');
});
});
describe('signingConfigFromAppConfig', () => {
it('returns the secret and api base url when both are set', () => {
expect(
signingConfigFromAppConfig({
url_signature_secret: 's3cret',
api_base_url: 'https://api.test',
} as never),
).toEqual({ secret: 's3cret', apiBaseUrl: 'https://api.test' });
});
it('fails loudly when the signing secret is missing', () => {
expect(() =>
signingConfigFromAppConfig({
api_base_url: 'https://api.test',
} as never),
).toThrowError(/url_signature_secret not set/);
});
it('fails loudly when the api base url is missing', () => {
expect(() =>
signingConfigFromAppConfig({
url_signature_secret: 's3cret',
} as never),
).toThrowError(/api_base_url not set/);
});
});
describe('toLegacyEntry', () => {
const baseEntry = (overrides: Partial<FSEntry> = {}): FSEntry =>
({
uuid: 'uuid-1',
parentUid: 'parent-1',
path: '/alice/Documents/report.pdf',
name: 'report.pdf',
isDir: false,
isShortcut: false,
shortcutTo: null,
isSymlink: false,
symlinkPath: null,
isPublic: false,
thumbnail: null,
immutable: false,
metadata: null,
modified: 1000,
created: 900,
accessed: 950,
size: 12,
layout: null,
subdomains: [],
workers: [],
suggestedApps: [],
associatedAppId: null,
userId: 7,
...overrides,
}) as unknown as FSEntry;
it('produces the snake_case v1 shape for a file', async () => {
const shaped = await toLegacyEntry(undefined, baseEntry());
expect(shaped).toMatchObject({
id: 'uuid-1',
uid: 'uuid-1',
uuid: 'uuid-1',
parent_id: 'parent-1',
parent_uid: 'parent-1',
dirname: '/alice/Documents',
dirpath: '/alice/Documents',
is_dir: false,
is_shortcut: 0,
is_symlink: 0,
has_website: false,
is_empty: false,
associated_app: null,
appdata_app: undefined,
});
expect(shaped.type).toContain('application/pdf');
});
it('names the owning app for an AppData path', async () => {
const shaped = await toLegacyEntry(
undefined,
baseEntry({ path: '/alice/AppData/app-42/state.json' }),
);
expect(shaped.appdata_app).toBe('app-42');
});
it('reports has_website when the entry carries a subdomain', async () => {
const shaped = await toLegacyEntry(
undefined,
baseEntry({
subdomains: [{ subdomain: 'site' }] as never,
}),
);
expect(shaped.has_website).toBe(true);
});
it('probes for children to set is_empty on a directory', async () => {
const fsEntryStore = {
listChildren: async () => [{ uuid: 'child' }],
} as never;
const shaped = await toLegacyEntry(
undefined,
baseEntry({ isDir: true, name: 'Documents' }),
{ fsEntryStore },
);
expect(shaped.is_empty).toBe(false);
expect(shaped.type).toBe('folder');
});
it('treats a failed child probe as non-empty rather than throwing', async () => {
const fsEntryStore = {
listChildren: async () => {
throw new Error('db down');
},
} as never;
const shaped = await toLegacyEntry(
undefined,
baseEntry({ isDir: true }),
{ fsEntryStore },
);
expect(shaped.is_empty).toBe(false);
});
it('hydrates the owner as a username-only object', async () => {
const userStore = {
getById: async () => ({ username: 'alice', email: '[email protected]' }),
};
const shaped = await toLegacyEntry(undefined, baseEntry(), {
userStore,
});
// Username only — the rest of the user row must not ride along.
expect(shaped.owner).toEqual({ username: 'alice' });
});
it('omits the owner when the lookup fails', async () => {
const userStore = {
getById: async () => {
throw new Error('db down');
},
};
const shaped = await toLegacyEntry(undefined, baseEntry(), {
userStore,
});
expect(shaped).not.toHaveProperty('owner');
});
it('embeds associated_app from the prebuilt map', async () => {
const appsById = new Map([[3, { uid: 'app-3', name: 'Editor' }]]);
const shaped = await toLegacyEntry(
undefined,
baseEntry({ associatedAppId: 3 }),
{ appsById },
);
expect(shaped.associated_app).toEqual({ uid: 'app-3', name: 'Editor' });
});
it('emits a null associated_app when the id is not in the map', async () => {
const shaped = await toLegacyEntry(
undefined,
baseEntry({ associatedAppId: 9 }),
{ appsById: new Map() },
);
expect(shaped.associated_app).toBeNull();
});
});
describe('loadLegacyAssociatedApps short-circuit', () => {
it('makes no store call when no entry carries an app id', async () => {
let calls = 0;
const store = {
getByIds: async () => {
calls += 1;
return new Map();
},
};
const out = await loadLegacyAssociatedApps(store, [
{ associatedAppId: null } as unknown as FSEntry,
]);
expect(out.size).toBe(0);
expect(calls).toBe(0);
});
it('dedupes repeated app ids into a single lookup', async () => {
const seen: number[][] = [];
const store = {
getByIds: async (ids: number[]) => {
seen.push(ids);
return new Map(ids.map((id) => [id, appRow({ id })]));
},
};
await loadLegacyAssociatedApps(store, [
entryWithApp(4),
entryWithApp(4),
entryWithApp(5),
]);
expect(seen).toEqual([[4, 5]]);
});
});
+10 -11
View File
@@ -1,21 +1,20 @@
/**
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify it under the
* terms of the GNU Affero General Public License as published by the Free
* Software Foundation, either version 3 of the License, or (at your option) any
* later version.
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
* details.
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { posix as pathPosix } from 'node:path';
+163
View File
@@ -0,0 +1,163 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { describe, expect, it } from 'vitest';
import * as limits from './limits.js';
import {
DEFAULT_FREE_SUBSCRIPTION,
DEFAULT_TEMP_SUBSCRIPTION,
} from '../../services/metering/consts.js';
import type { RouteOptions, RouteRateLimit } from '../../core/http/types';
type Spec = RouteRateLimit | NonNullable<RouteOptions['concurrent']>;
const all = Object.entries(limits) as Array<[string, Spec | RouteRateLimit[]]>;
const flat: Array<[string, Spec]> = all.flatMap(([name, value]) =>
Array.isArray(value)
? value.map((v, i): [string, Spec] => [`${name}[${i}]`, v])
: [[name, value] as [string, Spec]],
);
const windows = flat.filter(([, s]) => 'window' in s) as Array<
[string, RouteRateLimit]
>;
const concurrents = flat.filter(([, s]) => !('window' in s));
describe('filesystem limit specs', () => {
// The whole point of this module is that the legacy and v2 controllers
// import the same specs. That only ties the counters together if every
// spec carries an explicit scope — without one the gate falls back to
// the route path, and the two controllers get separate budgets.
it.each(flat)('%s pins an explicit scope', (_name, spec) => {
expect(spec.scope).toBeTruthy();
});
it.each(windows)('%s has a positive limit and window', (_name, spec) => {
expect(spec.limit).toBeGreaterThan(0);
expect(spec.window).toBeGreaterThan(0);
});
// Base is the paid value; the free tiers are carved out beneath it.
// A free tier above the base would mean paying made you worse off.
it.each(windows)('%s never lets a free tier exceed paid', (_name, spec) => {
for (const n of Object.values(spec.bySubscription ?? {})) {
expect(n).toBeLessThanOrEqual(spec.limit);
}
});
it.each(windows)(
'%s caps temp at or below registered-free',
(_name, spec) => {
const free = spec.bySubscription?.[DEFAULT_FREE_SUBSCRIPTION];
const temp = spec.bySubscription?.[DEFAULT_TEMP_SUBSCRIPTION];
if (free === undefined || temp === undefined) return;
expect(temp).toBeLessThanOrEqual(free);
},
);
// A single in-flight slot turns incidental client parallelism into a
// spurious 429; paid tiers keep room to actually parallelise.
it.each(concurrents)(
'%s keeps concurrency at 5+ paid and 2+ for every tier',
(_name, spec) => {
expect(spec.limit).toBeGreaterThanOrEqual(5);
for (const n of Object.values(spec.bySubscription ?? {})) {
expect(n).toBeGreaterThanOrEqual(2);
}
},
);
it('gives search the tightest window of the read paths', () => {
expect(limits.FS_SEARCH_LIMIT.limit).toBeLessThan(
limits.FS_STAT_LIMIT.limit,
);
expect(limits.FS_SEARCH_LIMIT.limit).toBeLessThan(
limits.FS_READ_LIMIT.limit,
);
});
// The desktop deletes/moves one item per request with no batching and no
// pacing, so a single "empty trash" or "select all, delete" has to fit
// inside the minute window on every tier. The hour window is the abuse
// ceiling that a minute window this wide can no longer be.
it('gives mutations a bulk-sized minute window plus an hourly backstop', () => {
const [minute, hourly] = limits.FS_MUTATE_LIMIT;
expect(limits.FS_MUTATE_LIMIT).toHaveLength(2);
expect(minute.window).toBe(60_000);
expect(hourly.window).toBe(60 * 60_000);
const tiers = (spec: RouteRateLimit) => [
spec.limit,
spec.bySubscription![DEFAULT_FREE_SUBSCRIPTION],
spec.bySubscription![DEFAULT_TEMP_SUBSCRIPTION],
];
// A few hundred items clears in one pass, anonymous included.
for (const n of tiers(minute)) expect(n).toBeGreaterThanOrEqual(500);
// Anonymous stays meaningfully tighter than paid on both windows.
expect(tiers(minute)[2]).toBeLessThanOrEqual(minute.limit / 2);
expect(tiers(hourly)[2]).toBeLessThanOrEqual(hourly.limit / 2);
// The hour window has to bind rather than decorate: it allows a
// handful of bulk passes an hour, not sixty minutes' worth.
for (const [perMinute, perHour] of tiers(minute).map(
(n, i): [number, number] => [n, tiers(hourly)[i]],
)) {
expect(perHour).toBeGreaterThan(perMinute);
expect(perHour).toBeLessThanOrEqual(perMinute * 10);
}
});
// The DAV gate is consumed imperatively, before the request has an actor:
// it keys on the network fingerprint and reads only `limit` / `window`. A
// `key: 'user'` or a `bySubscription` map here would describe tiering that
// never happens.
it('shapes the DAV specs the way the DAV gate consumes them', () => {
expect(limits.DAV_LIMIT.key).toBe('fingerprint');
expect(limits.DAV_CONCURRENT.key).toBe('fingerprint');
expect(limits.DAV_LIMIT.bySubscription).toBeUndefined();
expect(limits.DAV_CONCURRENT.bySubscription).toBeUndefined();
});
// With no session to key on, the alternative is the bare address — and an
// address is a household, an office or a carrier gateway, so keying there
// makes one bucket serve everyone behind it and tighten as more real users
// arrive. The fingerprint separates clients within a network while still
// being something one client can't vary per request.
it('keys the signed-URL routes on the network fingerprint', () => {
expect(limits.FS_SIGNED_READ_LIMIT.key).toBe('fingerprint');
expect(limits.FS_SIGNED_WRITE_LIMIT.key).toBe('fingerprint');
expect(limits.FS_SIGNED_CONCURRENT.key).toBe('fingerprint');
});
// These serve page subresources — a gallery, an app's own assets — so the
// in-flight cap has to sit above what a browser opens to one origin at
// once, or a normal page load is what trips it.
it('leaves the signed-URL in-flight cap above a browser`s own parallelism', () => {
expect(limits.FS_SIGNED_CONCURRENT.limit).toBeGreaterThan(30);
});
it('uses distinct scopes so counters cannot collide', () => {
const scopes = flat
.filter(([, s]) => 'window' in s)
.map(([, s]) => s.scope);
expect(new Set(scopes).size).toBe(scopes.length);
});
});
Loaded 100 of 877 files, more files were not shown because too many files have changed in this diff. Show more