Compare commits

...
47 Commits
Author SHA1 Message Date
Daniel Salazar 908ec23eda fix: sanitize user data (#3552) 2026-08-12 12:08:54 -07:00
Juan Fernando Castro d5ae5a0049 🔧 PUR-1072: Flatten driver permissions to hardcoded values (#3545)
* refactor(permissions): drop hardcoded group permission map for a flat default

* test(drivers): assert credential-gate intent instead of a 403 proxy
2026-08-12 12:08:38 -07:00
Daniel Salazar 419d0aaa89 doc: update bounty doc 2 (#3551) 2026-08-12 09:56:34 -07:00
Daniel Salazar c3a0c28705 docs: update bug bountu (#3550) 2026-08-12 09:55:13 -07:00
Nariman Jelveh 7ceb2090b7 feat: app user feedback system (#3546)
* feat: app user feedback system

Add puter.ui.showFeedbackDialog(), letting users send feedback to an
app's developer. In the app environment the Puter desktop renders the
dialog; on a third-party website a puter.com popup hosts it. The message
is stored in a new app_feedback table and emailed to the app owner's
confirmed email — it never passes through the app's own code.

Feedback is strictly opt-in per app via a new apps.feedback_enabled
column (a real column, not an app-metadata key, so Dev Center's
whole-blob metadata saves can't silently erase it), settable through the
existing puter.apps.update path (feedbackEnabled).

Backend follows the layered stack: AppFeedbackStore (durable count
queries) -> AppFeedbackService (opt-in check, message normalization,
abuse caps, best-effort owner email) -> AppFeedbackController
(POST /app-feedback, GET /app-feedback/target). New app-user-feedback
email template uses the escaping-safe nl2br triple-stash.

Defensive by design:
- requireUserActor blocks app tokens, so feedback can't be submitted
  programmatically; guiOriginOnly keeps cross-origin pages out.
- App identity comes only from the validated IPC sender (desktop) or the
  browser-attested opener origin (popup), never from message contents.
- The send-feedback popup action is in NON_AUTH_POPUP_ACTIONS, so it
  never delivers a token to the opener.
- Layered limits: route rate limits, plus DB-count caps that fail closed
  when the limiter backend is down, plus a per-app daily owner-email cap.
- Owner email is fully best-effort: an unconfigured transport,
  unconfirmed/unsubscribed/suspended owner, or send failure never fails
  the request or blocks storage.
- The dialog and SDK method are resolve-only and always settle, so a
  caller is never left hanging.

Migrations for sqlite/mysql/postgres, puter.js types, docs, backend
tests (sqlite + postgres), and a Playwright e2e spec are included.

* feat: add feedback control to the dashboard app-drawer

Surface the feedback dialog directly from the app window's chrome in
dashboard mode: apps that opt in (apps.feedbackEnabled) get a "Send
Feedback" button in the dashboard app-drawer, next to minimize/close.
It opens the same UIWindowAppFeedback dialog, targeting this app by uid.

The control is only rendered when the app opted in — feedback_enabled is
threaded from the launched app's metadata into the window options — and
the dialog still re-checks opt-in server-side, so a stale flag can't send
anywhere. Reuses the existing .dashboard-app-drawer-btn styling and the
app_feedback_title i18n string, so no new CSS or strings.

Adds e2e coverage: the control appears and opens the dialog for an
opted-in app, and is absent for an app that hasn't opted in.

* feat: enable app feedback by default in Dev Center

New apps created in Dev Center now have feedbackEnabled set on creation,
so users can send the developer feedback without any extra setup. A "User
Feedback" toggle in the app's settings lets developers turn it off (and
back on); it's wired into the save payload, the dirty-state tracking, and
the reset-to-original path like the neighboring toggles.

The Save update omits feedbackEnabled unless the toggle is present, and
the backend leaves an omitted field untouched, so the default survives
the create-then-save flow Dev Center runs. Add an SDK apps-suite guard
covering that round-trip (create-on -> unrelated update keeps it -> can
be turned off).

* fix: feedback modal polish + share sender email

Address four issues with the app feedback UI:

- Dashboard app-drawer: the extra "feedback" control pushed the close
  button past the drawer's derived width and clipped it. A `has-feedback`
  modifier widens the surface by one button + gap so all three controls
  fit. The control's glyph is now a message bubble with text lines, which
  reads more clearly at 14px than the previous bare speech bubble.

- The feedback dialog is no longer a UIWindow. It's a from-scratch
  overlay modal in the spirit of the dashboard modals (uninstall,
  add-app): a fixed scrim + centered card with self-contained,
  theme-aware color tokens (light default + dark override), a bottom-sheet
  layout on narrow screens, backdrop/Escape close, and an entrance
  transition. This renders consistently across the three contexts it's
  opened from (desktop app-IPC, dashboard drawer, standalone popup), so
  the callers no longer pass UIWindow-specific window_options.

- Feedback now shares the sender's email (not just their username) with
  the developer so they can respond: the owner email sets Reply-To to the
  sender and shows the address in the body — but only when the sender's
  email is verified (an unverified address could be anyone's, so it's
  never used as a reply target). EmailClient.send gains an optional
  replyTo. The dialog note now says the email will be shared.

Tests: e2e updated for the new modal (7 pass); backend feedback suite
covers the verified/unverified sender-email split (sqlite + postgres);
EmailClient + GUI unit suites pass; type-check clean.

* fix: resolve 'app-'-prefixed app names in feedback target lookup

APP_NAME_REGEX allows names beginning with "app-" (e.g. the seeded
app-center), but resolveTargetApp's startsWith('app-') heuristic sent
those to a uid-only lookup with no name fallback, so feedback for such
apps 403'd even when enabled. Use AppStore.resolveApp (uid, then name)
like the rest of the codebase.

* fix: make feedback daily caps fail closed under concurrent submissions

The per-user and per-user-per-app caps were check-then-insert and the
per-app email cap was count-then-send, so parallel requests (or multiple
nodes, or the route limiter failing open) could all read a stale
under-cap count and push past every limit — the exact scenario the
DB-backed caps exist to stop.

Now the user caps recount after the insert (own row included) and roll
the row back with 429 if a burst breached them, and the email cap claims
its slot (email_sent=1) before sending, recounts, and releases the slot
if over cap or if the send fails.

* docs: disclose the Dev Center's feedback-on-by-default in SDK docs/types

The Dev Center deliberately creates apps with feedbackEnabled (see
32c950de5), but the SDK docs, apps.d.ts, and AppFeedbackService's class
doc all described feedback as strictly opt-in / default-false with no
qualification — so a Dev Center developer reading them would wrongly
conclude feedback is off for their app. State the Dev Center behavior
alongside the API default, and correct the update-path docs: an omitted
feedbackEnabled leaves the current value unchanged rather than
defaulting to false.

* fix: don't mint a user-app token as a side effect of the feedback popup

Every embedded_in_popup boot ran the user-app token exchange, and the
exchange is a write: /auth/get-user-app-token bootstraps an app row for
the opener origin, grants flag:app-is-authenticated (what makes the
site count as connected to the account), and creates its AppData dir.
So merely opening — or immediately cancelling — a send-feedback popup
recorded a user<->site relationship the read-only feedback flow never
needs: the server resolves the feedback target from the attested origin
without any of it.

Gate the exchange behind runsUserAppTokenExchange(action) in all three
popup paths that mint (main postAuthActions exchange, temp-user signup
success, manual signup fallback). request-permission keeps the exchange
since grants are written against the app row it bootstraps.

* fix: refuse feedback when the deployment cannot deliver it

With no email transport configured (the common self-hosted default),
submissions were stored in app_feedback — a table with no read path
beyond the abuse-cap COUNTs — the owner email was silently skipped, and
the sender was still shown 'Feedback sent. Thank you!'. The developer
never learns the feedback exists while the user believes it was
delivered.

Gate acceptsFeedback on clients.email.isConfigured so the pre-flight
reports enabled:false (the dialog shows its 'not accepting feedback'
pane) and submit returns 403 instead of swallowing messages. Owner-level
store-without-email cases (unconfirmed owner email, per-app email cap
overflow) keep their existing deliberate semantics.

* fix: settle showFeedbackDialog instead of hanging on older host GUIs

In the app environment showFeedbackDialog awaited an IPC reply with no
capability check. A host GUI that predates this feature (self-hosted
Puter running the live js.puter.com SDK) has no handler for the message
and never replies, so the promise documented as 'never rejects' also
never resolved.

The GUI now advertises the IPC dialogs it can answer via a
puter.gui_features param on the app iframe URL, and the SDK resolves
false when 'feedback-dialog' isn't listed. A reply timeout could not
substitute: legitimate replies only arrive when the user closes the
dialog, so any timeout would false-negative while the user is typing.
Older SDKs ignore the extra param.

* fix: guard app-triggered feedback dialog against desktop-lockout loops

The showFeedbackDialog IPC handler had no re-entry or abuse guard, and
the dialog it opens is a full-viewport overlay above the taskbar and
every window — so 'while (true) await puter.ui.showFeedbackDialog()'
kept the desktop permanently covered (for signed-out users, the same
loop spams the full-page signup window instead). Every dismissal just
settled the promise and let the app immediately reopen it.

Allow one dialog at a time, and back off reopens per app after each
dismissal that sent nothing: 10s, then 60s, then blocked until page
reload. A successful send resets the backoff, and user-initiated paths
(dashboard drawer) are unaffected since they don't go through IPC.

* fix: carry feedback_enabled in suggested/recommended app summaries

launch_app uses options.app_obj verbatim when provided, and the
suggested-apps launch paths (open_item.js, UIWindowSearch.js,
UIDesktop.js) pass summaries from toAppSummary — which omitted
feedback_enabled. So an opted-in editor launched by opening a .txt file
showed no Send Feedback control in the dashboard drawer, while the same
app launched from the Apps tab (full puter.apps.get object) did.

* fix: gate feedback dialog Cancel/X on an in-flight submit

Escape and backdrop clicks were already ignored while the POST was
pending, but the X, Close, and Cancel buttons weren't — clicking one
mid-send settled the promise false and tore down the overlay while the
submission still landed server-side: the developer got the email, the
app was told sent=false, and a user who resubmitted 'the failed one'
sent a duplicate and burned a daily-cap slot.

Apply the same !sending gate to the buttons and disable them visually
while the send is in flight.

* fix: reject feedback origins longer than the source_origin column

readTargetParam accepted values up to 3000 chars but the raw origin is
stored verbatim into source_origin VARCHAR(2048) (MySQL/Postgres), so a
2049-3000 char origin passed every validation and then blew up the
INSERT with an HTTP 500 on Postgres/strict MySQL — or was silently
truncated on non-strict MySQL, corrupting the abuse-forensics value the
column exists for. Cap the param at the column size.

* fix: make the feedback dialog's privacy note match what is shared

The note unconditionally said 'Your email address will be shared with
the developer so they can respond', but AppFeedbackService shares the
username always and the email only when it exists and is verified — an
unverified or temp-user sender was promised a reply path that never
materializes, and nobody was told about the username.

Show 'username and email' when the signed-in user's email is verified,
and 'username' otherwise.

* fix: stop HTML-escaping email subject lines

Subjects were compiled with default Handlebars escaping, so the
app-user-feedback subject rendered a title like "Bob's App & Games" as
"Bob&#x27;s App &amp; Games" — literal entities in the recipient's mail
client. Subjects are plain-text headers, not HTML; compile them with
noEscape. Header safety is unaffected: the transport encodes newlines
and free-form values collapse whitespace upstream.

* docs: state exactly what showFeedbackDialog shares with the developer

The doc claimed the dialog 'tells the user their username will be
shared', while the dialog's note talked only about the email address and
the implementation shares the username always plus the email (as
Reply-To) only when verified. Describe the actual disclosure: username
always, email when verified.

* docs: document the COOP false-resolve limitation of showFeedbackDialog

Under COOP the popup's opener link is severed, so the SDK deliberately
resolves false while the popup stays open and the user can still submit
(a feedback submission has no server read-back the way a permission
grant does). The documented contract ('resolves to true if the user
submitted feedback') was silently wrong on cross-origin-isolated pages —
state the limitation in the doc and the SDK jsdoc: false means 'not
confirmed', not 'not sent'.

* feat: point the feedback email footer at the Dev Center

The footer told developers to turn feedback off via a puter.apps.update
one-liner, but the toggle lives in the Dev Center app settings — and
Dev Center is where apps get feedback enabled by default in the first
place. Reword it to 'manage it in the Dev Center' with a link built
from config.origin (like app_link) so it holds on self-hosted
deployments.

* test: cover the feedback service and store layers directly

The service owns every feedback business rule — target resolution,
eligibility, message normalization, the durable caps, and the owner-email
preconditions — but was only reachable through the controller's tests. Give
it and the store their own suites so a regression names the layer it broke.

Service coverage adds the branches the route tests could not reach: a
blocked origin resolving to null rather than surfacing a 403, owners who are
suspended or unsubscribed, length measured after normalization, the 24h cap
window boundary, subject-header injection via the app title, and the email
links being rooted at config.origin.

The two describes already labelled `AppFeedbackService ...` move out of the
controller test, which keeps only the caller-facing promise that a failed
send still returns success.

* refactor: prep the feedback cap error once instead of via a factory

Both throw sites are in one call and only one can ever run, so a plain
const reads the same and drops a function that existed only to defer a
constructor.

* Require verified users for app feedback API

Add `requireVerified: true` to both app feedback routes (`GET /target` and `POST /`) in `AppFeedbackController`. This tightens access control so only verified user accounts can fetch feedback targets or submit app feedback.
2026-08-12 09:49:24 -07:00
jelveh ea63c3093e feat(dashboard): add Publish as website and Properties to folder menu
The Files tab's folder-background context menu (empty listing area,
sidebar folders, breadcrumb segments) now offers Publish as website and
Properties for the folder the menu was opened on, matching the desktop's
folder menus. Both are omitted at the filesystem root and in Trash, and
Publish is omitted inside trashed folders.

The properties modal can now resolve an entry by path when the caller
has no uid, which the folder-background menu doesn't.

The publish gate (account creation + email confirmation) was duplicated
in three places, so it moves to a publish_as_website helper that all
call sites now share.
2026-08-12 08:50:03 -07:00
Daniel Salazar e8236a100c chore: bump versions (#3548) 2026-08-12 01:18:17 -07:00
Daniel Salazar 20b3b88e39 metering: big fixes to metering + jsdoc types (#3547)
Changes are:
- global egress metering
- remove file egress cost
- introduce file op cost for the per request cost s3 has
- enforce fs read/download etc to through 402 when out of usage; allow for subdomains
- enforce kv metering when out of usage through 402; allow for workers
- jsdoc as source of truth for puter.js types
- kv driver caching for get and batchget operations with decreased costs
2026-08-12 01:06:04 -07:00
Chen, Ting-An cbae39a08f fix(i18n): add Traditional Chinese app management strings (#3541)
Translate the app discovery, app request, desktop shortcut, and app folder controls that currently fall back to English in the Traditional Chinese interface. Preserve the positional placeholders used by accessibility labels.
2026-08-11 21:18:37 -07:00
Daniel Salazar afad3e37bf fix: server health tolerance for primary read (#3542) 2026-08-11 01:19:09 -07:00
Daniel Salazar a6b2161b2c fix: misc sec fixes (#3540) 2026-08-10 22:24:44 -07:00
Neal Shah 9e864e9574 PUT-1432 (#3539) 2026-08-10 23:33:20 -04:00
Daniel Salazar 79d4201f12 fix: rate limits, AI routing, and a type-check gate (#3529)
- declare rate + concurrency limits on every route and driver that lacked one
- add acquireConcurrent for websocket connections and the DAV mount
- bucket AI models by identity key only; keep resold duplicates of any vendor
- skip recently-failed provider routes; cap the fallback chain at 3 attempts
- let full-access access tokens bind a worker to an app their own user owns
- cache resolved subscriptions so tiered limits don't add a round trip
2026-08-10 19:09:47 -07:00
Daniel Salazar 0c2d7dfa34 feat: add user created timestamp on whoami (#3538) 2026-08-10 19:08:50 -07:00
Neal Shah 2a6d2fea32 allow max-tokenless models on openrouter (#3536) 2026-08-10 18:40:10 -04:00
Juan Fernando Castro 1dec5e90c4 docs: add requestAppData() to Perms sidebar (#3534) 2026-08-10 11:38:09 -07:00
Neal Shah 566298d89c make context length on togetherAI more lenient (#3533) 2026-08-10 13:14:33 -04:00
Daniel Salazar 6bb5eacf61 fix: metering kv spamc (#3532) 2026-08-10 08:36:56 -07:00
Reynaldi Chernando 390a83c860 Fix puter.js docs and types drift 10 aug (#3531) 2026-08-10 12:47:18 +07:00
Nariman Jelveh 8f23bf7b75 fix(gui): hover the add-an-app options with the accent wash, not grey
The rows already answer hover with a --select-ring border; filling them
with the grey --dashboard-hover under that accent ring read as two
systems disagreeing. The weak select tint keeps the hover in one voice
and stays under the icon chip's stronger tint in both themes.
2026-08-09 18:23:07 -07:00
Nariman Jelveh aa85dddd38 Refresh Add App AI description copy
Update the English `add_app_ai_desc` translation string in the GUI to a more direct, enthusiastic message: “Describe the app you want and AI will build it!”.
2026-08-09 18:21:31 -07:00
Nariman Jelveh 8284f35b93 fix(gui): honor godmode file_paths launches now that /apps returns booleans
The godmode gate in ExecService.launchApp checked `godmode === 1`, but
GET /apps/:name serializes the flag as a boolean, so the check always
failed and file_paths launches (e.g. Dev Center opening a worker's
source in the editor, #2218) silently dropped the file. Accept both
shapes, matching the existing guard in launch_app.js.

Also fix the two issues hiding behind that gate:
- puter.apps.get returns `uid`, not `uuid`, so the /sign call for the
  target app sent app_uid undefined — no write grant and no user-app
  token for the launched app. Use `uid` with a `uuid` fallback.
- The closeApp IPC handler had the same `godmode === 1` comparison,
  preventing godmode apps from closing other apps' windows.
2026-08-09 18:06:58 -07:00
Nariman Jelveh ccec3ce39b fix(gui): open add-an-app choices through the deep-link intro, not a + tile morph
60686643b made App Center and the AI builder morph out of the + tile the
user clicked, which answers where the window came from but not the
question these two options exist to answer: where the APP went. The grid
never showed it arriving, so the user still ended a launch with nothing
to find, no tile for minimize to fly back to, and no lesson learned.

Choosing either option now plays exactly what an /app/<name> landing
plays (beginDeepLinkLaunch): when the app already has a tile the intro
travels to it, plays the click flourish, and the window grows out of the
app's own slot; when it doesn't, the tile is INSTALLED first — the slot
opens, the progress stroke draws, the icon springs in — and only then
does the window morph out of the tile that just arrived. Same beats,
same exposure decay, same interruptibility, same duplicate-launch claim.

The app info is prefetched in parallel and handed to launch_app, the
same contract as initgui's landing path, so the intro (which may need to
draw the arriving tile from it) never costs a second round-trip.

This also reverts 60686643b's anchor plumbing (UIWindow's
dashboard_tile_el option and the dashboard_tile_in_view export): the
intro installs a real tile for the by-app-name morph to find, so no
stand-in anchor is needed.
2026-08-09 18:02:39 -07:00
Nariman Jelveh 60686643b3 fix(gui): launch from the add-an-app modal with the tile morph, not a fade
Choosing App Center or the AI builder opened the window with the plain
fade while every other launch in the Apps tab grows out of the icon that
was clicked — the one launch that most needs to say where it came from
read as the app appearing from nowhere.

The morph resolved its anchor by app name, which cannot work here: the
whole point of these two options is installing an app you do not have,
and an app with no tile has no name to find. So the launcher may name its
anchor element (dashboard_tile_el), and TabApps resolves it once before
launching so the click flourish and the window's half can never disagree
about which tile the launch came from: the app's own tile when it already
has one on screen — minimize flies the window back there, so that is
where opening should come from — and otherwise the + tile itself.

dashboard_tile_in_view's visibility test is now a named helper it shares
with the new path, so an anchor on an offscreen pager page is still
rejected and still falls through to the fade.
2026-08-09 17:49:50 -07:00
Nariman Jelveh 61bfcbdc83 feat(gui): add an add-an-app tile at the end of the Apps grid
The Apps grid showed what you have and offered no way to get more. A
dashed plus tile now rides at the tail of every page of it: after every
app, on the last page, and nowhere else. It wears the same tile skeleton
as an app so hover, focus, and arrow keys treat it identically, but
carries no app name or group id — which is what keeps it out of the
saved order, the running dots, and the deep-link tile lookups. It cannot
be picked up, is never a drop target (so no app can land past it), has
no context menu, and sits still while the apps jiggle in reorder mode.
Search results leave it out: a query asks which apps you have.

Clicking it asks how you would like to get an app: find one in the App
Center, have one built by the AI builder, or request one that does not
exist yet. The request is a form in the same card rather than the stock
Contact Us window — a titled pane one step deeper, whose Back arrow
returns to the options with the draft intact. It posts the same
/contactUs request that window did; a failure is said inside the form,
where the request is still there to retry, rather than in an alert
window the next click on the dashboard would bury.
2026-08-09 17:37:09 -07:00
Nariman Jelveh c8985b621d Add copy-link support for app tiles
Extract app tile URL generation into a shared helper and use it for a new dashboard context menu action that copies the tile's destination. This keeps copied links aligned with the existing open-in-new-tab behavior and adds regression coverage for app, external, encoded, and folder tiles.
2026-08-09 16:54:18 -07:00
Nariman Jelveh 42299a64ed Update recommended apps list
Add 'blockarena' and remove 'pretty-tiles', 'galaxy-troops', and 'blend-fruits' from the recommended apps list.
2026-08-09 11:20:40 -07:00
jelveh 344db5cb0b Remove vault from recommended apps
Drop `vault` from the recommended apps list so it no longer appears in app recommendations.
2026-08-08 11:54:56 -07:00
jelveh 66824f607d feat(gui): install an app onto the grid when its landing is what installs it
A /app/<name> landing for an app the dashboard didn't list opened with
a plain fade and left the app NOWHERE: the grid had been fetched before
the launch granted the permission that installedApps reports, so Back
found no tile to minimize into and the user landed on a dashboard
without the app they were just inside. The round trip that teaches
"windows are inflated tiles" broke on exactly the landing where a new
user is forming that model.

Now the landing SAYS what it does. The app joins the grid at the tail
(where new apps land) with a tile drawn from the landing's own app-info
prefetch -- only a confirmed-real app ever materializes, with its real
icon -- held invisible until the intro has revealed the grid and
travelled to the tile's page, and then it is INSTALLED, in the grammar
every app store taught: the slot opens (the folder well's own surface)
with the icon dim inside it, a progress stroke draws clockwise around
the slot, and on completion the icon springs to full color and size as
the label names it. Only then does the flourish play and the window
grow out of the tile. The arrival is the one beat that never decays:
per-app news delivered at most once per app, not a repeated lesson.
Minimize now has a target, and the app is simply there afterward.

The regression surface is guarded on every edge: the parked-invisible
state is re-applied by renderApps itself so mid-intro re-renders can't
strip it, and settleDeepLinkLaunch -- which the landing's finally
always reaches -- force-reveals it, so no path leaves an invisible
tile. A _pendingInstalls overlay (the mirror of _removedLocal) keeps a
refresh fetched before the grant lands from evicting the tile, and
retires itself once installedApps confirms the app or the user
uninstalls it again. The order is never saved on the splice, so the
default-position append can't freeze a custom order the user never
made. Failed or absent app info adds nothing; installed-app landings
are untouched; animations-off and reduced-motion users skip the
choreography but still get the tile, silently. The progress ring is a
transient element (the tile's ::before is the well, its ::after the
running dot) drawn as an SVG dashoffset sweep -- no Houdini dependency
-- and the app-info prefetch now asks for the 128px icons every other
tile already uses.
2026-08-08 11:25:48 -07:00
jelveh 66dedf3082 feat(gui): let the app drawer's intro flash retire once learned
The control drawer on headless dashboard apps opened itself on EVERY
window open, forever -- 2.6 seconds of tray over the app's own top
edge to teach a lesson (the controls live in this tongue) that is
learned in a few exposures. Same doctrine as the dashboard's
deep-link intro: pedagogy that knows when to step aside.

A per-account kv counter records deliveries, once per window: a flash
played while the user could see it, or the user opening the drawer
themselves (hover, tap, focus) -- the stronger proof, since the
drawer is a headless app's only chrome and must never decay out of a
user's awareness. After three, new windows keep the bare tongue and
the flash stops; hover, tap, and focus expansion never decay.

The attach is synchronous and the stored count is not: the first
drawer of a session races the read against a short cap and defaults
to teaching -- a hung read must not hold the intro hostage, and every
failure mode (unparseable value, failed read, timed-out read, failed
write) errs toward teaching once more, never toward never teaching.
The resolved count is cached and kept current in memory, so later
windows decide synchronously and a session's own not-yet-written
increments still count. kv.incr keeps two devices from losing an
increment, and the pre-read caps the key once the lesson is learned.

If the user beats a still-deciding flash to the drawer, the flash is
skipped rather than played over their hand -- its auto-collapse would
shut a drawer they are actively using. A flash in a hidden tab still
defers to first reveal (it rides requestAnimationFrame, as before)
and an unseen one is never counted.
2026-08-08 10:25:26 -07:00
Daniel Salazar 95da802b13 fix: metering hardening (#3527) 2026-08-08 06:10:01 -07:00
Daniel Salazar 0491d5fea7 fix: mcp better uploads (#3528) 2026-08-08 06:09:49 -07:00
Juan Fernando CastroandDaniel Salazar d202be10a9 feat: let apps use another app's data with user consent (#3516)
* feat(perms): add cross-app app-data permission vocabulary

* feat(perms): sweep app grants by permission prefix

* feat(perms): resolve and withdraw cross-app data grants

* feat(kv): support an authorized namespace override and per-key privacy

* feat(kv): gate cross-app KV access behind app-data grants

* feat(fs): allow cross-app AppData access and require a scope to delete

* feat(auth): accept permission lists and gate app-data grants

* feat(perms): add requestAppData to the puter.js SDK

* feat(gui): carry permission lists through the IPC and popup transports

* feat(gui): describe cross-app data requests in the consent dialog

* docs: document requestAppData and per-entry KV privacy

* perf(perms): sweep cross-app grants only for origin-bootstrapped apps

* fix(gui): stop double-encoding cross-app consent text

* fix(perms): close three gaps in cross-app grant enforcement

* fix(kv): meter and batch the per-entry privacy probe

* fix(perms): resolve app identifiers and scopes more strictly in the SDK

* test(perms): cover the cross-app consent flow end to end

* fix: small missing token resolution for app

also adds the same exclusion for the batchPut api, small change

* fix: make resolved actor optional

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-08-08 04:04:07 -07:00
jelveh 2c17476c07 fix(gui): folder pages, live drag-out, and one merge pulse in My Apps
Three fixes to the My Apps folders:

A folder big enough to need it now paginates like the grid outside
(iOS folders page too) instead of growing a scrollbar: pages of
cols x 3 tiles in a scroll-snap scroller with dots below, wheel and
swipe to flip, and a drag held at the card's edge flips folder pages
so reordering works across them. The card's height stays constant
across pages, and a deep-link landing on an app that lives on a later
folder page travels there before the launch morph.

Dragging an app out of the open folder no longer drops it blind:
after a beat held outside the card, the app leaves the folder, the
card closes behind it, and the SAME drag carries on over the grid --
placeholder shuffle, edge page flips, even a drop into another
folder -- so the app lands where the user watched themselves put it.
A quick drop during the beat still lands beside the folder, and
swinging back inside within the beat cancels cleanly.

The folder well's fill played twice on nearly every merge: the rest
countdown anchored at first contact with the target, so the hand
decelerating INTO the tile always read as movement when the dwell
elapsed, and the deliberate refill-from-empty replayed the fill. The
countdown now re-anchors on the move events themselves -- rest is
only visible at its edges -- so the fill that completes is the one
started by the last movement, played exactly once.
2026-08-08 01:28:09 -07:00
Nariman Jelveh 06d94534e8 feat(gui): iOS-style app folders in the My Apps grid (#3525)
* feat(gui): iOS-style app folders in the My Apps grid

Drag one app onto another and let it settle: a well opens under the
target and the drop makes a folder of the two. Dropping onto an existing
folder joins it. A folder opens by growing out of its own icon into a
card over a blurred grid, where its apps can be launched, rearranged,
renamed, or carried back out.

Hovering a tile mid-drag means two things — "push over, I'm passing
through" and "swallow me" — and the tile is barely bigger than its icon,
so pixels can't separate them; motion does. The shuffle is held while
the folder offer stands and fires when the icon leaves the tile, or at
the drop, so a quick drop onto a neighbour still reorders exactly as it
did. The offer itself re-arms rather than cancelling on movement: the
last events of a drag are the ones carrying the icon onto the target and
nothing is dispatched while it rests, so a cancel-on-movement dwell
could never fire at all.

Folders are stored in their own kv key; the grid's ORDER stays entirely
in the existing saved app order, with a folder occupying its first
member's slot and its members contiguous. Every saved order therefore
stays valid with no migration, and an app whose installedApps page
failed to load keeps both its folder and its position. Folders never
nest, one that drops below two apps dissolves, and a corrupt kv value
degrades to "no folders" rather than a broken tab.

Elsewhere:
- Search looks THROUGH folders — a match the user then has to hunt for
  inside one is not an answer.
- Minimize morphs into the FOLDER when an app lives in one, and a
  /app/<name> landing opens the folder so the launch grows out of the
  icon where the app actually is.
- The uninstall FLIP keyed surviving tiles by app name, which a folder
  tile doesn't have; it now keys by identity.

New pure model in appGroups.js with tests; verified end to end in the
running dashboard (create, join, open, rename, reorder, eject, ungroup,
launch-from-folder) alongside plain reorder, search, and uninstall.

* fix(gui): keep the folder name field from inheriting input[type=text] sizing

style.css styles every input[type=text] with `width: 100%` and grows it to
`padding: 7px; border: 2px` on focus. `input.myapps-group-name` matches at the
same specificity, so it only wins the properties it actually declares — width
was never one of them, and the focus rule declared neither padding nor border
width. The name field therefore spanned the entire folder card (so its hover
and focus chip read as a full-width bar rather than the name) and grew 8px
taller the moment it was clicked, shoving the folder's app grid down.

Spell the three out, in both the resting and the focus rule — the same trap
.myapps-search already documents next door.

* fix(gui): size folder icon ghosts to the icon they stand on

Border-box only reaches a folder's icon through `.dashboard * { box-sizing }`,
and every ghost cloned from one is appended to <body>, outside that rule: the
drag ghost, the click-time launch flourish, and the open/minimize morph ghosts
all fall back to content-box, where .myapps-group-icon's 5px padding is added
to the 56px slot. Each ghost rendered 66px square and 5px off, so it visibly
popped at exactly the moment it was supposed to sit flush on the real icon.

State box-sizing on the rule itself so a clone carries it wherever it lands.

* fix(gui): stop a closing folder from swallowing the next click

_closeGroup drops the open class and leaves the overlay in place for
GROUP_PANEL_CLOSE_MS so the card can recede into its tile. The scrim is
`position: fixed; inset: 0` and still hit-testable for that whole quarter
second, so a click on the grid during it landed on the outgoing overlay — whose
handler only re-runs _closeGroup, now a no-op. Shutting a folder and reaching
straight for an app did nothing.

Take the outgoing overlay out of hit-testing; it has no interactive job left.

* fix(gui): keep a folder name typed right up to the moment it closes

The name box commits on blur, and every exit that goes through a pointer blurs
it while the folder is still open — so clicking outside, or launching an app
from inside, keeps what was typed. Escape does not: _closeGroup clears
_openGroupId first and only then moves focus to the tile below (or removes the
card outright), so the blur arrives with no open folder to rename and
_renameGroup drops it. A brand-new folder opens with its name selected for
exactly this edit, so "type Games, press Escape" — the obvious way to dismiss
a dialog — was the path most likely to lose it.

Commit the pending name on the way out, before the folder id is gone.

* fix(gui): close an open folder when the Apps tab is re-entered

The dashboard hides an inactive section and calls onActivate on the way back
in; there is no deactivate hook, so a folder left open survives the round trip
and greets the user still open over a grid they walked away from. Worse,
onActivate's focusSearch then lands the caret in the search box behind the
folder's scrim, and typing filters the grid the card is covering — a modal with
the keyboard pointed outside it.

Shut the folder as the tab comes back: returning to the tab is returning to
the grid.

* fix(gui): move focus into a folder when it opens

Opening a folder called .focus({ preventScroll: true }) on a jQuery
object. jQuery's .focus() shorthand reads a lone non-function argument
as event DATA and binds a handler with it, so it never moved focus:
the folder opened modal over the grid with focus still on the tile
behind its scrim, where Tab walked away through the inert grid instead
of cycling inside the dialog — and the object it bound as a handler
threw a TypeError on that tile's every subsequent focus.

Focus the DOM node instead, as every other focus call in this file
already does.

* fix(gui): stop renaming a folder from swallowing the click that commits it

The folder's name box commits on blur, and blur fires on the PRESS —
before the click that press belongs to. Committing re-rendered, and the
re-render replaced every tile in the open folder, so by the time the
click was dispatched the tile under the pointer was detached and the
delegated handler never saw it. Typing a name and then tapping an app
in the folder — the path a brand-new folder puts the user on — renamed
the folder and did nothing else; the app only opened on a second click.

Rebuild the folder's contents only when they actually differ from what
is on screen. A rename doesn't change them, so nothing is detached, and
a background refresh no longer throws away hover/focus either. Tiles
that survive get their drag resting-rects cleared, since the card can
have moved under them since the rects were taken.

* fix(gui): keep Enter in a folder's name box from leaving the folder

Committing the name with Enter blurred the box, which left focus on
<body> — outside a dialog that is marked aria-modal and that traps Tab
on its own subtree. The next Tab therefore walked off through the inert
grid the folder is covering, exactly what the trap exists to prevent.

Step out onto the folder's first app instead; the same blur still
commits the name. The keystroke is stopped at the box because the
grid's document-level key handler reads Enter on a focused tile as
"launch it", and would otherwise have taken the focus move as its cue
to open an app the user never asked for.

* fix(gui): stop an open folder clipping its own uninstall badges

The folder's grid scrolls, so it clips anything outside its padding box
— and reorder mode's uninstall badge deliberately overhangs the top-left
corner of every tile. The top row's badges therefore rendered as flat
tabs rather than circles, on the one surface where they are a touch
user's only way to uninstall an app they have filed away.

Give the scroller 9px of top padding for the overhang to sit in and take
it straight back off as margin, so the card and everything in it stays
exactly where it was.

* fix(gui): hold page edge-flips while a folder merge is being offered

A tile in the pager's last column sits inside the 60px edge-flip zone, so
resting a dragged icon on it — the folder-making gesture — armed the edge
dwell alongside the merge dwell, and the page flipped out from under the
very folder the user was watching form (the merge target scrolls away
mid-offer, and the drop then resolves against its stale resting rect).
Worst on phones, where 72px tiles overlap the zone across the whole last
column.

A live merge offer now holds the edge flip: entering the zone arms no
dwell while an offer stands, and an offer that arrives during a running
dwell is re-checked at the flip (a resting pointer fires no event that
could clear the timer). Carrying the icon off the tile withdraws the offer
and the hold with it, so deliberate flips — resting in the bare edge
gutter — behave as before.

* fix(gui): make Escape cancel a folder rename instead of saving it

Escape pressed mid-edit fell through to the folder's close handler, and
closing commits whatever the name box holds — so the one key every inline
rename uses for "never mind" stored the abandoned half-typed name. Escape
in the box now puts the stored name back and steps out to the folder's
tiles, exactly the cancel Finder/Explorer taught; with nothing left to
cancel (name untouched) it falls through and closes the folder as before,
so a second press still exits.

* fix(gui): refill the folder well when the merge countdown restarts

The merge dwell pins its stillness anchor at first contact with the tile,
and a hand decelerating INTO a target routinely covers more than the
7px allowance between that moment and the first tick — so the countdown
quietly restarts. The well's fill, tuned to the same 460ms, had already
completed by then and just sat there half-open: a drop it seemed to
promise a folder for actually reordered. Restart the fill with the
countdown, so what the well shows is always the countdown that is
actually running.

* fix(gui): keep keyboard focus inside an open folder across its edits

The folder card is a modal dialog, but three of its flows stranded focus
on <body>, where Tab walks the inert grid behind the scrim:

- Remove from Folder rebuilds the card's grid after the context menu has
  dropped focus, so nothing inside the dialog holds it. The rebuild now
  hands focus back to the same app's tile (or the first) whenever it finds
  focus on <body> — never stealing from an uninstall modal, which holds it
  legitimately.
- The same eject dissolving the folder re-renders the grid right after
  _closeGroup's focus hand-back, replacing the very tile it chose. The
  ejected app's own tile — where the user is looking — takes focus instead.
- A click on the card's empty space focused nothing at all. The card now
  carries tabindex=-1 so such clicks land on it (no visible ring; a focus
  ring around the whole card would misread plumbing as selection), and the
  Tab trap wraps from there instead of stepping off through the scrim.

* fix(gui): size the folder name box to the name it holds

The box sat at the browser's default ~20ch regardless of content: a short
name floated in a hover pill far wider than the word, and a 40-character
one clipped while the card had room to spare. field-sizing: content hugs
the text between a 90px floor and the card's width, iOS-style; engines
without the property keep the default box exactly as before.
2026-08-08 00:28:03 -07:00
Neal Shah aea828ff12 add worker create event and new query methods (#3524) 2026-08-07 17:51:37 -04:00
Daniel Salazar c24f0fe08b fix: mcp fs copy, move, rename (#3517)
* fix: mcp fs copy, move, rename

* feat: more mcp tools for kv
2026-08-07 17:14:40 -04:00
jelveh 085ac116c7 fix(gui): keep app landings through signup, and unstack the session picker
Signing up from a direct /app/<name> landing redirected to '/' on
success, silently dropping the app the URL asked for — the deep link's
launch (and its dashboard intro) never happened. Signup now mirrors
login's pathname-preserving redirect, but only for app-landing routes
(/app/<name>, /desktop/app/<name>): every other route keeps the
historical '/', notably /action/signup, where returning to the same
path would just show the signup form again. The query string stays
dropped, matching login's credential-leak hygiene. This covers all
three signup entry points reachable from a landing: the login cover's
"Sign up", the session picker's "Create Account", and the
must_login_or_signup fallback.

The session picker (UIWindowSessionList) also left itself on top of
the cover windows its two links open, hiding their username fields:
"Create Account" tried to close the picker via the LOGIN window's
c2a selector (which matches nothing in the picker), and "Log Into
Another Account" never closed it at all. Both now close the picker —
in the reload flows only: the no-reload (popup) flows keep it open,
where it doubles as the fallback UI when the login/signup window is
abandoned mid-flow.

Picking an account was already correct (location.reload() keeps the
landing URL); with these fixes all three picker paths, plain login,
and signup all return to the app landing, where the boot replays the
launch and its intro.
2026-08-07 11:51:44 -07:00
jelveh 5538a5d24b feat(gui): let the deep-link intro step aside for users who know it
The intro exists to teach the dashboard's spatial model; once learned it
would only tax every bookmarked landing. Two mechanisms remove that tax:

- Interruptible: any real user input (pointer/key/wheel, isTrusted only)
  during the intro skips the remaining choreography and launches at
  once — it wakes the in-flight beat sleep, so the skip is immediate.
  Input never cancels the launch itself and is never swallowed;
  whatever it was doing (typing a search, clicking a tile) still
  happens.

- Exposure decay: after 3 delivered — or deliberately skipped — intros
  the beats collapse and the sequence plays in one breath, exactly like
  a warm tile click. Counted per account in kv (kv.incr, atomic across
  devices; capped at the threshold so the key stops changing) rather
  than per device: the lesson lives in the user's head and the account
  follows them, while localStorage would also bleed between accounts on
  a shared browser. Only real exposures count: a delivered flourish or
  an active skip with the grid on screen — hidden tabs, timeouts, and
  no-tile landings teach nothing and don't count. The animated page
  flip is exempt from decay: it isn't a repeated lesson, it's live
  wayfinding to where this app lives, and its settle is needed anyway
  to put the tile in view for the morph. Every failure mode (slow or
  failed kv read) errs toward teaching once more, never toward never
  teaching.

Also bounds the intro's wait on the app-list load by the same deadline
as the tile wait — fetch has no timeout of its own, so a stalled
installedApps request used to hold the deep-link launch hostage
indefinitely.
2026-08-07 11:23:47 -07:00
jelveh 14c14fb5c5 feat(gui): animate the pager flip when a deep-linked app lives on a later page
The deep-link intro used to flip to the tile's page instantly behind the
grid's load-fade, so landings on off-page apps woke up on page N with
nothing but the pager dots hinting a move happened. Now the grid always
reveals on its first page, holds the grid beat, and then visibly travels
to the tile's page before the tile's flourish — the journey shows where
the app lives, which is also where minimize will put it back.

Smooth scrollTo has no reliable completion event across engines, so the
travel uses a settle allowance (DEEP_LINK_INTRO_FLIP_SETTLE_MS, same
pattern as the drag code's DRAG_FLIP_SETTLE_MS): the scroll's ~450ms
plus a rest so the landing reads before the tile pops. First-page apps
skip the flip entirely and are unaffected; the hidden-page bail-out is
re-checked after the flip settles.
2026-08-07 11:12:03 -07:00
jelveh 41fc8d890b feat(gui): play the tile click→morph→open intro on /app/<name> landings
A direct landing on /app/<name> now plays the same sequence a real
Apps-tab tile click does — grid appears, a beat, the tile's icon ghost
pops out of its slot, a beat, and the window morphs out of the tile —
so the landing shows the user what is being opened and where minimize
puts it back.

TabApps.beginDeepLinkLaunch waits (3s cap) for the tile to be genuinely
visible (list loaded, render done, pager flipped to the tile's page,
load-fade revealed, icon painted), paces the beats, and claims the app
in _launchingApps so a click mid-intro can't spawn a duplicate. The
launch's app-info fetch is prefetched in parallel so the intro never
delays the app's own round-trip. No tile, animations off, reduced
motion, or a background tab (hidden pages throttle timers and defer
rendering) all skip straight to the immediate plain-fade launch.

Also fixes the tab title sticking as the app's name after closing a
deep-linked app: the landing's replaceState('/') committed the
dashboard's history entry while the page still carried the server's
app-name title, and Chrome shows an entry's stored title when close/
Back traverses onto it — document.title is now reset before the
replaceState so the entry is stamped with the dashboard's own title.
2026-08-07 10:34:21 -07:00
Reynaldi Chernando 45bdc46df0 Remove deprecated models 7 aug (#3522) 2026-08-07 23:20:50 +07:00
jelveh 82ca390bf4 fix(gui): create-in-folder via sidebar right-click renders in target dir
Right-clicking a sidebar/breadcrumb folder in the dashboard Files tab and
choosing New > Folder (or any file type) created the item in the right-
clicked folder correctly, but the UI inserted the new row and started the
inline rename in whatever directory was currently open — so the item
appeared to be created in the wrong place until a refresh.

Now, when the creation target isn't the directory on screen, navigate to
the target and run the select + rename flow there; same-directory
creation keeps the incremental insert.
2026-08-07 08:49:13 -07:00
Neal Shah edfd67a83c fix PUT-1444 (#3515) 2026-08-06 17:11:04 -04:00
404oops ff16ec8900 Add Neuralwatt support (#3509) 2026-08-06 17:01:33 -04:00
Daniel Salazar c7edfc0c74 perf: improve metering perf + add extra compat for monthly charges (#3513)
closes PUT-1445 and PUT-1446
2026-08-06 10:26:47 -07:00
velzie 09f30d693c Peer: allow anontoken in place of authentication (#3501) 2026-08-06 02:03:53 -04:00
360 changed files with 33879 additions and 7109 deletions

No files matched your search

+32
View File
@@ -13,6 +13,9 @@ on:
- 'tools/**'
- 'package.json'
- 'package-lock.json'
# The type-check job below reads these.
- 'tsconfig.json'
- 'tsconfig.build.json'
- '.github/workflows/backend-tests.yaml'
permissions:
@@ -20,6 +23,35 @@ permissions:
pull-requests: write
jobs:
# `tsconfig.build.json` builds with `noCheck: true`, so nothing else in CI
# runs the type checker and a missing export compiles to `undefined`,
# surfacing only when the call is finally reached at runtime. This diffs
# against tools/typecheck-baseline.json and fails only on *new* errors.
#
# The consuming repo runs an equivalent gate, but only once someone bumps the
# submodule pointer — too late to keep the error off this repo's default
# branch. Hence a gate here, on this repo's own pull requests.
#
# Its own job rather than a step in `test`: that one runs a base/PR matrix for
# coverage comparison, and the check only needs the PR ref, once.
typecheck:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '24'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Type check (new errors only)
run: npm run typecheck
test:
runs-on: ubuntu-latest
strategy:
+27
View File
@@ -23,6 +23,33 @@ permissions:
pull-requests: write
jobs:
# The JSDoc in src/puter-js/src is the source of truth for the SDK's public
# types. The declarations shipped to npm are generated from it at build time
# and never committed, so what needs guarding is the JSDoc itself: this
# generates the declarations and type-checks the published surface without
# skipLibCheck, which is how broken re-exports used to go unnoticed.
#
# Its own job rather than a step in `test`: that one builds bundles and
# installs a browser, and this needs neither.
types:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '24'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Check the puter.js JSDoc produces declarations that type-check
run: npm run check:puterjs:types
test:
runs-on: ubuntu-latest
+8 -9
View File
@@ -98,24 +98,23 @@ Follow the same layered structure inside an extension — unless it only needs a
[src/puter-js/](src/puter-js/) is the public SDK. It ships live from `https://js.puter.com/v2/` with no version pinning — every existing app picks up changes immediately. Treat every observable behavior (signatures, response fields, error codes) as something a production app depends on.
Layout: SDK modules in [src/puter-js/src/modules/](src/puter-js/src/modules/) (one file or directory per area — `FileSystem/`, `KV.js`, `ai/`, …), shared plumbing in [src/puter-js/src/lib/](src/puter-js/src/lib/), hand-maintained type declarations in [src/puter-js/types/](src/puter-js/types/), API tests in [src/puter-js/tests/api/](src/puter-js/tests/api/), UI e2e tests in [src/puter-js/tests/e2e/](src/puter-js/tests/e2e/), developer docs in [src/docs/](src/docs/).
Layout: SDK modules in [src/puter-js/src/modules/](src/puter-js/src/modules/) (one file or directory per area — `FileSystem/`, `kv/`, `ai/`, …), shared plumbing in [src/puter-js/src/lib/](src/puter-js/src/lib/), generated (gitignored) type declarations in `src/puter-js/types/`, API tests in [src/puter-js/tests/api/](src/puter-js/tests/api/), UI e2e tests in [src/puter-js/tests/e2e/](src/puter-js/tests/e2e/), developer docs in [src/docs/](src/docs/).
Language & typing: puter.js source is plain JavaScript — never TypeScript files — typed via JSDoc. Reference the hand-maintained declarations in [src/puter-js/types/](src/puter-js/types/) with `import(...)` types rather than re-declaring shapes:
Language & typing: puter.js source is plain JavaScript — never TypeScript files — typed via JSDoc. **The JSDoc is the source of truth for the SDK's types.** `src/puter-js/types/` is `tsc --emitDeclarationOnly` output: the SDK build (`npm run build` in `src/puter-js`) generates it, the npm tarball ships it so TypeScript consumers get declarations, and git ignores it — it is never committed and never edited by hand. `npm run check:puterjs:types` generates it in CI and type-checks the result without `skipLibCheck`. The one hand-written declaration file is [src/puter-js/index.d.ts](src/puter-js/index.d.ts), which decides what is public and re-exports the generated names.
Declare a shape where it belongs, and reference it with an `import(...)` type from elsewhere. A shape more than one file needs lives in the module's `types.js`; a shape shared across modules lives in [src/puter-js/src/lib/types.js](src/puter-js/src/lib/types.js); a shape with one consumer can stay next to it:
```js
/** @typedef {import('../../types/modules/ai').ChatOptions} ChatOptions */
/** @type {ChatOptions} */
const options = { model: 'gpt-5-nano' };
/** @typedef {import('./types.js').KVOptConfig} KVOptConfig */
```
When a shape only exists locally, declare it as an inline object literal — not the `@typedef {Object}` + `@property` list form — and use `unknown` over `*`:
Use `@typedef {Object}` + `@property` for any shape whose fields need documenting — it is the only JSDoc form that carries a doc comment per field into the generated declaration. Keep the inline object-literal form for small internal shapes with nothing to say about each field, and prefer `unknown` over `*`:
```js
/** @typedef {{ key: string, value: unknown }} KVEntry */
```
Public (exposed) methods must carry JSDoc types — parameters and return value, matching the `.d.ts` declarations exactly. Unexposed/private helpers are typed at the contributor's discretion: annotate where it helps the next reader, and either way keep them clean.
Public (exposed) methods must carry JSDoc types — parameters and return value — with one `@overload` block per accepted call form, since those overloads *are* the published signature. Unexposed/private helpers are typed at the contributor's discretion: annotate where it helps the next reader, and either way keep them clean. Members tagged `@internal` are stripped from the generated declarations, so use that tag rather than `@private` to keep something off the public surface.
Typing in JS files is encouraged: annotate with JSDoc `@type`/`@param`/`@returns` using the TypeScript type system, and define shared shapes with `@typedef`. API types must not be `unknown` or untyped `...args` — spell out the real parameter and return shapes; the only exception is values passed through transparently to an upstream layer that owns their type. For example:
@@ -133,7 +132,7 @@ Every SDK change carries all five of the following — a puter.js PR missing one
1. **Backward compatibility.** Mandatory unless a maintainer explicitly signs off on a break. Existing call signatures keep working (including both positional and options-object forms where a method supports them); new parameters are optional with defaults that preserve old behavior; never rename or repurpose existing params, response fields, or error codes. New parameter names are `camelCase` (existing `snake_case` stays for compatibility). Say in the PR how existing callers are unaffected.
2. **Tests.** Add or extend a suite in [tests/api/suites/](src/puter-js/tests/api/suites/) (`<area>.suite.ts`; register new suites in `suites/index.ts` — no globbing). One suite runs unchanged on node, browser, and workerd via `npm run test:puterjs`; never write a per-platform test. The runners execute the **built** bundle — run `npm run build:workerLib` after SDK changes or the suite silently tests stale code. For `puter.ui.*` methods rendered by the desktop, use the Playwright e2e harness instead — see [src/puter-js/TESTING.md](src/puter-js/TESTING.md).
3. **Docs.** New or changed APIs update [src/docs/src/](src/docs/src/): the method page (`<Area>/<method>.md`, with frontmatter and a runnable example) and the area overview when the surface changes. Docs are the contract users code against — signatures, defaults, and return shapes must match the implementation exactly.
4. **Types.** Update [src/puter-js/types/modules/](src/puter-js/types/modules/)`<module>.d.ts` and re-export new types through `index.d.ts` / `types/puter.d.ts`. Declarations must match runtime behavior exactly — a wrong type is worse than a missing one.
4. **Types.** Type the change in JSDoc on the implementation, then run `npm run check:puterjs:types` to confirm it still produces declarations that type-check. Name any new type in [src/puter-js/index.d.ts](src/puter-js/index.d.ts) if consumers should be able to import it. Declarations must match runtime behavior exactly — a wrong type is worse than a missing one.
5. **Error handling.** Reject/throw `{ message, code }` objects with stable `snake_case` codes, matching the existing modules (see `KV.js`). Validate cheap preconditions client-side before making the network call; pass backend errors through unchanged rather than swallowing or re-wrapping them. Error codes are API surface — changing one is a breaking change.
[doc/contributing-apis.md](doc/contributing-apis.md) walks the full lifecycle of adding an API across backend + SDK.
+39 -35
View File
@@ -6,33 +6,33 @@ We at **Puter** are committed to maintaining a secure experience for our users a
The following are in scope for this program:
* **The Puter open-source project** (available at [github.com/HeyPuter/puter](https://github.com/HeyPuter/puter))
* **`puter.com`**
* **`api.puter.com`**
- **The Puter open-source project** (available at [github.com/HeyPuter/puter](https://github.com/HeyPuter/puter))
- **`puter.com`**
- **`api.puter.com`**
Out-of-scope:
* Third-party services, applications, or libraries not maintained by Puter.
* Social engineering attacks (e.g., phishing against staff).
* Denial of Service (DoS), spam, or volumetric attacks.
* Physical security issues.
- Third-party services, applications, or libraries not maintained by Puter.
- Social engineering attacks (e.g., phishing against staff).
- Denial of Service (DoS), spam, or volumetric attacks.
- Physical security issues.
## Known Non-Issues (Please Check Before Submitting)
The following have already been reviewed and determined **not to be vulnerabilities**. Reports that only re-describe one of these are **not eligible for a reward** and will be closed as non-issues — even if they include new code references. Please review this list before submitting:
* **XSS / CORS / token issues scoped to `api.puter.com`.** The primary user session cookie lives on `puter.com`, not on the API origin, so reflected/stored XSS, CORS, or token handling on `api.puter.com` is generally out of scope. Two exceptions we *do* evaluate on their merits — please report these: (a) attacker-controlled content served **inline** (e.g. an HTML `Content-Type`) from API file/response endpoints, and (b) anything that abuses the app-scoped `puter_token_v2` companion cookie set on the API host.
* **SSRF via `secureFetch`.** Production routes outbound requests through an isolated proxy that has no access to internal/SSRF-sensitive resources.
* **Attacks that depend on guessing an `appInstanceID` or app UID.** These are random 128-bit secret values and are not considered guessable.
* **Apps invoking drivers, creating workers, or using KV.** Applications are intended to do this; worker permissions are scoped to the owning app. This is by design.
* **App metadata or app user-count "leaks".** This information is currently public by design.
* **General "token in a URL" / token-lifetime designs** — signed directory URLs exposing children, a write signature implying read, or app tokens outliving a web session. These are current intended behaviors.
* **Missing PKCE, unverified `id_token` signatures, or other OIDC hardening.** Provider tokens are obtained through a server-to-server authorization-code exchange with the provider's token endpoint over TLS, so the resulting `id_token` / userinfo claims are trusted from that channel rather than from local JWKS signature verification — the callback never accepts a caller-supplied `id_token`. Adding local signature / `aud` / `iss` / `exp` checks is welcome defense-in-depth (please open a GitHub issue/PR), but their absence is not an account-takeover vector on its own.
* **JWT "algorithm confusion" / unpinned `algorithms` in `jwt.verify`.** Puter's session tokens are HMAC-signed (HS256) with a server-side secret, and there is no asymmetric public key anywhere in the verification path, so `alg`-substitution attacks do not apply. Explicitly pinning `algorithms` is a fine hardening PR, but it is not a vulnerability.
* **Static-source "SQL injection" in internal pagination.** Findings such as `LIMIT ${limit}` in list/notification queries: the limit is numerically coerced and clamped before it reaches the query, so it is not reachable with attacker-controlled string input. Hardening PRs to the internal stores are welcome, but these are not exploitable as reported.
* **Deprecated `saveTo*` GUI app messages.** The legacy `saveToDesktop` / `saveToDocuments` / etc. app-IPC handlers can create — never overwrite — new files in standard user folders. The behavior is non-destructive, path-traversal-safe, and deprecated (slated for removal); it is not treated as a sandbox escape.
* **Best-practice suggestions** such as login/registration username enumeration (kept intentionally for UX) or unauthenticated unsubscribe links (industry norm).
* **Rate-limiting suggestions for TURN credential issuance** (intentional; not billed per tunnel).
- **XSS / CORS / token issues scoped to `api.puter.com`.** The primary user session cookie lives on `puter.com`, not on the API origin, so reflected/stored XSS, CORS, or token handling on `api.puter.com` is generally out of scope. Two exceptions we _do_ evaluate on their merits — please report these: (a) attacker-controlled content served **inline** (e.g. an HTML `Content-Type`) from API file/response endpoints, and (b) anything that abuses the app-scoped `puter_token_v2` companion cookie set on the API host.
- **SSRF via `secureFetch`.** Production routes outbound requests through an isolated proxy that has no access to internal/SSRF-sensitive resources.
- **Attacks that depend on guessing an `appInstanceID` or app UID.** These are random 128-bit secret values and are not considered guessable.
- **Apps invoking drivers, creating workers, or using KV.** Applications are intended to do this; worker permissions are scoped to the owning app. This is by design.
- **App metadata or app user-count "leaks".** This information is currently public by design.
- **General "token in a URL" / token-lifetime designs** — signed directory URLs exposing children, a write signature implying read, or app tokens outliving a web session. These are current intended behaviors.
- **Missing PKCE, unverified `id_token` signatures, or other OIDC hardening.** Provider tokens are obtained through a server-to-server authorization-code exchange with the provider's token endpoint over TLS, so the resulting `id_token` / userinfo claims are trusted from that channel rather than from local JWKS signature verification — the callback never accepts a caller-supplied `id_token`. Adding local signature / `aud` / `iss` / `exp` checks is welcome defense-in-depth (please open a GitHub issue/PR), but their absence is not an account-takeover vector on its own.
- **JWT "algorithm confusion" / unpinned `algorithms` in `jwt.verify`.** Puter's session tokens are HMAC-signed (HS256) with a server-side secret, and there is no asymmetric public key anywhere in the verification path, so `alg`-substitution attacks do not apply. Explicitly pinning `algorithms` is a fine hardening PR, but it is not a vulnerability.
- **Static-source "SQL injection" in internal pagination.** Findings such as `LIMIT ${limit}` in list/notification queries: the limit is numerically coerced and clamped before it reaches the query, so it is not reachable with attacker-controlled string input. Hardening PRs to the internal stores are welcome, but these are not exploitable as reported.
- **Deprecated `saveTo*` GUI app messages.** The legacy `saveToDesktop` / `saveToDocuments` / etc. app-IPC handlers can create — never overwrite — new files in standard user folders. The behavior is non-destructive, path-traversal-safe, and deprecated (slated for removal); it is not treated as a sandbox escape.
- **Best-practice suggestions** such as login/registration username enumeration (kept intentionally for UX) or unauthenticated unsubscribe links (industry norm).
- **Rate-limiting suggestions for TURN credential issuance** (intentional; not billed per tunnel).
If you believe you have a **genuinely new** exploit chain that defeats one of these rationales (for example, demonstrating a sensitive credential that really is reachable on `api.puter.com`), say so explicitly and show why the reasoning above does not apply.
@@ -40,7 +40,7 @@ If you believe you have a **genuinely new** exploit chain that defeats one of th
To participate, you must:
1. **Report responsibly**: Provide detailed steps to reproduce the issue, including proof-of-concept code, screenshots, or a screen recording (see *Proof of Reproduction* below).
1. **Report responsibly**: Provide detailed steps to reproduce the issue, including proof-of-concept code, screenshots, or a screen recording (see _Proof of Reproduction_ below).
2. **Do no harm**: Do not exfiltrate, modify, or delete data. Only access your own account or test data.
3. **Respect availability**: Do not perform denial-of-service attacks or automated scans that degrade service.
4. **Follow disclosure policy**: Do not publicly disclose vulnerabilities until we have confirmed and patched the issue.
@@ -51,26 +51,30 @@ Reports that do not meet these guidelines may not be eligible for a reward.
## Proof of Reproduction
Reports must demonstrate a **working, reproducible exploit with real impact** — not a theoretical or static-source-review finding. Please include:
Reports **MUST** demonstrate a **working, reproducible exploit with real impact** — not a theoretical or static-source-review finding. Please include:
* Exact steps to reproduce, the relevant request/response or code path, and the commit or version you tested.
* The **observed** result versus the **expected** result.
* For client-side, UI, or authentication-flow bugs: a short screen recording (≤ 2 minutes) showing the exploit working end-to-end on a real Puter instance.
* For server-side bugs: a runnable proof-of-concept.
- Exact steps to reproduce, the relevant request/response or code path, and the commit or version you tested.
- The **observed** result versus the **expected** result.
- For client-side, UI, or authentication-flow bugs: a short screen recording (≤ 2 minutes) showing the exploit working end-to-end on a real Puter instance.
- For server-side bugs: a runnable proof-of-concept and screenshot or video showing the attack process.
Reports based solely on reading the source ("source review only, not tested") or unverified AI/LLM-generated reports are the **lowest triage priority and are generally not eligible**. If you used an AI tool to help find an issue, you must personally verify that it actually reproduces before submitting.
Reports based solely on reading the source ("source review only, not tested") or unverified AI/LLM-generated reports are the **lowest triage priority and not eligible for bounty**.
If you used an AI tool to help find an issue, you must personally verify that it actually reproduces before submitting.
Please submit **one issue per report**. Bundled "audit packs" of many speculative findings will be declined; send each confirmed issue separately.
Please submit **one issue per report**.
Bundled "audit packs" of many speculative findings will not be elligible for bounty; send each confirmed issue separately.
Additionally please **DO NOT** submit too many issues at once; these will be treated as a bundled report, and will not be elligible for bounty.
## Reporting Process
To report a vulnerability, email us at: **[[email protected]](mailto:[email protected])**.
Include:
* A description of the vulnerability
* Steps to reproduce
* Potential impact
* Suggested remediation (if available)
- A description of the vulnerability
- Steps to reproduce
- Potential impact
- Suggested remediation (if available)
- Paypal id where we can send the bounty (if eligible)
We aim to acknowledge receipt within **72 hours** and provide a resolution timeline.
@@ -78,10 +82,10 @@ We aim to acknowledge receipt within **72 hours** and provide a resolution timel
We offer monetary rewards based on the severity of the vulnerability, as determined by our internal assessment (using CVSS as a guide).
* **Critical: \$1,000 – \$2,000**
* **High: \$500 – \$1,000**
* **Medium: \$200 – \$500**
* **Low: \$50 – \$100**
- **Critical: \$1,000 – \$2,000**
- **High: \$500 – \$1,000**
- **Medium: \$200 – \$500**
- **Low: \$50 – \$100**
Non-security issues, suggestions, and best practices feedback are always welcome, but may not qualify for a reward.
If multiple researchers report the same issue, the bounty will be awarded to the first eligible report we receive.
+20 -2
View File
@@ -437,6 +437,24 @@
],
// ── Metering ────────────────────────────────────────────────────────
// When true, all metering checks pass — no per-actor limits enforced.
"unlimitedMetering": false
// When true, every account resolves to an unlimited policy: usage is still
// recorded, but nothing is ever refused for lack of budget. This is the
// setting for a deployment with no way to buy more — without it, accounts
// are held to the free monthly allowance and start getting 402s from the AI
// surfaces, file transfers and KV once they pass it.
"unlimitedMetering": false,
// Whether an account that has spent its whole allowance is refused the
// operations that spend it — file transfers, KV calls. Recording is
// unaffected either way. `workers` extends the same refusal to
// worker-driven calls, which are exempt by default because a deployed
// worker has nowhere to surface a payment prompt.
// "meteringEnforcement": { "enabled": true, "workers": false },
// Fleet-wide spend rate, in micro-cents per minute, past which metering
// raises the `metering:excessiveGlobalUsageRate` alarm. Omit it (the
// default) to leave the check off: the only useful value is a multiple of
// what this deployment's normal traffic costs, so it has to be measured
// rather than guessed, and a stale number here alarms on healthy growth.
// "maxGlobalUsagePerMinute": 200000000
}
+11 -5
View File
@@ -6,9 +6,9 @@ transports by **severity**.
```ts
this.clients.alarm.create(
`driver_rate_limit_hit:${iface}:${method}`, // de-dupe key
`Driver rate limit hit on ${iface}:${method}`, // what a human reads
{ iface, method, userUuid }, // context fields
`metering_write_failed:${userUuid}`, // de-dupe key
'Metering write failed', // what a human reads
{ userUuid, appId, error }, // context fields
'info', // severity
);
```
@@ -37,8 +37,14 @@ pass one explicitly unless you really mean "page someone".
- Did the server fail to do its job in a way nobody expected? → `critical`
- Is a background job, rate, or dependency degraded? → `warning`
- Is this a user doing something notable (hitting a limit, tripping an abuse
heuristic, overspending)? → `info`
- Is this a user doing something notable (tripping an abuse heuristic,
overspending)? → `info`
- Did one of *our own* limits reject a caller — a rate limit, a concurrency
cap, a quota? → don't alarm at all. The limit doing its job is not an
event; the 429 is the whole signal, and alarming on it only produces noise
proportional to traffic. An *upstream provider* rate-limiting us is the
opposite case and still alarms (`upstream_rate_limited`, `info`) — that one
is not something we chose.
An extension whose signals are all one tier can default its own local
`raiseAlarm` helper to that tier instead of repeating it at every call site —
+5 -1
View File
@@ -54,6 +54,7 @@ Both are supported ways to define an API. **Prefer a controller when you need fi
- **Controller routes** (and extension routes — same options) take [`RouteOptions`](../src/backend/core/http/types.ts): auth gates (`requireAuth`, `requireUserActor`, `noUserSession`, `adminOnly`, `allowedAppIds`, and the access-token controls), `subdomain` routing, per-route `rateLimit`, body parsers, and arbitrary extra `middleware`. The auth flavors are subtle and default-deny — read the JSDoc on each field before picking.
- **Driver methods** get their policies from the `@Driver` options: per-method `rateLimit` (limit/window/backend), `concurrent` in-flight caps (optionally `bySubscription`), and `noUserSession`. The `/drivers/call` surface enforces them.
- **An endpoint that spends metered resources** on the caller's behalf — moving file content, making object-store requests, anything else the account is billed for — also declares `requireCredits: true`, which turns an account with nothing left of its budget away with a 402 before the handler runs. Endpoints that only describe or delete things deliberately don't: an account that has run out still has to be able to see what it has, clear it, and reach its billing pages. Drivers have no route options to declare this on, so they call `assertActorHasCredits` themselves ([src/backend/services/metering/enforcement.ts](../src/backend/services/metering/enforcement.ts)) — see `KVStoreDriver`, which does it once for every method.
### 3. puter.js
@@ -62,7 +63,10 @@ Both are supported ways to define an API. **Prefer a controller when you need fi
### 4. Types
- Add or extend the declaration in [src/puter-js/types/modules/](../src/puter-js/types/modules/)`<module>.d.ts` and re-export new types through `index.d.ts`. Declarations must match the runtime exactly — optionality, defaults, and return types included.
- Type the method where you wrote it, in JSDoc: `@param`/`@returns` on the implementation, one `@overload` block per accepted call form, and `@typedef {Object}` + `@property` for any new shape. The JSDoc is the source of truth — declarations are generated from it, so there is nothing to keep in sync by hand.
- Put a shape more than one file needs in the module's `types.js` (e.g. [src/puter-js/src/modules/kv/types.js](../src/puter-js/src/modules/kv/types.js)); anything shared across modules goes in [src/puter-js/src/lib/types.js](../src/puter-js/src/lib/types.js). A shape with one consumer can stay next to it.
- Run `npm run check:puterjs:types` — it generates the declarations and type-checks the published surface without `skipLibCheck`. **Never edit anything under `src/puter-js/types/`**: it is gitignored build output, produced by the SDK build and shipped in the npm tarball, and the next build overwrites it.
- Name the new type in [src/puter-js/index.d.ts](../src/puter-js/index.d.ts) if consumers should be able to import it. That file is the one hand-written declaration in the package: it decides what is public and re-exports nothing else.
### 5. Docs
+31
View File
@@ -370,6 +370,37 @@ Default is 100 MB per user.
Set `is_storage_limited: false` for unlimited (bounded by host disk).
### Usage metering and budgets
Puter meters what an account costs to serve — bytes sent back, object-store
requests, KV capacity, AI tokens — against a monthly budget, and refuses the
operations that spend it once that budget is gone. The refusal is a `402` with
code `insufficient_funds`; reads that only describe things, and every kind of
deletion, stay available so an account can always see what it has and clear it.
On a self-hosted install this is almost certainly not what you want. There is
nowhere to buy more, so accounts are held to the free monthly allowance
(US$0.25 of measured cost — roughly 2 GiB of downloads) and start being refused
after that. Turn it off:
```json
"unlimitedMetering": true
```
Every account then resolves to an unlimited policy. Usage is still recorded, so
the dashboard still shows what is being consumed; nothing is ever refused for
lack of budget.
To keep the budgets but stop them blocking anything — recording only:
```json
"meteringEnforcement": { "enabled": false }
```
Calls driven by a deployed worker are exempt from enforcement by default,
because a worker has no prompt to show and nobody watching it fail. Set
`"meteringEnforcement": { "workers": true }` to include them.
### Captcha on signup / login
Built-in proof-of-work captcha — no external service needed.
+37 -4
View File
@@ -2,6 +2,14 @@ import { Context } from '@heyputer/backend/src/core';
import type { Actor } from '@heyputer/backend/src/core/actor';
import { HttpError } from '@heyputer/backend/src/core/http';
import { PuterDriver } from '@heyputer/backend/src/drivers/types';
import type {
DriverConcurrentConfig,
DriverRateLimitConfig,
} from '@heyputer/backend/src/drivers/meta';
import {
DEFAULT_FREE_SUBSCRIPTION,
DEFAULT_TEMP_SUBSCRIPTION,
} from '@heyputer/backend/src/services/metering/consts';
import { extension } from '@heyputer/backend/src/extensions';
// App-telemetry lets an app owner enumerate the users who have
@@ -50,17 +58,42 @@ const parseIntParam = (
* Driver exposing the `app-telemetry` interface.
*
* The `/drivers/call` permission gate checks
* `service:app-telemetry:ii:app-telemetry`, which every actor already holds
* via the blanket `service` grant (hardcoded-permissions.js +
* `service:app-telemetry:ii:app-telemetry`, which every actor already holds via
* the blanket `service` grant (hardcoded-permissions.js +
* `default_implicit_user_app_permissions`). The real authorization — "is the
* caller the app owner?" — is enforced inside `get_users` below, exactly as
* v1 did.
* caller the app owner?" — is enforced inside `get_users` below, exactly as v1
* did.
*/
export class AppTelemetryDriver extends PuterDriver {
readonly driverInterface = 'app-telemetry';
readonly driverName = 'app-telemetry';
readonly isDefault = true;
// Declaring nothing here would leave both methods on the generic
// 600/minute driver default, which does not fit a paginated scan that
// can ask for MAX_LIMIT rows at MAX_OFFSET. This is a dashboard read —
// nobody calls it in a loop.
readonly rateLimit: DriverRateLimitConfig = {
default: {
limit: 60,
window: 60_000,
bySubscription: {
[DEFAULT_FREE_SUBSCRIPTION]: 30,
[DEFAULT_TEMP_SUBSCRIPTION]: 10,
},
},
};
readonly concurrent: DriverConcurrentConfig = {
default: {
limit: 5,
bySubscription: {
[DEFAULT_FREE_SUBSCRIPTION]: 2,
[DEFAULT_TEMP_SUBSCRIPTION]: 2,
},
},
};
/** Users who have authenticated into the given app (owner-only). */
async get_users({
app_uuid,
+11 -1
View File
@@ -81,6 +81,16 @@ export const handleInstalledApps = async (
extension.get(
'/installedApps',
{ subdomain: 'api', requireUserActor: true, allowFullAccessToken: true },
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: {
scope: 'installed-apps',
limit: 120,
window: 60_000,
key: 'user',
},
},
handleInstalledApps,
);
+29 -5
View File
@@ -3,6 +3,7 @@ import { HttpError } from '@heyputer/backend/src/core/http';
import {
controllersContainers,
driversContainers,
servicesContainers,
} from '@heyputer/backend/src/exports';
import { extension } from '@heyputer/backend/src/extensions';
import type { Request, Response } from 'express';
@@ -18,7 +19,7 @@ function collectAllCosts(): Record<string, unknown>[] {
const all: Record<string, unknown>[] = [];
const collect = (
source: Record<string, unknown>,
kind: 'driver' | 'controller',
kind: 'driver' | 'controller' | 'service',
) => {
for (const [name, instance] of Object.entries(source)) {
const fn = (
@@ -43,6 +44,9 @@ function collectAllCosts(): Record<string, unknown>[] {
};
collect(driversContainers as Record<string, unknown>, 'driver');
collect(controllersContainers as Record<string, unknown>, 'controller');
// Services report the costs that aren't tied to one endpoint — egress,
// which is metered for every response there is.
collect(servicesContainers as Record<string, unknown>, 'service');
return all;
}
@@ -110,26 +114,46 @@ export const handleMeteringAllCosts = async (
res.json({ costs: cachedAllCosts });
};
/** Dashboard reads over the per-actor KV aggregates. */
const USAGE_READ_LIMIT = {
scope: 'metering-usage',
limit: 120,
window: 60_000,
key: 'user' as const,
};
extension.get(
'/metering/usage',
{ subdomain: 'api', requireAuth: true },
{ subdomain: 'api', requireAuth: true, rateLimit: USAGE_READ_LIMIT },
handleMeteringUsage,
);
extension.get(
'/metering/usage/:appIdOrName',
{ subdomain: 'api', requireAuth: true },
{ subdomain: 'api', requireAuth: true, rateLimit: USAGE_READ_LIMIT },
handleMeteringUsageForApp,
);
extension.get(
'/metering/globalUsage',
{ subdomain: 'api', adminOnly: true },
{
subdomain: 'api',
adminOnly: true,
// Sums across every shard of the global aggregate. Admin-gated, so
// this is loop protection — but one accidental poll is an
// expensive minute.
rateLimit: {
scope: 'metering-global-usage',
limit: 10,
window: 60_000,
key: 'user',
},
},
handleMeteringGlobalUsage,
);
extension.get(
'/metering/allCosts',
{ subdomain: 'api', requireAuth: true },
{ subdomain: 'api', requireAuth: true, rateLimit: USAGE_READ_LIMIT },
handleMeteringAllCosts,
);
+84 -1
View File
@@ -109,6 +109,9 @@ describe('whoami extension — handleWhoami', () => {
expect(body.oidc_only).toBe(false);
// is_user_token is present (true) for user actors.
expect(body.is_user_token).toBe(true);
// Account creation time, in unix seconds.
expect(typeof body.created_ts).toBe('number');
expect(body.created_ts).toBeGreaterThan(0);
// `directories` is only sent to user actors — confirm it’s present.
expect(body.directories).toBeDefined();
// taskbar_items is only sent to user actors.
@@ -132,7 +135,7 @@ describe('whoami extension — handleWhoami', () => {
expect(flags.payment_bypass).toBeUndefined();
});
it('strips desktop_bg_* and human_readable_age fields for app actors', async () => {
it('strips desktop_bg_*, created_ts and human_readable_age fields for app actors', async () => {
const user = await seedUser();
const { res, captured } = makeRes();
@@ -154,6 +157,8 @@ describe('whoami extension — handleWhoami', () => {
expect(body.desktop_bg_color).toBeUndefined();
expect(body.desktop_bg_fit).toBeUndefined();
expect(body.human_readable_age).toBeUndefined();
// Account age, in either form, is not exposed to apps.
expect(body.created_ts).toBeUndefined();
// Directories are user-only.
expect(body.directories).toBeUndefined();
});
@@ -201,6 +206,84 @@ describe('whoami extension — handleWhoami', () => {
expect(JSON.stringify(body)).not.toContain('bootstrap-secret');
});
it('never exposes phone, card fingerprint or signup identity', async () => {
const user = await seedUser();
await server.stores.user.update(user.id as number, {
phone: '+15551234567',
card_fingerprint: 'fp_ABC123',
requires_phone_verification: false,
requires_card_verification: false,
});
// Guard against a vacuous assertion below: the columns really do hold
// the values we then expect never to see on the wire.
const stored = await server.stores.user.getById(user.id as number, {
cached: false,
force: true,
});
expect(stored?.phone).toBe('+15551234567');
expect(stored?.card_fingerprint).toBe('fp_ABC123');
for (const actor of [
{ user: { uuid: user.uuid, id: user.id as number } },
{
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
},
]) {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleWhoami(makeReq(), res),
);
const body = captured.body as Record<string, unknown>;
expect(body.phone).toBeUndefined();
expect(body.card_fingerprint).toBeUndefined();
expect(body.password).toBeUndefined();
expect(body.otp_secret).toBeUndefined();
expect(body.signup_ip).toBeUndefined();
// Not just absent as a key — the values must not appear anywhere
// in the payload (nested under metadata, taskbar items, …).
const serialized = JSON.stringify(body);
expect(serialized).not.toContain('5551234567');
expect(serialized).not.toContain('fp_ABC123');
// The verification flags, which the GUI acts on, still ship.
expect(body).toHaveProperty('requires_phone_verification');
expect(body).toHaveProperty('requires_card_verification');
}
});
it('scrubs sensitive keys added to metadata, and leaves the cached row intact', async () => {
const user = await seedUser();
await server.stores.user.updateMetadata(user.id as number, {
tmp_password: 'bootstrap-secret',
billing: { card_fingerprint: 'fp_NESTED', tier: 'pro' },
});
const { res, captured } = makeRes();
await runWithContext(
{ actor: { user: { uuid: user.uuid, id: user.id as number } } },
() => handleWhoami(makeReq(), res),
);
const body = captured.body as Record<string, unknown>;
const metadata = body.metadata as Record<string, unknown>;
expect(metadata.tmp_password).toBeUndefined();
// Nested sensitive keys are removed too; siblings survive.
expect(metadata.billing).toEqual({ tier: 'pro' });
expect(JSON.stringify(body)).not.toContain('fp_NESTED');
// The scrub works on a copy — server-side state is untouched.
const fresh = await server.stores.user.getById(user.id as number, {
cached: false,
force: true,
});
expect(fresh?.metadata?.tmp_password).toBe('bootstrap-secret');
expect(
(fresh?.metadata?.billing as Record<string, unknown>)
?.card_fingerprint,
).toBe('fp_NESTED');
});
it('marks the user as oidc_only when password is null', async () => {
const slug = Math.random().toString(36).slice(2, 8);
const oidcUser = await server.stores.user.create({
+107 -14
View File
@@ -14,6 +14,15 @@ const timeago = (() => {
return new TimeAgo('en-US');
})();
// User timestamps come off the DB as SQL datetime strings; the wire format
// for all of them is unix seconds. Unparseable values are dropped rather
// than sent as NaN.
const toUnixSeconds = (value: unknown): number | undefined => {
if (!value) return undefined;
const ms = new Date(value as string | number | Date).getTime();
return Number.isNaN(ms) ? undefined : Math.round(ms / 1000);
};
// Allowlist of `config.feature_flags` keys safe to surface via /whoami.
// Anything not listed here stays server-side, so internal flags
// (payment_bypass, staff_only_*, etc.) cannot leak by accident. Add a
@@ -24,6 +33,68 @@ const CLIENT_VISIBLE_FEATURE_FLAGS: ReadonlySet<string> = new Set([
'prompt_user_when_navigation_away_from_puter',
]);
// Keys that must never leave the server, whoever put them on the response.
// `details` is an explicit pick, but the `whoami.details` event hands
// listeners the full UserRow next to the object they may write to, and
// `metadata` is a free-form blob — so any of these can arrive on the
// response without an edit to the pick above. The scrub runs last, over the
// whole payload, and is the one place that decides what "sensitive" means.
//
// Credentials and single-use tokens, the payment/phone identifiers used for
// verification (`card_fingerprint` is the Stripe fingerprint, stable per
// card number), the network identity recorded at signup, and internal
// anti-abuse bookkeeping. `requires_phone_verification` /
// `requires_card_verification` stay: they are the flags the GUI acts on, and
// they carry no identifier.
const SENSITIVE_KEYS: ReadonlySet<string> = new Set([
'password',
'tmp_password',
'pass_recovery_token',
'email_confirm_code',
'email_confirm_token',
'change_email_confirm_token',
'otp_secret',
'otp_recovery_codes',
'card_fingerprint',
'phone',
'clean_email',
'signup_ip',
'signup_ip_forwarded',
'signup_user_agent',
'signup_origin',
'signup_server',
'audit_metadata',
]);
// Depth-limited, cycle-safe walk deleting every SENSITIVE_KEYS entry it finds
// at any level (`metadata` and `taskbar_items` are both nested structures).
const scrubSensitive = (
value: unknown,
seen: Set<object> = new Set(),
depth = 0,
): void => {
if (depth > 8 || value === null || typeof value !== 'object') return;
if (seen.has(value as object)) return;
seen.add(value as object);
if (Array.isArray(value)) {
for (const entry of value) scrubSensitive(entry, seen, depth + 1);
return;
}
for (const key of Object.keys(value as Record<string, unknown>)) {
if (SENSITIVE_KEYS.has(key)) {
delete (value as Record<string, unknown>)[key];
continue;
}
scrubSensitive(
(value as Record<string, unknown>)[key],
seen,
depth + 1,
);
}
};
export const handleWhoami = async (
req: Request,
res: Response,
@@ -65,8 +136,12 @@ export const handleWhoami = async (
}
}
const metadata = user.metadata ? { ...user.metadata } : user.metadata;
if (metadata) delete metadata.tmp_password;
// Deep-copied (it is decoded JSON) so the scrub below edits the response
// and not the cached UserRow. Sensitive keys inside it — tmp_password and
// anything else on the denylist — are removed by scrubSensitive.
const metadata = user.metadata
? structuredClone(user.metadata)
: user.metadata;
const details: Record<string, unknown> = {
username: user.username,
@@ -75,7 +150,9 @@ export const handleWhoami = async (
unconfirmed_email: user.email,
email_confirmed: user.email_confirmed || user.username === 'admin',
requires_email_confirmation: user.requires_email_confirmation,
phone: user.phone,
// The phone number itself is deliberately absent: nothing on the
// client reads it, and it is PII that would otherwise be handed to
// every app actor. Only the verification flag ships.
requires_phone_verification: user.requires_phone_verification,
requires_card_verification: user.requires_card_verification,
desktop_bg_url: user.desktop_bg_url,
@@ -98,6 +175,7 @@ export const handleWhoami = async (
: undefined,
otp: !!user.otp_enabled,
feature_flags,
created_ts: toUnixSeconds(user.timestamp),
human_readable_age: user.timestamp
? timeago.format(new Date(user.timestamp as string))
: null,
@@ -141,14 +219,9 @@ export const handleWhoami = async (
}
// Last activity
if (user.last_activity_ts) {
try {
details.last_activity_ts = Math.round(
new Date(user.last_activity_ts as string).getTime() / 1000,
);
} catch {
/* ignore parse error */
}
const lastActivityTs = toUnixSeconds(user.last_activity_ts);
if (lastActivityTs !== undefined) {
details.last_activity_ts = lastActivityTs;
}
// Strip sensitive fields for app actors
@@ -164,6 +237,7 @@ export const handleWhoami = async (
delete details.desktop_bg_color;
delete details.desktop_bg_fit;
delete details.human_readable_age;
delete details.created_ts;
delete details.is_user_token;
delete details.metadata;
}
@@ -183,19 +257,38 @@ export const handleWhoami = async (
}
const subscription = details.subscription as
| { offering?: Record<string, unknown> }
| undefined;
{ offering?: Record<string, unknown> } | undefined;
if (subscription?.offering) {
delete subscription.offering.group;
delete subscription.offering.benefits;
delete subscription.offering.price_id;
}
// Last word on what ships, after every listener has had its say.
scrubSensitive(details);
res.json(details);
};
extension.get(
'/whoami',
{ subdomain: 'api', requireAuth: true, allowUnconfirmed: true },
{
subdomain: 'api',
requireAuth: true,
allowUnconfirmed: true,
// The GUI polls this, and each call fans out to every `whoami`
// event listener — so it costs more than the response suggests.
//
// It is also the call everything else leans on to find out who it is
// talking to, so it rides along with unrelated work rather than
// arriving at its own pace: the ceiling has to clear whatever the
// busiest session is doing, not what a person clicks.
rateLimit: {
scope: 'whoami',
limit: 1_800,
window: 60_000,
key: 'user',
},
},
handleWhoami,
);
+4 -3
View File
@@ -1,12 +1,12 @@
{
"name": "puter.com",
"version": "2.5.1",
"version": "26.07",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "puter.com",
"version": "2.5.1",
"version": "26.07",
"license": "AGPL-3.0-only",
"workspaces": [
"src/*",
@@ -1304,6 +1304,7 @@
},
"node_modules/@clack/prompts/node_modules/is-unicode-supported": {
"version": "1.3.0",
"extraneous": true,
"inBundle": true,
"license": "MIT",
"engines": {
@@ -20252,7 +20253,7 @@
},
"src/puter-js": {
"name": "@heyputer/puter.js",
"version": "2.5.4",
"version": "2.6.1",
"license": "Apache-2.0",
"dependencies": {
"@heyputer/kv.js": "^0.2.1",
+3
View File
@@ -67,6 +67,9 @@
"check-translations": "node tools/check-translations.js",
"prepare": "husky",
"build:ts": "tsc -p tsconfig.build.json && node ./tools/write-dist-package-json.mjs",
"check:puterjs:types": "node tools/checkPuterjsTypes.mjs",
"typecheck": "node tools/typecheck.mjs",
"typecheck:update": "node tools/typecheck.mjs --update",
"setupExtensions": "node ./tools/extensionSetup.mjs"
},
"workspaces": [
@@ -371,6 +371,24 @@ describe('AlarmClient alarm registry', () => {
expect(client.get('flap')?.occurrences).toHaveLength(2);
});
it('caps retained occurrences while still counting every repeat', () => {
const client = makeClient();
const seen = capture(client);
for (let i = 0; i < 50; i++) {
client.create('hot', `occurrence ${i}`, { i });
}
const alarm = client.get('hot')!;
expect(alarm.count).toBe(50);
expect(alarm.occurrences).toHaveLength(20);
expect(alarm.timestamps).toHaveLength(20);
// The window kept is the most recent one, not the oldest.
expect(alarm.occurrences[19].message).toBe('occurrence 49');
// Trimming history must not rewind what the transports are told.
expect(seen[49]).toMatchObject({ repeatCount: 50, isRepeat: true });
});
it('names anonymous handlers by their registration order', () => {
const client = makeClient();
client.addAlertHandler(async () => {
+27 -8
View File
@@ -60,6 +60,15 @@ interface RegisteredHandler {
/** Severity used when neither the call site nor config picks one. */
const FALLBACK_SEVERITY: PagerSeverity = 'critical';
/**
* How many recent occurrences an alarm keeps. An alarm is never cleared unless
* something calls `clear`, and a hot one repeats for as long as the fault lasts
* — so retaining every occurrence means retaining every message and field set
* it was ever raised with, request bodies and actors included, for the life of
* the process. The last few are what a human reads; the rest is only a count,
* and `count` keeps that.
*/
const OCCURRENCE_HISTORY_LIMIT = 20;
/** Keeps `info` alarms out of the paging system unless config says otherwise. */
const DEFAULT_PAGERDUTY_MIN_SEVERITY: PagerSeverity = 'warning';
/** Slack's ceiling once a pager exists: chat gets what doesn't page. */
@@ -321,6 +330,7 @@ export class AlarmClient extends PuterClient {
started: Date.now(),
// `recordOccurrence` below stamps the first occurrence; seeding one
// here too would report every alarm as one occurrence ahead.
count: 0,
timestamps: [],
occurrences: [],
};
@@ -381,16 +391,25 @@ export class AlarmClient extends PuterClient {
message: string,
fields: AlarmFields,
): void {
const now = Date.now();
alarm.message = message;
alarm.fields = { ...alarm.fields, ...fields };
alarm.timestamps.push(Date.now());
alarm.count++;
if (fields.error) alarm.error = fields.error;
alarm.occurrences.push({
message,
fields,
timestamp: Date.now(),
});
alarm.timestamps.push(now);
alarm.occurrences.push({ message, fields, timestamp: now });
if (alarm.timestamps.length > OCCURRENCE_HISTORY_LIMIT) {
alarm.timestamps.splice(
0,
alarm.timestamps.length - OCCURRENCE_HISTORY_LIMIT,
);
alarm.occurrences.splice(
0,
alarm.occurrences.length - OCCURRENCE_HISTORY_LIMIT,
);
}
}
private applyKnownErrors(alarm: Alarm): void {
@@ -438,7 +457,7 @@ export class AlarmClient extends PuterClient {
this.applyKnownErrors(alarm);
console.warn(
`[alarm] REPEAT ${displayId(alarm)} :: ${alarm.message} (${alarm.timestamps.length})`,
`[alarm] REPEAT ${displayId(alarm)} :: ${alarm.message} (${alarm.count})`,
);
if (alarm.noAlert) return;
@@ -476,7 +495,7 @@ export class AlarmClient extends PuterClient {
alarm.severity = resolved;
const fieldsClean = cleanFields(alarm.fields);
const repeatCount = alarm.timestamps.length;
const repeatCount = alarm.count;
const id = alarm.id || 'something-bad';
+10
View File
@@ -48,7 +48,17 @@ export interface Alarm extends AlarmOptions {
fields: AlarmFields;
error?: Error;
started: number;
/**
* Every occurrence ever counted, including those aged out of the two lists
* below.
*/
count: number;
/**
* Timestamps of the most recent occurrences only — see
* `OCCURRENCE_HISTORY_LIMIT`.
*/
timestamps: number[];
/** The most recent occurrences only — see `OCCURRENCE_HISTORY_LIMIT`. */
occurrences: AlarmOccurrence[];
severity?: PagerSeverity;
noAlert?: boolean;
+11 -4
View File
@@ -22,6 +22,7 @@ import {
POOL_ACQUIRE_TIMEOUT,
isNeverSentError,
isRetriableError,
isRolledBackError,
} from './retriableErrors.js';
const DEFAULT_MAX_QUEUE_SIZE = 1000;
@@ -361,8 +362,14 @@ export class SQLBatcher {
// Run one fallback item, retrying transient failures with backoff.
// A read-only batcher may retry anything transient; a batcher that
// carries writes only retries failures where the statement provably
// never reached the server — a write that died mid-flight may have
// committed, and re-running it would double-apply.
// did not apply — either it never reached the server, or the server
// rolled it back itself. A write that died mid-flight may have
// committed, and re-running that one would double-apply.
//
// Lock contention lands in the second group and is worth retrying rather
// than surfacing: an item is a single statement, so a deadlock victim has
// been fully undone, and the caller sees an unhandled 500 for what the
// database is telling us to just run again.
async #runFallbackItem(b) {
let attempt = 0;
while (true) {
@@ -379,8 +386,8 @@ export class SQLBatcher {
};
} catch (error) {
const canRetry = this.readOnly
? isRetriableError(error)
: isNeverSentError(error);
? isRetriableError(error) || isRolledBackError(error)
: isNeverSentError(error) || isRolledBackError(error);
if (!canRetry || attempt >= ITEM_RETRY_ATTEMPTS) {
return { ok: false, error };
}
@@ -208,6 +208,36 @@ describe('SQLBatcher', () => {
expect(fallbackAttempts).toBe(2);
});
it('retries a deadlocked write instead of surfacing it', async () => {
let fallbackAttempts = 0;
const conn = makeConnection((sql) => {
if (isBatchQuery(sql)) throw makeError('ER_LOCK_DEADLOCK');
fallbackAttempts++;
if (fallbackAttempts === 1) throw makeError('ER_LOCK_DEADLOCK');
return [[{ ok: 1 }], undefined];
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, { maxTimeInQueue: 5 });
// The victim statement was rolled back by the server, so re-running it
// can't double-apply — the caller should never see the deadlock.
const result = await batcher.query('UPDATE notification SET x', []);
expect(result[0]).toEqual([{ ok: 1 }]);
expect(fallbackAttempts).toBe(2);
});
it('gives up on a write that deadlocks past the retry budget', async () => {
const conn = makeConnection(() => {
throw makeError('ER_LOCK_DEADLOCK');
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, { maxTimeInQueue: 5 });
await expect(batcher.query('UPDATE hot SET x', [])).rejects.toMatchObject(
{ code: 'ER_LOCK_DEADLOCK' },
);
});
it('never retries deterministic row-level errors and does not escalate the breaker', async () => {
let fallbackAttempts = 0;
const conn = makeConnection((sql) => {
@@ -27,7 +27,7 @@ import { DatabaseClientFactory } from './index.js';
import { SqliteDatabaseClient } from './SqliteDatabaseClient.js';
/** Highest schema version the migration table can reach. */
const CURRENT_SCHEMA_VERSION = 60;
const CURRENT_SCHEMA_VERSION = 61;
const SYSTEM_USER_UUID = '5d4adce0-a381-4982-9c02-6e2540026238';
const sqliteConfig = (
@@ -94,6 +94,7 @@ const AVAILABLE_MIGRATIONS: [number, string[]][] = [
[57, ['0062_blocked-app-origins.sql']],
[58, ['0063_add_suspended_reason.sql']],
[59, ['0064_abuse-moderation-events.sql']],
[60, ['0065_app-feedback.sql']],
];
export class SqliteDatabaseClient extends AbstractDatabaseClient {
@@ -0,0 +1,48 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- User-to-developer feedback for apps that opt in. Mirrors SQLite migration
-- 0065. Opt-in is the new `apps.feedback_enabled` column (developer-writable
-- through the regular `puter.apps.update` path). Each row of `app_feedback`
-- is one message a signed-in user submitted through the GUI feedback dialog;
-- a copy is emailed to the app owner unless the per-app daily email cap
-- suppressed it (`email_sent` records which). `app_uid` is denormalized
-- alongside `app_id` so rows stay attributable after an app is deleted.
-- `created_at` is unix seconds.
--
-- Idempotent: the column add uses _puter_add_col (defined in mig_1, which
-- always runs first) and the table uses `CREATE TABLE IF NOT EXISTS`, so the
-- directory can replay safely.
CALL _puter_add_col('apps', 'feedback_enabled', '`feedback_enabled` tinyint(1) DEFAULT ''0''');
CREATE TABLE IF NOT EXISTS `app_feedback` (
`id` INT NOT NULL AUTO_INCREMENT,
`uid` CHAR(36) NOT NULL,
`app_id` INT NOT NULL,
`app_uid` CHAR(40) NOT NULL,
`user_id` INT NOT NULL,
`message` TEXT NOT NULL,
`source_env` VARCHAR(16) DEFAULT NULL,
`source_origin` VARCHAR(2048) DEFAULT NULL,
`email_sent` TINYINT(1) NOT NULL DEFAULT 0,
`created_at` BIGINT NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `uq_app_feedback_uid` (`uid`),
KEY `idx_app_feedback_app_created` (`app_id`, `created_at`),
KEY `idx_app_feedback_user_created` (`user_id`, `created_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
@@ -0,0 +1,47 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- User-to-developer feedback for apps that opt in. Mirrors SQLite migration
-- 0065 / MySQL mysql_mig_20. Opt-in is the new `apps.feedback_enabled` column
-- (developer-writable through the regular `puter.apps.update` path). Each row
-- of `app_feedback` is one message a signed-in user submitted through the GUI
-- feedback dialog; a copy is emailed to the app owner unless the per-app
-- daily email cap suppressed it (`email_sent` records which). `app_uid` is
-- denormalized alongside `app_id` so rows stay attributable after an app is
-- deleted. `created_at` is unix seconds.
--
-- Idempotent via IF NOT EXISTS.
ALTER TABLE apps ADD COLUMN IF NOT EXISTS feedback_enabled boolean NOT NULL DEFAULT FALSE;
CREATE TABLE IF NOT EXISTS app_feedback (
id BIGSERIAL PRIMARY KEY,
uid CHAR(36) NOT NULL UNIQUE,
app_id BIGINT NOT NULL,
app_uid VARCHAR(40) NOT NULL,
user_id BIGINT NOT NULL,
message TEXT NOT NULL,
source_env VARCHAR(16) DEFAULT NULL,
source_origin VARCHAR(2048) DEFAULT NULL,
email_sent BOOLEAN NOT NULL DEFAULT FALSE,
created_at BIGINT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_app_feedback_app_created
ON app_feedback (app_id, created_at);
CREATE INDEX IF NOT EXISTS idx_app_feedback_user_created
ON app_feedback (user_id, created_at);
@@ -0,0 +1,51 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- User-to-developer feedback for apps that opt in. Opt-in is the new
-- `apps.feedback_enabled` column (developer-writable through the regular
-- `puter.apps.update` path; a dedicated column rather than a `metadata` key
-- because Dev Center saves replace the whole metadata blob and would erase
-- it). Each row of `app_feedback` is one message a signed-in user submitted
-- through the GUI feedback dialog; a copy is emailed to the app owner unless
-- the per-app daily email cap suppressed it (`email_sent` records which).
-- `app_uid` is denormalized alongside `app_id` so rows stay attributable
-- after an app is deleted (abuse forensics). `source_env` is 'app' (desktop
-- dialog) or 'web' (puter.com popup opened from an external site);
-- `source_origin` is the popup opener's browser-attested origin, null for
-- desktop submissions. `created_at` is unix seconds. The (user_id,
-- created_at) and (app_id, created_at) indexes serve the sliding-window
-- rate-limit counts in AppFeedbackStore.
ALTER TABLE apps ADD COLUMN "feedback_enabled" tinyint(1) DEFAULT '0';
CREATE TABLE IF NOT EXISTS `app_feedback` (
"id" INTEGER PRIMARY KEY AUTOINCREMENT,
"uid" TEXT NOT NULL UNIQUE,
"app_id" INTEGER NOT NULL,
"app_uid" TEXT NOT NULL,
"user_id" INTEGER NOT NULL,
"message" TEXT NOT NULL,
"source_env" TEXT DEFAULT NULL, -- 'app' | 'web'
"source_origin" TEXT DEFAULT NULL, -- attested opener origin (web popups)
"email_sent" INTEGER NOT NULL DEFAULT 0,
"created_at" INTEGER NOT NULL -- unix seconds
);
CREATE INDEX IF NOT EXISTS idx_app_feedback_app_created
ON `app_feedback` (`app_id`, `created_at`);
CREATE INDEX IF NOT EXISTS idx_app_feedback_user_created
ON `app_feedback` (`user_id`, `created_at`);
@@ -54,6 +54,26 @@ const NEVER_SENT_ERROR_CODES = new Set([
POOL_ACQUIRE_TIMEOUT,
]);
/**
* Failures the server itself rolled back before returning. The statement did
* reach the database, so this is not `NEVER_SENT`, but InnoDB guarantees it
* left no effect — which makes a retry just as safe for writes.
*
* Only true for a _single_ statement: under autocommit each statement is its
* own transaction, so "the transaction was rolled back" means "this statement
* was rolled back". Retrying a multi-statement string on one of these would
* re-run the statements that already committed ahead of the failure.
*/
const ROLLED_BACK_ERROR_CODES = new Set([
// Deadlock — InnoDB picked this transaction as the victim and undid it.
// MySQL's own message for it is "try restarting transaction".
'ER_LOCK_DEADLOCK',
// Lock wait timeout. Rolls back the statement rather than the transaction
// unless innodb_rollback_on_timeout is set — the same thing when the
// transaction is one statement.
'ER_LOCK_WAIT_TIMEOUT',
]);
const errorCode = (error: unknown): string | undefined =>
(error as { code?: string } | null)?.code;
@@ -74,3 +94,13 @@ export const isNeverSentError = (error: unknown): boolean => {
const code = errorCode(error);
return Boolean(code && NEVER_SENT_ERROR_CODES.has(code));
};
/**
* Lock-contention failures the server rolled back on its own. Safe to retry a
* single statement on, writes included — see `ROLLED_BACK_ERROR_CODES` for the
* one-statement precondition.
*/
export const isRolledBackError = (error: unknown): boolean => {
const code = errorCode(error);
return Boolean(code && ROLLED_BACK_ERROR_CODES.has(code));
};
@@ -179,6 +179,29 @@ describe('EmailClient — template rendering', () => {
expect(captured.html).toContain('<strong>424242</strong>');
});
it('does not HTML-escape values in the plain-text subject header', async () => {
const client = startClient();
const captured: { subject?: string } = {};
vi.spyOn(client, 'sendRaw').mockImplementation(async (options) => {
captured.subject = options.subject;
return null;
});
await client.send('[email protected]', 'app-user-feedback', {
owner_username: 'dev',
sender_username: 'user',
sender_email: null,
app_title: "Bob's App & Games",
app_name: 'bobs-app',
app_link: 'https://puter.example/app/bobs-app',
message: 'hi',
});
// A subject is not HTML — entities would render literally in the
// recipient's mail client.
expect(captured.subject).toBe("New user feedback for Bob's App & Games");
});
it('escapes html and converts newlines in nl2br values', async () => {
const client = startClient();
let html = '';
+15 -2
View File
@@ -152,11 +152,16 @@ export class EmailClient extends PuterClient {
// -- Public API: sending ------------------------------------------
/** Render a template and send it to `to`. */
/**
* Render a template and send it to `to`. `options.replyTo` sets the
* Reply-To header (e.g. so a recipient can respond to the originator of the
* message rather than the no-reply From address).
*/
async send<T extends EmailTemplateName>(
to: string,
template: T,
values: Record<string, unknown> = {},
options: { replyTo?: string } = {},
): Promise<void> {
const compiled = this.compiledTemplates[template];
if (!compiled) {
@@ -168,6 +173,7 @@ export class EmailClient extends PuterClient {
to,
subject: compiled.subject(values),
html: compiled.html(values),
...(options.replyTo ? { replyTo: options.replyTo } : {}),
});
}
@@ -285,7 +291,14 @@ export class EmailClient extends PuterClient {
private compileTemplates(): void {
for (const [name, template] of Object.entries(EMAIL_TEMPLATES)) {
this.compiledTemplates[name as EmailTemplateName] = {
subject: handlebars.compile(template.subject),
// Subjects are plain-text headers: HTML-escaping would put
// literal entities in front of the recipient (&amp; etc.).
// Header safety is handled elsewhere — the transport encodes
// newlines, and free-form values (e.g. app_title) collapse
// whitespace upstream.
subject: handlebars.compile(template.subject, {
noEscape: true,
}),
html: handlebars.compile(dedent(template.html)),
};
}
+19
View File
@@ -75,6 +75,25 @@ Please update <a href="https://puter.com/app/{{app_name}}">{{app_title}}</a>.
<blockquote>{{nl2br message}}</blockquote>
<p>Best,<br />
The Puter Team
</p>
`,
},
'app-user-feedback': {
subject: 'New user feedback for {{app_title}}',
html: `
<p>Hi{{#if owner_username}} {{owner_username}}{{/if}},</p>
<p>
<strong>{{sender_username}}</strong>{{#if sender_email}} (<a href="mailto:{{sender_email}}">{{sender_email}}</a>){{/if}} sent feedback about <a href="{{app_link}}">{{app_title}}</a>:
</p>
<blockquote>{{{nl2br message}}}</blockquote>
{{#if sender_email}}<p>Just reply to this email to respond to them directly.</p>{{/if}}
<p>
You're receiving this because feedback is enabled for
<a href="{{app_link}}">{{app_title}}</a> — manage it in the
<a href="{{dev_center_link}}">Dev Center</a> under your app's settings.
</p>
<p>Best,<br />
The Puter Team
</p>
`,
},
@@ -214,6 +214,38 @@ describe('EventClient', () => {
});
});
describe('hasListeners', () => {
it('is false for a key nobody subscribed to', () => {
expect(target.hasListeners(key)).toBe(false);
});
it('is true for an exact subscriber', () => {
target.on(key, vi.fn());
expect(target.hasListeners(key)).toBe(true);
});
it('is true when only a wildcard prefix matches', () => {
target.on(`${key}.*`, vi.fn());
expect(target.hasListeners(`${key}.child.deep`)).toBe(true);
});
it('is true for an extension-registered listener', () => {
extensionStore.events[key] = [vi.fn()];
try {
expect(target.hasListeners(key)).toBe(true);
} finally {
delete extensionStore.events[key];
}
});
it('goes back to false once the last listener is removed', () => {
const listener = vi.fn();
target.on(key, listener);
target.off(key, listener);
expect(target.hasListeners(key)).toBe(false);
});
});
describe('extension-registered listeners', () => {
afterEach(() => {
delete extensionStore.events[key];
+21
View File
@@ -129,6 +129,27 @@ export class EventClient extends PuterClient {
});
}
/**
* Whether anything would run if `key` were emitted, wildcards included.
*
* For emitters whose work is only worth doing when someone is listening —
* gathering the payload costs a round trip, say. Emitting into the void is
* otherwise perfectly cheap and doesn't need this.
*/
hasListeners<T extends keyof EventMap>(key: T): boolean {
const parts = key.split('.');
for (let i = 0; i < parts.length; i++) {
const matchKey = (
i === parts.length - 1
? key
: `${parts.slice(0, i + 1).join('.')}.*`
) as ListenKey;
if (this.#eventListeners[matchKey]?.length) return true;
if (extensionStore.events[matchKey]?.length) return true;
}
return false;
}
/**
* Subscribe to an event by exact key OR a wildcard prefix.
*
+58
View File
@@ -20,6 +20,7 @@ import type {
Response as ExpressResponse,
} from 'express';
import { Actor } from '../../core';
import type { UsageInput } from '../../services/metering/types';
import { FSEntry } from '../../stores/fs/FSEntry';
// GUI write events spread an entry plus per-event metadata into `response`.
@@ -305,6 +306,38 @@ export type EventMap = {
};
'storage.quota.bonus': { userId: number; extra: number };
// ---- Metering ----
// Recurring charges are pure mechanism here: the metering service knows
// when to ask (once per user per month, the first time that month's usage
// record is touched) and how to record the answer, and an extension
// decides what the account owes. Listeners push onto `charges`; the
// service applies them as ordinary usage once every listener has run.
// Nothing listening → no charges and no claim is ever taken.
//
// Emitted after the claim is settled, so a listener that calls back into
// metering can't re-trigger it.
'metering.monthly.charges': {
/**
* User-scoped, with no app on it: the account is what recurs, and the
* app that happened to trigger the month's first call has nothing to do
* with what is owed. Price against what the user owns.
*/
actor: Actor;
/** The month being charged for, `YYYY-MM` in UTC. */
month: string;
/**
* Push what the user owes here. Every listener's charges are merged
* into one amount map and recorded as a single increment.
*/
charges: UsageInput[];
};
// ---- Workers ----
// Only a genuinely new worker. Redeploying an existing name updates its
// row instead, and never reaches here — so a listener that prices this
// is pricing workers that came into existence, not deploys.
'worker.create': { actor: Actor; workerName: string };
// ---- Outer / GUI broadcast ----
'outer.cacheUpdate': {
cacheKey: string[];
@@ -312,6 +345,15 @@ export type EventMap = {
ttlSeconds?: number;
};
'outer.fs.write-hash': { hash: string; uuid: string };
/**
* Cache keys the KV read cache must stop serving, because the entries
* behind them were just written somewhere else.
*
* `outer.*` rather than `outer.pubsub.*` on purpose: the cache lives in the
* Redis a cluster shares, so one node applying the invalidation covers the
* whole cluster — fanning it out to siblings would just repeat the write.
*/
'outer.kv.cacheInvalidated': { cacheKeys: string[] };
'outer.gui.item.added': GuiEvent;
'outer.gui.item.updated': GuiEvent;
'outer.gui.item.moved': GuiEvent;
@@ -402,6 +444,22 @@ export type EventMap = {
[K in `route.${string}`]: RouteLifecycleEvent;
} & {
[K in `pubsub.login.${string}`]: { authtoken: string };
} & {
/**
* A user's subscription now resolves to a different policy. Carried on the
* `outer.pubsub.*` channel so it reaches sibling nodes and peer clusters,
* not just the one that handled the change — every node caches the resolved
* policy, so a purchase is only live once they have all dropped theirs.
*/
'outer.pubsub.metering.subscription-changed': { userUuid: string };
/**
* A user's purchased credit balance changed. Separate from a policy change
* because it moves the other half of the same budget, and carried on the
* `outer.pubsub.*` channel for the same reason: every node caches whether
* an account has budget left, so a top-up only lifts enforcement once they
* have all dropped that answer.
*/
'outer.pubsub.metering.credits-changed': { userUuid: string };
};
/**
+14
View File
@@ -21,6 +21,7 @@ import {
AbortMultipartUploadCommand,
CompleteMultipartUploadCommand,
CreateMultipartUploadCommand,
HeadBucketCommand,
PutObjectCommand,
S3Client as AwsS3Client,
type S3ClientConfig,
@@ -244,6 +245,19 @@ export class S3Client extends PuterClient {
return client;
}
/**
* Cheapest round-trip that proves the object store answers for a bucket: no
* object data, no listing, just a HEAD. Throws on any failure (missing
* bucket, bad credentials, unreachable endpoint) so callers can treat it as
* a liveness probe. Reuses the pooled per-region client.
*/
async headBucket(
bucket = this.config.s3_bucket || LEGACY_STORAGE_BUCKET,
region?: string,
): Promise<void> {
await this.get(region).send(new HeadBucketCommand({ Bucket: bucket }));
}
// ------------------------------------------------------------------
// Legacy storage migration
// ------------------------------------------------------------------
+56 -5
View File
@@ -35,6 +35,39 @@ import DEFAULT_APP_ICON from './default-app-icon.js';
* are just thin shape adapters that translate REST conventions into driver
* calls.
*/
/**
* Desktop boot reads the app list and individual app records repeatedly, so the
* ceiling is set well above normal boot traffic and exists to catch a runaway
* client rather than to pace one.
*
* "Repeatedly" is the operative word: an app record is read on launch, on
* permission checks, and again by anything resolving an app by name, so these
* accumulate against whatever else a session is doing rather than arriving on
* their own. Sized for a session working hard, not for a person clicking.
*/
const APP_READ_LIMIT = {
scope: 'app-read',
limit: 1_800,
window: 60_000,
key: 'user',
};
/**
* Unauthenticated icon serving; no actor to key on, so the bucket is the
* address — and an address is a NAT, a campus or a carrier gateway that can
* hold hundreds of desktops. Each desktop boot pulls the taskbar's icons at
* once, so a single burst from one network is already thousands of requests.
* Responses are publicly cacheable and usually a redirect, so the ceiling is
* not protecting bandwidth; it is there so a client looping on a broken icon
* can't spin unbounded.
*/
const APP_ICON_LIMIT = {
scope: 'app-icon',
limit: 12_000,
window: 60_000,
key: 'ip',
};
export class AppController extends PuterController {
get appStore() {
return this.stores.app;
@@ -100,7 +133,9 @@ export class AppController extends PuterController {
return d;
}
registerRoutes(router) {
registerRoutes(
/** @type {import('../../core/http/PuterRouter').PuterRouter} */ router,
) {
// GET /apps — list apps owned by the current user
router.get(
'/apps',
@@ -108,6 +143,7 @@ export class AppController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: APP_READ_LIMIT,
},
async (req, res) => {
const apps = await this.appDriver.select({
@@ -122,8 +158,16 @@ export class AppController extends PuterController {
'/apps/nameAvailable',
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
requireAuth: true,
// Answers "does this name exist?" for any name, so it is a
// name-enumeration oracle however cheap it is to serve.
// Mirrors the `isNameAvailable` budget on AppDriver.
rateLimit: {
scope: 'app-name-available',
limit: 60,
window: 60_000,
key: 'user',
},
},
async (req, res) => {
const name = req.query?.name;
@@ -158,6 +202,7 @@ export class AppController extends PuterController {
{
subdomain: 'api',
requireAuth: true,
rateLimit: APP_READ_LIMIT,
},
async (req, res) => {
const actor = req.actor;
@@ -216,6 +261,7 @@ export class AppController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: APP_READ_LIMIT,
},
async (req, res) => {
const raw = req.params.name;
@@ -289,6 +335,7 @@ export class AppController extends PuterController {
{
subdomain: 'api',
requireAuth: true,
rateLimit: APP_READ_LIMIT,
},
async (req, res) => {
const appList = Array.isArray(req.body) ? req.body : [];
@@ -497,10 +544,14 @@ export class AppController extends PuterController {
// Icons are <img src> targets from the GUI (root) AND resolved via
// api_base_url in taskbar payloads. Register on both so either origin
// works without a cross-subdomain redirect.
router.get('/app-icon/:app_uid', { subdomain: ['api', ''] }, serveIcon);
router.get(
'/app-icon/:app_uid',
{ subdomain: ['api', ''], rateLimit: APP_ICON_LIMIT },
serveIcon,
);
router.get(
'/app-icon/:app_uid/:size',
{ subdomain: ['api', ''] },
{ subdomain: ['api', ''], rateLimit: APP_ICON_LIMIT },
serveIcon,
);
}
@@ -1018,3 +1018,25 @@ describe('AppController GET /app-icon remote icons', () => {
expect(captured.headers['content-type']).toContain('image/svg+xml');
});
});
// ── app-icon rate limit ─────────────────────────────────────────────
describe('AppController app-icon rate limit', () => {
it('sizes the icon bucket for a whole network rather than one desktop', () => {
// Icons are unauthenticated `<img src>` targets, so the only key is
// the address — which one NAT shares across every desktop behind it,
// and each boot pulls the taskbar's icons in a burst.
for (const path of ['/app-icon/:app_uid', '/app-icon/:app_uid/:size']) {
const route = router.routes.find(
(r) => r.method === 'get' && r.path === path,
);
if (!route) throw new Error(`No GET ${path} route`);
expect(route.options.rateLimit).toEqual({
scope: 'app-icon',
limit: 12_000,
window: 60_000,
key: 'ip',
});
}
});
});
@@ -42,6 +42,7 @@ import type { TokenSource } from '../../core/http/types.js';
import { PuterServer } from '../../server.js';
import { FULL_API_ACCESS } from '../../services/permission/consts.js';
import { setupTestServer } from '../../testUtil.js';
import { FS_READ_LIMIT } from '../fs/limits.js';
// ── Test harness ────────────────────────────────────────────────────
@@ -1390,6 +1391,26 @@ describe('AuthController account-lifecycle route gating', () => {
expect(opts.requireUserActor).toBe(true);
});
// A fresh token is minted per protected mutation and nothing caches them,
// so issuance has to outrun the COMBINED rate of everything that spends
// one. The session-authenticated download path dominates — a multi-select
// download spends a token per file — with logout and the handful of
// session-management writes behind it.
it('GET /get-anticsrf-token clears the budgets that consume tokens', () => {
type Window = { limit: number; window: number };
const issuance = routeOptions('get', '/get-anticsrf-token')
.rateLimit as Window;
const logout = routeOptions('post', '/logout').rateLimit as Window;
expect(issuance.window).toBe(60_000);
expect(logout.window).toBe(60_000);
expect(FS_READ_LIMIT.window).toBe(60_000);
expect(issuance.limit).toBeGreaterThan(
FS_READ_LIMIT.limit + logout.limit,
);
});
it('requireUserActorGate rejects app-under-user and access-token actors', () => {
const gate = requireUserActorGate();
const run = (actor: Partial<Actor>) =>
@@ -6907,3 +6928,253 @@ describe('AuthController.loginWait audience binding', () => {
expect(res.body).toBeUndefined();
});
});
// -- Batch grant / revoke + cross-app data grants -----------------------
describe('AuthController — app-data grants', () => {
let issuer: { id: number; username: string };
let issuerActor: Actor;
beforeAll(async () => {
const name = `ad_${Math.random().toString(36).slice(2, 10)}`;
await controller.handleSignup(
makeReq({
username: name,
email: `${name}@test.local`,
password: 'correct-horse-battery',
}),
makeRes(),
);
const row = await server.stores.user.getByUsername(name);
await server.stores.user.update(row!.id, { email_confirmed: 1 });
issuer = { id: row!.id, username: row!.username };
issuerActor = {
user: {
id: row!.id,
uuid: row!.uuid,
username: row!.username,
email: row!.email,
email_confirmed: true,
},
} as Actor;
});
const makeAppRow = async (fields: Record<string, unknown> = {}) =>
(await server.stores.app.create(
{
name: `ad-${uuidv4()}`,
title: 'AppDataTest',
index_url: 'https://example.test/index.html',
...fields,
},
{ ownerUserId: issuer.id },
)) as { id: number; uid: string };
const grantedPermissions = async (appUid: string): Promise<string[]> => {
const rows = (await server.clients.db.read(
'SELECT p.`permission` FROM `user_to_app_permissions` p ' +
'JOIN `apps` a ON a.`id` = p.`app_id` ' +
'WHERE p.`user_id` = ? AND a.`uid` = ?',
[issuer.id, appUid],
)) as Array<{ permission: string }>;
return rows.map((r) => r.permission);
};
const post = (
handler: 'handleGrantUserApp' | 'handleRevokeUserApp',
body: Record<string, unknown>,
) => {
const res = makeRes();
return inCtx(issuerActor, () =>
controller[handler](makeReq(body, { actor: issuerActor }), res),
).then(() => res);
};
it('grants every permission in a list in one request', async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
const permissions = [
`app-data:${target.uid}:kv:read`,
`app-data:${target.uid}:kv:delete`,
];
await post('handleGrantUserApp', {
app_uid: grantee.uid,
permissions,
});
expect(await grantedPermissions(grantee.uid)).toEqual(
expect.arrayContaining(permissions),
);
});
it('revokes every permission in a list in one request', async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
const permissions = [
`app-data:${target.uid}:kv:read`,
`app-data:${target.uid}:fs:read`,
];
await post('handleGrantUserApp', {
app_uid: grantee.uid,
permissions,
});
await post('handleRevokeUserApp', {
app_uid: grantee.uid,
permissions,
});
const remaining = await grantedPermissions(grantee.uid);
for (const permission of permissions) {
expect(remaining).not.toContain(permission);
}
});
it('rejects the scalar and list forms together', async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permission: `app-data:${target.uid}:kv:read`,
permissions: [`app-data:${target.uid}:kv:write`],
}),
).rejects.toMatchObject({ statusCode: 400 });
});
it('rejects an empty, oversized, or `*`-bearing list', async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
for (const permissions of [
[],
Array.from(
{ length: 17 },
(_x, i) => `app-data:${target.uid}:kv:read${i}`,
),
[`app-data:${target.uid}:kv:read`, '*'],
]) {
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permissions,
}),
).rejects.toMatchObject({ statusCode: 400 });
}
});
it('writes nothing when one entry in the list is invalid', async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permissions: [
`app-data:${target.uid}:kv:read`,
`app-data:app-does-not-exist:kv:read`,
],
}),
).rejects.toMatchObject({ statusCode: 404 });
// The valid entry must not have landed: validation runs over the whole
// list before any row is written.
expect(await grantedPermissions(grantee.uid)).not.toContain(
`app-data:${target.uid}:kv:read`,
);
});
it('writes nothing when an entry is too wide for the column it lands in', async () => {
// `#validateAppPermissionParams` allows 4096 chars but the column is 255,
// so this passes shape validation and fails inside the grant. Before the
// pre-flight, the first entry committed and the caller still got a 400 —
// and the dialog reads a 4xx as "nothing was written".
const grantee = await makeAppRow();
const target = await makeAppRow();
const good = `app-data:${target.uid}:kv:read`;
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permissions: [good, 'x'.repeat(300)],
}),
).rejects.toMatchObject({ statusCode: 400 });
expect(await grantedPermissions(grantee.uid)).not.toContain(good);
});
it('404s when the target app does not exist', async () => {
const grantee = await makeAppRow();
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permission: 'app-data:app-nope/:kv:read',
}),
).rejects.toMatchObject({ statusCode: 404 });
});
it('rejects a bare `app-data` with no target app', async () => {
const grantee = await makeAppRow();
for (const permission of ['app-data', 'app-data:', 'app-data::kv']) {
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permission,
}),
).rejects.toMatchObject({ statusCode: 400 });
}
});
it('refuses a grant when the target opted out of sharing', async () => {
const grantee = await makeAppRow();
const closed = await makeAppRow({
metadata: JSON.stringify({ share_app_data: false }),
});
await expect(
post('handleGrantUserApp', {
app_uid: grantee.uid,
permission: `app-data:${closed.uid}:kv:read`,
}),
).rejects.toMatchObject({ statusCode: 403 });
});
it('still allows revoking a grant after the target opts out', async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
const permission = `app-data:${target.uid}:kv:read`;
await post('handleGrantUserApp', {
app_uid: grantee.uid,
permission,
});
await server.stores.app.update(target.id, {
metadata: JSON.stringify({ share_app_data: false }),
});
// A user must always be able to withdraw consent, whatever the target
// now says about sharing.
await post('handleRevokeUserApp', { app_uid: grantee.uid, permission });
expect(await grantedPermissions(grantee.uid)).not.toContain(permission);
});
it("creates the target's AppData directory for an fs grant", async () => {
const grantee = await makeAppRow();
const target = await makeAppRow();
const path = `/${issuer.username}/AppData/${target.uid}`;
expect(await server.stores.fsEntry.getEntryByPath(path)).toBeFalsy();
await post('handleGrantUserApp', {
app_uid: grantee.uid,
permission: `app-data:${target.uid}:fs:read`,
});
// Without this the grant is valid but every read 404s until the target
// app happens to run for the first time.
expect(await server.stores.fsEntry.getEntryByPath(path)).toBeTruthy();
});
it('leaves unrelated permissions untouched by the new validation', async () => {
const grantee = await makeAppRow();
const permission = 'service:unrelated:ii:read';
await post('handleGrantUserApp', {
app_uid: grantee.uid,
permission,
});
expect(await grantedPermissions(grantee.uid)).toContain(permission);
});
});
+354 -29
View File
@@ -27,6 +27,7 @@ import type { HttpErrorOptions } from '../../core/http/HttpError.js';
import { HttpError } from '../../core/http/HttpError.js';
import { antiCsrf } from '../../core/http/middleware/antiCsrf.js';
import { generateCaptcha } from '../../core/http/middleware/captcha.js';
import type { Actor } from '../../core/actor.js';
import { checkRateLimit } from '../../core/http/middleware/rateLimit.js';
import {
signStepUpToken,
@@ -60,11 +61,19 @@ import {
generateDefaultFsentries,
promoteToVerifiedGroup,
} from '../../util/userProvisioning.js';
import {
APP_DATA_PERMISSION_PREFIX,
appDataSharingAllowed,
parseAppDataPermission,
} from '../../services/permission/appDataScopes.js';
import { PuterController } from '../types.js';
const USERNAME_REGEX = /^\w{1,}$/;
const USERNAME_MAX_LENGTH = 45;
const FINGERPRINT_MAX_LENGTH = 128;
// One consent prompt covers a handful of scopes at most. The cap keeps a
// crafted request from turning a single grant call into a bulk write.
const MAX_PERMISSIONS_PER_REQUEST = 16;
const DISPATCH_ID_MAX_LENGTH = 128;
// Default SMS send attempts before the card fallback opens.
const DEFAULT_CARD_FALLBACK_ATTEMPTS = 2;
@@ -74,6 +83,94 @@ const DEFAULT_CARD_FALLBACK_ATTEMPTS = 2;
// crossed.
const SEND_PHONE_RATE_LIMIT = 10;
const SEND_PHONE_RATE_WINDOW_MS = 60 * 60_000;
// -- Post-login route limits -----------------------------------------
//
// The credential legs above (login, signup, recovery, confirmation) each
// carry their own limit. Everything a session can reach *after* signing
// in shares the four shapes below, keyed on the actor rather than the
// network — a per-account ceiling is the meaningful one once we know who
// is calling.
/**
* Mints or reconfigures a credential. Deliberately an hour-scale window: these
* are human actions taken a handful of times, and an unbounded rate turns one
* compromised session into a durable foothold.
*/
const CREDENTIAL_MINT_LIMIT = {
scope: 'auth-credential-mint',
limit: 20,
window: 60 * 60_000,
key: 'user',
} as const;
/**
* Second-factor configuration, including the verify leg. Shorter window than
* the mint limit because enabling 2FA legitimately involves a few attempts in a
* row, but unbounded verification is a TOTP brute force.
*/
const TWO_FACTOR_LIMIT = {
scope: 'auth-2fa-configure',
limit: 30,
window: 15 * 60_000,
key: 'user',
} as const;
/** Permission and membership writes. Never called in a loop by a client. */
const GRANT_LIMIT = {
scope: 'auth-grant',
limit: 60,
window: 60_000,
key: 'user',
} as const;
/**
* Read-only checks the GUI makes on nearly every interaction. The ceiling is
* high enough that only a runaway loop reaches it.
*/
const AUTH_CHECK_LIMIT = {
scope: 'auth-check',
limit: 300,
window: 60_000,
key: 'user',
} as const;
/**
* Anti-CSRF token issuance. Clients mint a fresh token per protected mutation
* and cache nothing, so this ceiling has to clear the SUM of the budgets that
* spend tokens — matching any single one of them guarantees the gate fires
* before the mutation it guards does.
*
* What spends them: the session-authenticated download path, one token per
* file, at the read budget of 600/min — a multi-selection download burns tokens
* exactly the way a bulk delete burns its own budget, so this tracks the bulk
* figure used for filesystem mutations; logout at 60/min; and the
* session-management writes (revoke, rename), which a person triggers a handful
* of times. Call it ~700/min of real demand, and leave enough on top that a
* bulk operation runs out of files before it runs out of tokens.
*/
const ANTI_CSRF_MINT_LIMIT = {
scope: 'anticsrf',
limit: 1200,
window: 60_000,
key: 'user',
} as const;
/** Settings-page reads — enumerating sessions, permissions, groups. */
const AUTH_LIST_LIMIT = {
scope: 'auth-list',
limit: 120,
window: 60_000,
key: 'user',
} as const;
/** Session plumbing: logout, GUI token, cookie sync. */
const SESSION_LIMIT = {
scope: 'auth-session',
limit: 60,
window: 60_000,
key: 'user',
} as const;
// Once the threshold is crossed the fallback stays open this long, so the
// user can finish the card flow without racing the attempt counter's expiry.
const CARD_FALLBACK_OPEN_TTL_SECONDS = 24 * 60 * 60;
@@ -1040,6 +1137,7 @@ export class AuthController extends PuterController {
requireUserActor: true,
allowUnconfirmed: true,
antiCsrf: true,
rateLimit: SESSION_LIMIT,
})
async handleLogout(req: Request, res: Response): Promise<void> {
// Clear the session cookie + `puter_token_v2`. Nothing issues the
@@ -2703,7 +2801,25 @@ export class AuthController extends PuterController {
// -- Captcha generation -------------------------------------------
@Get('/api/captcha/generate', { subdomain: '*' })
@Get('/api/captcha/generate', {
subdomain: '*',
// Unauthenticated, renders an image per call, and is the gate
// protecting /login and /signup — so bulk pre-generation is
// directly useful to an attacker. Per-fingerprint for fairness on
// shared IPs, plus a per-IP backstop against header rotation.
//
// The fingerprint bucket is the one sized for a person: a handful of
// refreshes while getting a captcha right. The IP bucket is not — one
// address is a whole office, campus or carrier gateway, and everyone
// behind it is signing in through the same counter, so sizing it for
// a browser would deny the captcha to a network rather than to an
// attacker. It stays wide enough for that population and narrow
// enough that header rotation still runs out.
rateLimit: [
{ scope: 'captcha', limit: 30, window: 60_000 },
{ scope: 'captcha-ip', limit: 3_000, window: 60_000, key: 'ip' },
],
})
async handleCaptchaGenerate(_req: Request, res: Response): Promise<void> {
const difficulty =
(this.config as { captcha?: { difficulty?: string } }).captcha
@@ -2715,6 +2831,7 @@ export class AuthController extends PuterController {
// -- Anti-CSRF token generation ----------------------------------
@Get('/get-anticsrf-token', {
rateLimit: ANTI_CSRF_MINT_LIMIT,
// Anti-CSRF tokens are only consumed by `requireUserActor` routes,
// so issuance is scoped to the same actor kind for consistency.
requireUserActor: true,
@@ -2735,6 +2852,7 @@ export class AuthController extends PuterController {
@Post('/auth/grant-user-user', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGrantUserUser(req: Request, res: Response): Promise<void> {
const { target_username, permission, extra, meta } = req.body;
@@ -2827,13 +2945,114 @@ export class AuthController extends PuterController {
return app.uid;
}
/**
* Resolve the `permission` / `permissions` pair into the list to act on.
*
* One consent prompt can cover several scopes (read a store, write
* another), and a client looping the single form would have to invent its
* own partial-failure and rollback handling. Accepting the array keeps that
* in one request.
*/
#appPermissionList(body: {
permission?: unknown;
permissions?: unknown;
}): string[] {
const { permission, permissions } = body;
if (permissions !== undefined && permissions !== null) {
if (permission !== undefined && permission !== null) {
throw new HttpError(
400,
'Pass `permission` or `permissions`, not both',
{ legacyCode: 'bad_request' },
);
}
if (!Array.isArray(permissions) || permissions.length === 0) {
throw new HttpError(400, 'Invalid `permissions`', {
legacyCode: 'bad_request',
});
}
if (permissions.length > MAX_PERMISSIONS_PER_REQUEST) {
throw new HttpError(400, 'Too many `permissions`', {
legacyCode: 'bad_request',
});
}
for (const entry of permissions) {
this.#validateAppPermissionParams({ permission: entry });
// `*` means "revoke everything" in the scalar form only —
// inside a list it would silently widen a targeted request.
if (!entry || entry === '*') {
throw new HttpError(400, 'Invalid `permissions`', {
legacyCode: 'bad_request',
});
}
}
return [...new Set(permissions as string[])];
}
return typeof permission === 'string' && permission ? [permission] : [];
}
/**
* Gate a cross-app data grant: the target must exist, must not have opted
* out of sharing, and must be named. Also creates the target's AppData
* directory for an `fs` scope, since it is only created lazily when the app
* first runs — without this a valid grant would 404 until then.
*/
async #prepareAppDataGrant(
actor: Actor,
permission: string,
): Promise<void> {
const parsed = parseAppDataPermission(permission);
if (!parsed) {
// A bare `app-data` (or one with an empty target) would cover every
// app the user has by prefix implication, which no prompt can
// describe. Reject rather than treat it as an unrelated permission.
if (
permission === APP_DATA_PERMISSION_PREFIX ||
permission.startsWith(`${APP_DATA_PERMISSION_PREFIX}:`)
) {
throw new HttpError(
400,
'Invalid `app-data` permission: missing target app',
{ legacyCode: 'bad_request' },
);
}
return;
}
const target = await this.stores.app.getByUid(parsed.targetAppUid);
if (!target) {
throw new HttpError(
404,
`entity_not_found: app:${parsed.targetAppUid}`,
{ legacyCode: 'subject_does_not_exist' },
);
}
if (!appDataSharingAllowed(target)) {
throw new HttpError(
403,
'This app does not share its data with other apps',
{ legacyCode: 'forbidden' },
);
}
const username = actor.user?.username;
const userId = actor.user?.id;
if ((parsed.store === 'fs' || !parsed.store) && username && userId) {
await this.services.fs.mkdir(userId, {
path: `/${username}/AppData/${parsed.targetAppUid}`,
createMissingParents: true,
} as never);
}
}
@Post('/auth/grant-user-app', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGrantUserApp(req: Request, res: Response): Promise<void> {
let { app_uid } = req.body;
const { origin, permission, extra, meta } = req.body;
const { origin, permission, permissions, extra, meta } = req.body;
this.#validateAppPermissionParams({
app_uid,
origin,
@@ -2841,27 +3060,43 @@ export class AuthController extends PuterController {
extra,
meta,
});
const list = this.#appPermissionList({ permission, permissions });
if (origin) {
app_uid = await this.#registeredAppUidFromOrigin(origin);
}
if (!app_uid || !permission) {
if (!app_uid || list.length === 0) {
throw new HttpError(400, 'Missing `app_uid` or `permission`', {
legacyCode: 'bad_request',
});
}
await this.services.permission.grantUserAppPermission(
req.actor!,
app_uid,
permission,
extra ?? undefined,
meta ?? undefined,
);
// Validate every entry before writing any, so a bad one in the list
// cannot leave a partially-granted set behind: the dialog reads a 4xx as
// "nothing was written" and skips its withdrawal, so a partial commit
// leaves live access the user was told they refused. The rewrite running
// twice is cheaper than splitting the grant into prepare/commit.
for (const entry of list) {
await this.services.permission.assertUserAppPermissionWritable(
entry,
);
await this.#prepareAppDataGrant(req.actor!, entry);
}
for (const entry of list) {
await this.services.permission.grantUserAppPermission(
req.actor!,
app_uid,
entry,
extra ?? undefined,
meta ?? undefined,
);
}
res.json({});
}
@Post('/auth/grant-user-group', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGrantUserGroup(req: Request, res: Response): Promise<void> {
const { group_uid, permission, extra, meta } = req.body;
@@ -2890,6 +3125,7 @@ export class AuthController extends PuterController {
@Post('/auth/revoke-user-user', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleRevokeUserUser(req: Request, res: Response): Promise<void> {
const { target_username, permission, meta } = req.body;
@@ -2912,24 +3148,28 @@ export class AuthController extends PuterController {
@Post('/auth/revoke-user-app', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleRevokeUserApp(req: Request, res: Response): Promise<void> {
let { app_uid } = req.body;
const { origin, permission, meta } = req.body;
const { origin, permission, permissions, meta } = req.body;
this.#validateAppPermissionParams({
app_uid,
origin,
permission,
meta,
});
const list = this.#appPermissionList({ permission, permissions });
if (origin) {
app_uid = await this.#registeredAppUidFromOrigin(origin);
}
if (!app_uid || !permission) {
if (!app_uid || list.length === 0) {
throw new HttpError(400, 'Missing `app_uid` or `permission`', {
legacyCode: 'bad_request',
});
}
// Deliberately not gated by the target's sharing flag: a user must
// always be able to withdraw a grant, whatever the target now says.
if (permission === '*') {
await this.services.permission.revokeUserAppAll(
req.actor!,
@@ -2937,12 +3177,14 @@ export class AuthController extends PuterController {
meta ?? undefined,
);
} else {
await this.services.permission.revokeUserAppPermission(
req.actor!,
app_uid,
permission,
meta ?? undefined,
);
for (const entry of list) {
await this.services.permission.revokeUserAppPermission(
req.actor!,
app_uid,
entry,
meta ?? undefined,
);
}
}
res.json({});
}
@@ -2950,6 +3192,7 @@ export class AuthController extends PuterController {
@Post('/auth/revoke-user-group', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleRevokeUserGroup(req: Request, res: Response): Promise<void> {
const { group_uid, permission, meta } = req.body;
@@ -2969,7 +3212,11 @@ export class AuthController extends PuterController {
// -- Permission checks -------------------------------------------
@Post('/auth/check-permissions', { subdomain: 'api', requireAuth: true })
@Post('/auth/check-permissions', {
subdomain: 'api',
requireAuth: true,
rateLimit: AUTH_CHECK_LIMIT,
})
async handleCheckPermissions(req: Request, res: Response): Promise<void> {
const { permissions } = req.body;
if (!Array.isArray(permissions)) {
@@ -2997,7 +3244,11 @@ export class AuthController extends PuterController {
// -- Session management ------------------------------------------
@Get('/auth/list-sessions', { subdomain: 'api', requireUserActor: true })
@Get('/auth/list-sessions', {
subdomain: 'api',
requireUserActor: true,
rateLimit: AUTH_LIST_LIMIT,
})
async handleListSessions(req: Request, res: Response): Promise<void> {
const sessions = await this.services.auth.listSessions(req.actor!);
res.json(sessions);
@@ -3079,7 +3330,11 @@ export class AuthController extends PuterController {
// -- Dev app permissions -----------------------------------------
@Post('/auth/grant-dev-app', { subdomain: 'api', requireUserActor: true })
@Post('/auth/grant-dev-app', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGrantDevApp(req: Request, res: Response): Promise<void> {
let { app_uid } = req.body;
const { origin, permission, extra, meta } = req.body;
@@ -3112,6 +3367,7 @@ export class AuthController extends PuterController {
@Post('/auth/revoke-dev-app', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleRevokeDevApp(req: Request, res: Response): Promise<void> {
let { app_uid } = req.body;
@@ -3147,6 +3403,7 @@ export class AuthController extends PuterController {
@Get('/auth/list-permissions', {
subdomain: 'api',
requireUserActor: true,
rateLimit: AUTH_LIST_LIMIT,
})
async handleListPermissions(req: Request, res: Response): Promise<void> {
const userId = req.actor!.user.id;
@@ -3211,7 +3468,11 @@ export class AuthController extends PuterController {
// -- App origin resolution ---------------------------------------
@Post('/auth/app-uid-from-origin', { subdomain: 'api', requireAuth: true })
@Post('/auth/app-uid-from-origin', {
subdomain: 'api',
requireAuth: true,
rateLimit: AUTH_CHECK_LIMIT,
})
async handleAppUidFromOrigin(req: Request, res: Response): Promise<void> {
const origin = req.body?.origin || req.query?.origin;
if (!origin)
@@ -3227,6 +3488,9 @@ export class AuthController extends PuterController {
@Post('/auth/get-user-app-token', {
subdomain: 'api',
requireUserActor: true,
// Called once per app launch, and the GUI can legitimately launch
// several in quick succession.
rateLimit: { ...AUTH_CHECK_LIMIT, scope: 'app-token', limit: 120 },
})
async handleGetUserAppToken(req: Request, res: Response): Promise<void> {
let { app_uid } = req.body;
@@ -3250,6 +3514,22 @@ export class AuthController extends PuterController {
app = await this.stores.app.createFromOrigin(app_uid, origin, {
ownerUserId,
});
// An origin's uid is a deterministic uuidv5, so a deleted app
// reappears here under the identical uid. Withdraw any cross-app
// data grants left pointing at it before this new row can inherit
// consent the user gave its predecessor. Only *this* path can reuse
// a uid: `AppStore.create` mints a random uuid4, which no deleted
// app can ever hold again.
//
// Called directly rather than through `app.changed`: the token is
// issued below, so this has to be able to stop that, and
// `emitAndWait` swallows listener errors. Letting it throw is the
// point — a sweep that failed leaves the old grants live against an
// app whoever controls the origin now has just claimed.
await this.services.appPermission.withdrawAppDataGrants(
app_uid,
'uid reused by a new app',
);
}
if (!app) {
throw new HttpError(404, `App ${app_uid} does not exist`, {
@@ -3329,7 +3609,11 @@ export class AuthController extends PuterController {
res.json({ token, app_uid });
}
@Post('/auth/check-app', { subdomain: 'api', requireUserActor: true })
@Post('/auth/check-app', {
subdomain: 'api',
requireUserActor: true,
rateLimit: AUTH_CHECK_LIMIT,
})
async handleCheckApp(req: Request, res: Response): Promise<void> {
let { app_uid } = req.body;
const { origin } = req.body;
@@ -3368,6 +3652,7 @@ export class AuthController extends PuterController {
@Post('/auth/create-access-token', {
subdomain: 'api',
requireAuth: true,
rateLimit: CREDENTIAL_MINT_LIMIT,
})
async handleCreateAccessToken(req: Request, res: Response): Promise<void> {
const { permissions, expiresIn, label } = req.body;
@@ -3436,6 +3721,7 @@ export class AuthController extends PuterController {
@Post('/auth/configure-2fa/:action', {
subdomain: 'api',
requireUserActor: true,
rateLimit: TWO_FACTOR_LIMIT,
})
async handleConfigure2fa(req: Request, res: Response): Promise<void> {
const action = req.params.action;
@@ -3566,7 +3852,11 @@ export class AuthController extends PuterController {
// -- Developer profile -------------------------------------------
@Get('/get-dev-profile', { subdomain: 'api', requireUserActor: true })
@Get('/get-dev-profile', {
subdomain: 'api',
requireUserActor: true,
rateLimit: AUTH_LIST_LIMIT,
})
async handleGetDevProfile(req: Request, res: Response): Promise<void> {
const user = await this.stores.user.getById(req.actor!.user.id!, {
force: true,
@@ -3596,7 +3886,13 @@ export class AuthController extends PuterController {
// -- Group management --------------------------------------------
@Post('/group/create', { subdomain: 'api', requireUserActor: true })
@Post('/group/create', {
subdomain: 'api',
requireUserActor: true,
// Creates a persistent row per call with no quota behind it, so it
// sits on the hour-scale budget rather than the grant one.
rateLimit: { ...CREDENTIAL_MINT_LIMIT, scope: 'group-create' },
})
async handleGroupCreate(req: Request, res: Response): Promise<void> {
const extra = req.body.extra ?? {};
const metadata = req.body.metadata ?? {};
@@ -3617,7 +3913,11 @@ export class AuthController extends PuterController {
res.json({ uid });
}
@Post('/group/add-users', { subdomain: 'api', requireUserActor: true })
@Post('/group/add-users', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGroupAddUsers(req: Request, res: Response): Promise<void> {
const { uid, users } = req.body ?? {};
if (!uid)
@@ -3649,7 +3949,11 @@ export class AuthController extends PuterController {
res.json({});
}
@Post('/group/remove-users', { subdomain: 'api', requireUserActor: true })
@Post('/group/remove-users', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGroupRemoveUsers(req: Request, res: Response): Promise<void> {
const { uid, users } = req.body ?? {};
if (!uid)
@@ -3681,7 +3985,11 @@ export class AuthController extends PuterController {
res.json({});
}
@Get('/group/list', { subdomain: 'api', requireUserActor: true })
@Get('/group/list', {
subdomain: 'api',
requireUserActor: true,
rateLimit: AUTH_LIST_LIMIT,
})
async handleGroupList(req: Request, res: Response): Promise<void> {
const userId = req.actor!.user.id!;
const [owned, member] = await Promise.all([
@@ -3694,7 +4002,22 @@ export class AuthController extends PuterController {
});
}
@Get('/group/public-groups', { subdomain: 'api' })
@Get('/group/public-groups', {
subdomain: 'api',
// The only unauthenticated route in the group set, so IP is the
// only key available — and that makes the bucket an aggregate:
// one office, campus or carrier gateway is a single key for
// everybody behind it, and each of them reads this once while
// bootstrapping. Sized for that population of real people rather
// than one browser, and no wider: this sits next to the sign-in
// surface, so it stays a real bound on enumeration.
rateLimit: {
scope: 'public-groups',
limit: 1_200,
window: 60_000,
key: 'ip',
},
})
async handleGroupPublicGroups(_req: Request, res: Response): Promise<void> {
res.json({
user: this.config.default_user_group ?? null,
@@ -3707,6 +4030,7 @@ export class AuthController extends PuterController {
@Get('/get-gui-token', {
requireUserActor: true,
allowUnconfirmed: true,
rateLimit: SESSION_LIMIT,
})
async handleGetGuiToken(req: Request, res: Response): Promise<void> {
if (!req.actor?.session?.uid)
@@ -3726,6 +4050,7 @@ export class AuthController extends PuterController {
}
@Get('/session/sync-cookie', {
rateLimit: SESSION_LIMIT,
// Installs the session cookie. Only page script on our own origin
// should be able to ask for that (the `tokenSource` check below is the
// companion rule: the token has to come from an Authorization header,
@@ -32,6 +32,19 @@ const ALLOWED_SORT_ORDER = ['asc', 'desc'];
* - User-level: desktop background, taskbar items (UserStore)
* - Folder-level: layout, sort_by/sort_order (fsentries table)
*/
/**
* Desktop preference writes — background, taskbar, layout, sort order. All four
* persist to the user row on every call, and the GUI fires them on direct user
* action, so a per-minute ceiling well above human speed is enough to catch a
* stuck client.
*/
const PREFERENCE_WRITE_LIMIT = {
scope: 'desktop-preference',
limit: 120,
window: 60_000,
key: 'user',
};
export class DesktopController extends PuterController {
constructor(config, clients, stores, services) {
super(config, clients, stores, services);
@@ -53,6 +66,7 @@ export class DesktopController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: PREFERENCE_WRITE_LIMIT,
},
async (req, res) => {
const { url, color, fit } = req.body ?? {};
@@ -108,6 +122,7 @@ export class DesktopController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: PREFERENCE_WRITE_LIMIT,
},
async (req, res) => {
const { items } = req.body ?? {};
@@ -134,6 +149,7 @@ export class DesktopController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: PREFERENCE_WRITE_LIMIT,
},
async (req, res) => {
const { item_uid, item_path, layout } = req.body ?? {};
@@ -161,6 +177,7 @@ export class DesktopController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: PREFERENCE_WRITE_LIMIT,
},
async (req, res) => {
const { item_uid, item_path, sort_by, sort_order } =
@@ -124,9 +124,9 @@ const makeSyntheticDriver = () => ({
const buildController = (driver: ReturnType<typeof makeSyntheticDriver>) => {
// The controller reads `this.services?.permission` only when an actor
// is on the request; otherwise the services bag is unused. The
// alarm client is invoked on the rate-limit / concurrency rejection
// paths, so stub a no-op `create`.
// is on the request; otherwise the services bag is unused. The rejection
// paths no longer alarm, but other paths still reach the client, so keep
// a no-op `create` stubbed.
const clients = { alarm: { create: () => {} } };
return new DriverController(
{} as any,
@@ -409,7 +409,7 @@ describe('DriverController per-method rate limiting', () => {
configureRateLimit({ disabled: false } as never);
});
it('answers 429 and raises a de-duped info alarm once the per-method budget is spent', async () => {
it('answers 429 without alarming once the per-method budget is spent', async () => {
const { handler, alarms, iface } = build({
run: () => ({ ok: true }),
rateLimit: { default: { limit: 1, window: 60_000 } },
@@ -430,9 +430,8 @@ describe('DriverController per-method rate limiting', () => {
legacyCode: 'too_many_requests',
});
expect(alarms).toContainEqual({
id: 'driver_rate_limit_hit:rate-limited-iface:run',
severity: 'info',
});
// Spending your own budget is the limit working as designed, so it
// must not raise anything — the 429 is the whole signal.
expect(alarms).toEqual([]);
});
});
@@ -158,11 +158,14 @@ interface MockRes {
sentBody: string | undefined;
contentType: string | undefined;
pipedFrom: Readable | undefined;
listeners: Record<string, Array<() => void>>;
status(code: number): MockRes;
json(body: unknown): MockRes;
setHeader(key: string, value: string): MockRes;
type(t: string): MockRes;
send(body: string): MockRes;
once(event: string, fn: () => void): MockRes;
emit(event: string): void;
}
const makeRes = (): MockRes => {
const res: MockRes = {
@@ -172,6 +175,19 @@ const makeRes = (): MockRes => {
sentBody: undefined,
contentType: undefined,
pipedFrom: undefined,
// `#handleCall` releases a driver's concurrency slot on `finish` /
// `close`, so the stub has to behave like an emitter for any driver
// that declares a `concurrent` policy.
listeners: {},
once(event: string, fn: () => void) {
(this.listeners[event] ??= []).push(fn);
return this;
},
emit(event: string) {
const fns = this.listeners[event] ?? [];
this.listeners[event] = [];
for (const fn of fns) fn();
},
status(code: number) {
this.statusCode = code;
return this;
@@ -196,6 +212,30 @@ const makeRes = (): MockRes => {
return res;
};
/**
* Run the call route and return the rejection, or null when it resolves.
* Lets a test assert on *which* gate rejected without depending on how far
* an admitted call gets afterwards.
*/
const callForError = async (
routes: Captured,
req: Request,
actor: Actor,
): Promise<{ statusCode?: number; legacyCode?: string } | null> => {
try {
await runWithContext({ actor }, () =>
routes['POST /call'](
req,
makeRes() as unknown as Response,
() => {},
),
);
return null;
} catch (e) {
return e as { statusCode?: number; legacyCode?: string };
}
};
const makeReq = (body: Record<string, unknown> = {}, actor?: Actor): Request =>
({
body,
@@ -322,8 +362,7 @@ describe('DriverController.#handleCall (via captured router)', () => {
it('admits a bare session ("root" token) actor on the AI drivers', async () => {
// Privileged ("godmode") apps call AI with the user's own session
// token, so the AI drivers no longer set `noUserSession`: the
// session actor reaches the permission scan (403 forbidden — no
// perms granted) instead of the credential-shape rejection.
// session actor must get past the credential-shape gate.
const actor = await makeUserActor();
const req = makeReq(
{
@@ -333,18 +372,8 @@ describe('DriverController.#handleCall (via captured router)', () => {
},
actor,
);
await expect(
runWithContext({ actor }, () =>
routes['POST /call'](
req,
makeRes() as unknown as Response,
() => {},
),
),
).rejects.toMatchObject({
statusCode: 403,
legacyCode: 'forbidden',
});
const err = await callForError(routes, req, actor);
expect(err?.legacyCode).not.toBe('app_or_api_token_required');
});
it('admits app and access-token actors on the AI drivers (they fail later on permission, not credential shape)', async () => {
@@ -369,29 +398,15 @@ describe('DriverController.#handleCall (via captured router)', () => {
},
actor,
);
await expect(
runWithContext({ actor }, () =>
routes['POST /call'](
req,
makeRes() as unknown as Response,
() => {},
),
),
).rejects.toMatchObject({
statusCode: 403,
// Fresh actors hold no service:* perms, so the call still
// 403s — but at the permission scan, proving the
// credential-shape gate let the delegated actor through.
legacyCode: 'forbidden',
});
const err = await callForError(routes, req, actor);
expect(err?.legacyCode).not.toBe('app_or_api_token_required');
}
});
it('admits a user-scoped worker session on the AI drivers', async () => {
// Workers deployed without an app binding authenticate as a user
// actor whose session row is kind='worker' — they must keep their
// AI access (they fail later on permission here, not on
// credential shape).
// AI access rather than be rejected for their credential shape.
const base = await makeUserActor();
const actor: Actor = {
...base,
@@ -405,24 +420,13 @@ describe('DriverController.#handleCall (via captured router)', () => {
},
actor,
);
await expect(
runWithContext({ actor }, () =>
routes['POST /call'](
req,
makeRes() as unknown as Response,
() => {},
),
),
).rejects.toMatchObject({
statusCode: 403,
legacyCode: 'forbidden',
});
const err = await callForError(routes, req, actor);
expect(err?.legacyCode).not.toBe('app_or_api_token_required');
});
it('still allows bare session actors on drivers without the noUserSession flag', async () => {
// puter-kvstore doesn't set the flag; the session actor reaches
// the permission scan (403 forbidden — no perms granted) instead
// of being rejected for its credential shape.
// puter-kvstore doesn't set the flag, so the session actor must get
// past the credential-shape gate.
const actor = await makeUserActor();
const req = makeReq(
{
@@ -432,24 +436,24 @@ describe('DriverController.#handleCall (via captured router)', () => {
},
actor,
);
await expect(
runWithContext({ actor }, () =>
routes['POST /call'](
req,
makeRes() as unknown as Response,
() => {},
),
),
).rejects.toMatchObject({
statusCode: 403,
legacyCode: 'forbidden',
});
const err = await callForError(routes, req, actor);
expect(err?.legacyCode).not.toBe('app_or_api_token_required');
});
it('rejects with 403 when the actor lacks the service permission', async () => {
// A fresh user has no service:* perms → permission.check returns
// false → 403.
const actor = await makeUserActor();
// A scoped access token holds only the permissions written to its
// own row, so it is the actor shape that can still be denied — the
// `driver`/`service` grants every user inherits are resolved from
// the issuer only for full-access tokens.
const base = await makeUserActor();
const actor: Actor = {
...base,
accessToken: {
uid: `tok-${uuidv4()}`,
issuer: base,
fullAccess: false,
},
};
const req = makeReq(
{
interface: 'puter-kvstore',
@@ -458,15 +462,8 @@ describe('DriverController.#handleCall (via captured router)', () => {
},
actor,
);
await expect(
runWithContext({ actor }, () =>
routes['POST /call'](
req,
makeRes() as unknown as Response,
() => {},
),
),
).rejects.toMatchObject({ statusCode: 403 });
const err = await callForError(routes, req, actor);
expect(err).toMatchObject({ statusCode: 403 });
});
it('returns the wrapped {success, result, service} envelope on a successful call', async () => {
@@ -45,6 +45,27 @@ import { PuterController } from '../types.js';
type DriverInstance = WithLifecycle & Record<string, unknown>;
/**
* Coarse envelope over the whole `/call` surface, so that spreading calls
* across many interfaces can't dodge every individual bucket. Per-driver limits
* are what actually shape traffic.
*
* "Coarse" is a constraint, not a description: for this to be an envelope it
* has to sit _above_ every per-driver budget, or it silently becomes the real
* limit for the widest ones and overrides the tier policy they declare.
* `driverPolicies.test.ts` asserts that ordering against every registered
* driver, so raising a driver's budget past this number fails there rather than
* in production. The headroom above the widest driver (notifications, at
* 3000/30s) is what leaves room for one caller to be busy on two interfaces at
* once.
*/
export const DRIVERS_CALL_LIMIT = {
scope: 'drivers-call',
limit: 8000,
window: 60_000,
key: 'user' as const,
};
// Every driver call is timed here already, for the lifecycle events below.
// Recording the same number as a histogram makes the per-interface latency
// distribution available downstream; which interfaces are worth keeping is a
@@ -180,12 +201,26 @@ export class DriverController extends PuterController {
registerRoutes(router: PuterRouter): void {
router.post(
'/call',
{ subdomain: 'api', requireAuth: true },
{
subdomain: 'api',
requireAuth: true,
rateLimit: DRIVERS_CALL_LIMIT,
},
this.#handleCall,
);
router.get(
'/list-interfaces',
{ subdomain: 'api', requireAuth: true },
{
subdomain: 'api',
requireAuth: true,
// Static introspection output, read once at boot.
rateLimit: {
scope: 'drivers-list-interfaces',
limit: 60,
window: 60_000,
key: 'user',
},
},
this.#handleListInterfaces,
);
}
@@ -262,8 +297,7 @@ export class DriverController extends PuterController {
if (req.actor) {
const permService = this.services.permission as unknown as
| PermissionService
| undefined;
PermissionService | undefined;
if (permService) {
// Build via PermissionUtil.join so any `:` in a driver or
// interface name is escaped — raw interpolation would let a
@@ -305,19 +339,8 @@ export class DriverController extends PuterController {
if (
!(await checkDriverRateLimit(req, ifaceName, method, rateLimitSpec))
) {
// De-dupe on (iface, method) so a hot loop across many users
// aggregates as occurrences on a single low-severity alarm
// instead of fanning out one per user.
this.clients.alarm.create(
`driver_rate_limit_hit:${ifaceName}:${method}`,
`Driver rate limit hit on ${ifaceName}:${method}`,
{
iface: ifaceName,
method,
userUuid: req.actor?.user?.uuid,
},
'info',
);
// Deliberately unalarmed: a caller spending its own budget is
// the limit working, not an incident. The 429 is the signal.
throw new HttpError(429, 'Too many requests.', {
legacyCode: 'too_many_requests',
});
@@ -340,16 +363,8 @@ export class DriverController extends PuterController {
concurrentSpec,
);
if (!handle.ok) {
this.clients.alarm.create(
`driver_concurrent_limit_hit:${ifaceName}:${method}`,
`Driver concurrency limit hit on ${ifaceName}:${method}`,
{
iface: ifaceName,
method,
userUuid: req.actor?.user?.uuid,
},
'info',
);
// Unalarmed for the same reason as the rate-limit rejection
// above: hitting a declared cap is the cap doing its job.
throw new HttpError(429, 'Too many concurrent requests.', {
legacyCode: 'too_many_requests',
});
@@ -0,0 +1,594 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, RequestHandler, Response } from 'express';
import {
afterAll,
afterEach,
beforeAll,
describe,
expect,
it,
vi,
} from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import type { Actor } from '../../core/actor.js';
import { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterServer } from '../../server.js';
import { AppFeedbackService } from '../../services/feedback/AppFeedbackService.js';
import { setupTestServer } from '../../testUtil.js';
// Boots one real PuterServer (in-memory sqlite + mocked externals) and
// registers AppFeedbackController's decorated routes onto a fresh
// PuterRouter. Tests drive the captured handlers with stub req/res; the
// stores/services underneath are the live wired ones, so rows land in the
// real `app_feedback` table.
let server: PuterServer;
let router: PuterRouter;
beforeAll(async () => {
server = await setupTestServer();
router = new PuterRouter();
server.controllers.appFeedback.registerRoutes(router);
});
afterAll(async () => {
await server?.shutdown();
});
afterEach(() => {
vi.restoreAllMocks();
});
const makeUser = async (): Promise<{ actor: Actor; userId: number }> => {
const username = `fdbk-${Math.random().toString(36).slice(2, 10)}`;
const created = await server.stores.user.create({
username,
uuid: uuidv4(),
password: null,
email: `${username}@test.local`,
free_storage: 100 * 1024 * 1024,
requires_email_confirmation: false,
});
const refreshed = (await server.stores.user.getById(created.id))!;
return {
userId: refreshed.id,
actor: {
user: {
id: refreshed.id,
uuid: refreshed.uuid,
username: refreshed.username,
email: refreshed.email ?? null,
email_confirmed: true,
} as Actor['user'],
},
};
};
const makeApp = async (
ownerUserId: number,
opts: { feedbackEnabled?: boolean; indexUrl?: string; name?: string } = {},
) => {
const name =
opts.name ?? `fdbk-app-${Math.random().toString(36).slice(2, 10)}`;
return await server.stores.app.create(
{
name,
title: `Feedback Test ${name}`,
index_url: opts.indexUrl ?? `https://${name}.example.com`,
...(opts.feedbackEnabled ? { feedback_enabled: 1 } : {}),
},
{ ownerUserId },
);
};
// Feedback is only offered when the deployment can deliver it (email
// transport configured); most tests want that baseline without asserting
// anything about the mail itself.
const mockEmailConfigured = () =>
vi.spyOn(server.clients.email, 'isConfigured', 'get').mockReturnValue(
true,
);
const confirmOwnerEmail = async (userId: number) => {
await server.clients.db.write(
'UPDATE `user` SET `email_confirmed` = ? WHERE `id` = ?',
[server.clients.db.booleanValue(true), userId],
);
const user = await server.stores.user.getById(userId);
if (user) await server.stores.user.invalidate(user);
};
const makeReq = (init: {
body?: unknown;
actor?: Actor;
query?: Record<string, unknown>;
}): Request => {
return {
body: init.body ?? {},
query: init.query ?? {},
headers: {},
actor: init.actor,
} as unknown as Request;
};
const makeRes = () => {
const captured: { statusCode: number; body: unknown } = {
statusCode: 200,
body: undefined,
};
const res = {
json: vi.fn((value: unknown) => {
captured.body = value;
return res;
}),
status: vi.fn((code: number) => {
captured.statusCode = code;
return res;
}),
};
return { res: res as unknown as Response, captured };
};
const findRoute = (method: string, path: string) => {
const route = router.routes.find(
(r) => r.method === method && r.path === path,
);
if (!route) throw new Error(`No ${method.toUpperCase()} ${path} route`);
return route;
};
const callRoute = async (
method: string,
path: string,
req: Request,
res: Response,
) => {
const handler: RequestHandler = findRoute(method, path).handler;
await handler(req, res, () => {
throw new Error('handler called next() unexpectedly');
});
};
const submit = (actor: Actor, body: unknown) => {
const { res, captured } = makeRes();
return callRoute('post', '/', makeReq({ body, actor }), res).then(
() => captured,
);
};
// ── Route gates ─────────────────────────────────────────────────────
describe('AppFeedbackController route options', () => {
it('rejects app actors and cross-origin pages on submit', () => {
const { options } = findRoute('post', '/');
// requireUserActor is what makes feedback impossible to submit
// programmatically with an app token; guiOriginOnly keeps
// cross-origin browser pages out even with a leaked user token.
expect(options.requireUserActor).toBe(true);
expect(options.guiOriginOnly).toBe(true);
});
it('stacks a per-user budget with a per-IP backstop', () => {
const { options } = findRoute('post', '/');
const limits = options.rateLimit;
expect(Array.isArray(limits)).toBe(true);
const keys = (limits as Array<{ key?: unknown }>).map((l) => l.key);
expect(keys).toContain('user');
expect(keys).toContain('ip');
});
it('requires a user actor on the target pre-flight too', () => {
const { options } = findRoute('get', '/target');
expect(options.requireUserActor).toBe(true);
});
});
// ── GET /app-feedback/target ────────────────────────────────────────
describe('AppFeedbackController GET /target', () => {
it('throws 400 when neither or both of app/origin are given', async () => {
const { actor } = await makeUser();
for (const query of [
{},
{ app: 'x', origin: 'https://x.example.com' },
]) {
const { res } = makeRes();
await expect(
callRoute('get', '/target', makeReq({ query, actor }), res),
).rejects.toMatchObject({ statusCode: 400 });
}
});
it('reports enabled:false for an unknown app', async () => {
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({ query: { app: 'no-such-app-xyz' }, actor }),
res,
);
expect(captured.body).toEqual({ enabled: false, app: null });
});
it('reports enabled:false for an app that has not opted in', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId);
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({ query: { app: app.name }, actor }),
res,
);
expect(captured.body).toMatchObject({ enabled: false });
});
it('reports enabled:true with canonical title/name for an opted-in app', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({ query: { app: app.uid }, actor }),
res,
);
expect(captured.body).toEqual({
enabled: true,
app: { name: app.name, title: app.title },
});
});
it('resolves an opted-in app whose name starts with "app-"', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const name = `app-fdbk-${Math.random().toString(36).slice(2, 10)}`;
const app = await makeApp(ownerId, { feedbackEnabled: true, name });
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({ query: { app: name }, actor }),
res,
);
expect(captured.body).toEqual({
enabled: true,
app: { name: app.name, title: app.title },
});
});
it('resolves an origin to the app whose index_url it matches', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const origin = new URL(app.index_url).origin;
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({ query: { origin }, actor }),
res,
);
expect(captured.body).toEqual({
enabled: true,
app: { name: app.name, title: app.title },
});
});
it('reports enabled:false when the email transport is unconfigured', async () => {
// No mockEmailConfigured(): this is the self-hosted no-SMTP default.
// Feedback that can never be delivered must not be solicited.
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({ query: { app: app.name }, actor }),
res,
);
expect(captured.body).toMatchObject({ enabled: false });
});
it('reports enabled:false for an origin with no registered app', async () => {
const { actor } = await makeUser();
const { res, captured } = makeRes();
await callRoute(
'get',
'/target',
makeReq({
query: { origin: 'https://nobody-registered.example.com' },
actor,
}),
res,
);
expect(captured.body).toEqual({ enabled: false, app: null });
});
});
// ── POST /app-feedback ──────────────────────────────────────────────
describe('AppFeedbackController POST /', () => {
it('throws 400 when message is missing or not a string', async () => {
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor } = await makeUser();
for (const message of [undefined, 12345, '']) {
await expect(
submit(actor, { app: app.name, message }),
).rejects.toMatchObject({ statusCode: 400 });
}
});
it('throws 400 when both app and origin are given', async () => {
const { actor } = await makeUser();
await expect(
submit(actor, {
app: 'x',
origin: 'https://x.example.com',
message: 'hi',
}),
).rejects.toMatchObject({ statusCode: 400 });
});
it('throws 400 when the origin exceeds the stored column size', async () => {
// source_origin is VARCHAR(2048) on MySQL/Postgres; a longer origin
// must be rejected up front, not fail (or silently truncate) at the
// INSERT after passing every other validation.
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const origin = `${new URL(app.index_url).origin}/${'x'.repeat(2500)}`;
const { actor } = await makeUser();
await expect(
submit(actor, { origin, message: 'hi' }),
).rejects.toMatchObject({ statusCode: 400 });
});
it('throws 403 feedback_not_enabled when the app has not opted in', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId);
const { actor } = await makeUser();
await expect(
submit(actor, { app: app.name, message: 'hi there' }),
).rejects.toMatchObject({
statusCode: 403,
legacyCode: 'feedback_not_enabled',
});
});
it('throws 403 for an unknown app and an unknown origin alike', async () => {
const { actor } = await makeUser();
await expect(
submit(actor, { app: 'no-such-app-xyz', message: 'hi' }),
).rejects.toMatchObject({ statusCode: 403 });
await expect(
submit(actor, {
origin: 'https://nobody-registered.example.com',
message: 'hi',
}),
).rejects.toMatchObject({ statusCode: 403 });
});
it('throws 403 feedback_not_enabled when the email transport is unconfigured', async () => {
// No mockEmailConfigured(): the opted-in app must still refuse — a
// stored row nothing can read, sold to the sender as delivered, is
// worse than an honest refusal.
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor } = await makeUser();
await expect(
submit(actor, { app: app.name, message: 'into the void' }),
).rejects.toMatchObject({
statusCode: 403,
legacyCode: 'feedback_not_enabled',
});
});
it('throws 400 when the message exceeds the length limit', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor } = await makeUser();
await expect(
submit(actor, {
app: app.name,
message: 'x'.repeat(
AppFeedbackService.MESSAGE_MAX_LENGTH + 1,
),
}),
).rejects.toMatchObject({ statusCode: 400 });
});
it('stores a normalized row and responds with an empty object', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor, userId } = await makeUser();
const captured = await submit(actor, {
app: app.uid,
message: ' Great\r\napp! ',
context: 'app',
});
expect(captured.body).toEqual({});
const rows = (await server.clients.db.read(
'SELECT * FROM `app_feedback` WHERE `user_id` = ?',
[userId],
)) as Array<Record<string, unknown>>;
expect(rows).toHaveLength(1);
expect(rows[0]).toMatchObject({
app_uid: app.uid,
message: 'Great\napp!',
source_env: 'app',
source_origin: null,
});
// Engine-agnostic reads: pg returns BIGINT as string and BOOLEAN as
// boolean, sqlite returns numbers for both.
expect(Number(rows[0].app_id)).toBe(app.id);
expect(Boolean(rows[0].email_sent)).toBe(false);
expect(typeof rows[0].uid).toBe('string');
expect(Number.isFinite(Number(rows[0].created_at))).toBe(true);
});
it('records the attested origin on web submissions', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const origin = new URL(app.index_url).origin;
const { actor, userId } = await makeUser();
await submit(actor, { origin, message: 'from the web', context: 'web' });
const rows = (await server.clients.db.read(
'SELECT `source_env`, `source_origin` FROM `app_feedback` WHERE `user_id` = ?',
[userId],
)) as Array<Record<string, unknown>>;
expect(rows).toHaveLength(1);
expect(rows[0]).toMatchObject({
source_env: 'web',
source_origin: origin,
});
});
it('enforces the per-user-per-app daily cap with 429', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor, userId } = await makeUser();
const since = Math.floor(Date.now() / 1000);
for (let i = 0; i < AppFeedbackService.PER_USER_APP_DAILY_LIMIT; i++) {
await server.stores.appFeedback.create({
appId: app.id,
appUid: app.uid,
userId,
message: `seed ${i}`,
});
}
expect(
await server.stores.appFeedback.countByUserAndAppSince(
userId,
app.id,
since - 60,
),
).toBe(AppFeedbackService.PER_USER_APP_DAILY_LIMIT);
await expect(
submit(actor, { app: app.name, message: 'one too many' }),
).rejects.toMatchObject({
statusCode: 429,
legacyCode: 'too_many_requests',
});
});
it('rolls back the stored row when a concurrent burst breaches the cap', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor, userId } = await makeUser();
for (let i = 0; i < AppFeedbackService.PER_USER_APP_DAILY_LIMIT; i++) {
await server.stores.appFeedback.create({
appId: app.id,
appUid: app.uid,
userId,
message: `seed ${i}`,
});
}
// Simulate the losing side of the check-then-insert race: the
// pre-insert check reads a stale under-cap count; the post-insert
// recount (real implementation) sees the truth.
vi.spyOn(
server.stores.appFeedback,
'countByUserAndAppSince',
).mockResolvedValueOnce(0);
await expect(
submit(actor, { app: app.name, message: 'raced past the cap' }),
).rejects.toMatchObject({
statusCode: 429,
legacyCode: 'too_many_requests',
});
const rows = (await server.clients.db.read(
'SELECT COUNT(*) AS n FROM `app_feedback` WHERE `user_id` = ?',
[userId],
)) as Array<{ n: unknown }>;
expect(Number(rows[0]?.n)).toBe(
AppFeedbackService.PER_USER_APP_DAILY_LIMIT,
);
});
it('enforces the per-user daily cap across apps with 429', async () => {
mockEmailConfigured();
const { userId: ownerId } = await makeUser();
const target = await makeApp(ownerId, { feedbackEnabled: true });
const other = await makeApp(ownerId, { feedbackEnabled: true });
const { actor, userId } = await makeUser();
for (let i = 0; i < AppFeedbackService.PER_USER_DAILY_LIMIT; i++) {
await server.stores.appFeedback.create({
appId: other.id,
appUid: other.uid,
userId,
message: `seed ${i}`,
});
}
await expect(
submit(actor, { app: target.name, message: 'over the limit' }),
).rejects.toMatchObject({ statusCode: 429 });
});
});
// ── Owner email delivery ────────────────────────────────────────────
// Which submissions get emailed, and what the mail contains, is service
// logic covered in AppFeedbackService.test.ts. What the controller owes the
// caller is that mail trouble never becomes the sender's problem.
describe('AppFeedbackController owner email', () => {
it('a failing email send never fails the request', async () => {
mockEmailConfigured();
vi.spyOn(server.clients.email, 'send').mockRejectedValue(
new Error('smtp down'),
);
const { userId: ownerId } = await makeUser();
await confirmOwnerEmail(ownerId);
const app = await makeApp(ownerId, { feedbackEnabled: true });
const { actor, userId } = await makeUser();
const captured = await submit(actor, {
app: app.name,
message: 'still stored',
});
expect(captured.body).toEqual({});
const rows = (await server.clients.db.read(
'SELECT `email_sent` FROM `app_feedback` WHERE `user_id` = ?',
[userId],
)) as Array<{ email_sent: unknown }>;
expect(rows).toHaveLength(1);
expect(Boolean(rows[0]?.email_sent)).toBe(false);
});
});
@@ -0,0 +1,169 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, Response } from 'express';
import { Controller, Get, Post } from '../../core/http/decorators.js';
import { HttpError } from '../../core/http/HttpError.js';
import { AppFeedbackService } from '../../services/feedback/AppFeedbackService.js';
import { PuterController } from '../types.js';
/**
* Endpoints behind the "send feedback to this app's developer" dialog
* (`puter.ui.showFeedbackDialog()`). Only the GUI (desktop dialog or the
* puter.com popup) calls these; apps cannot — both routes reject app actors,
* which is what makes feedback impossible to submit programmatically on a
* user's behalf.
*
* The target may be named either by `app` (uid or name — the desktop knows
* which app asked) or by `origin` (external site — the popup passes its
* browser-attested opener origin). Exactly one must be provided.
*/
/** Sanity cap on the raw body field; the service enforces the real limit. */
const RAW_MESSAGE_CAP = 50_000;
// Upper bound on the `app`/`origin` target params. Must not exceed the
// `source_origin` column (VARCHAR(2048) on MySQL/Postgres): the raw origin is
// stored verbatim, and a longer value would fail the INSERT after passing
// every validation — or be silently truncated on non-strict MySQL.
const TARGET_PARAM_MAX_LENGTH = 2048;
const readTargetParam = (value: unknown): string | undefined => {
return typeof value === 'string' &&
value.length > 0 &&
value.length <= TARGET_PARAM_MAX_LENGTH
? value
: undefined;
};
@Controller('/app-feedback')
export class AppFeedbackController extends PuterController {
/**
* GET /app-feedback/target — pre-flight for the dialog: whether the target
* app accepts feedback, plus its canonical title/name for display. Reveals
* nothing that `puter.apps.get` doesn't already.
*/
@Get('/target', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: {
scope: 'app-feedback-target',
limit: 60,
window: 60_000,
key: 'user',
},
})
async target(req: Request, res: Response): Promise<void> {
const app = readTargetParam(req.query.app);
const origin = readTargetParam(req.query.origin);
if (!app === !origin) {
throw new HttpError(
400,
'Exactly one of `app` and `origin` is required',
{ legacyCode: 'bad_request' },
);
}
const service = this.services.appFeedback as AppFeedbackService;
res.json(await service.getTarget({ app, origin }));
}
/**
* POST /app-feedback — store one feedback message and email the app's
* developer. Strict limits: the route limits below are the cheap first
* line; AppFeedbackService enforces durable per-user/per-app daily caps
* from the database (the route limiter fails open, the DB caps don't).
*/
@Post('/', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
// Submissions only ever originate from our own GUI pages (desktop
// dialog / popup). Cross-origin browser pages get stopped here even
// if they somehow hold a user token; non-browser clients still pass
// and are handled by the caps.
guiOriginOnly: true,
rateLimit: [
{
scope: 'app-feedback-user',
limit: 5,
window: 30 * 60_000,
key: 'user',
},
// IP backstop so freshly minted accounts can't stack per-user
// budgets from one machine.
{
scope: 'app-feedback-ip',
limit: 30,
window: 24 * 60 * 60_000,
key: 'ip',
},
],
})
async submit(req: Request, res: Response): Promise<void> {
const body = req.body ?? {};
const app = readTargetParam(body.app);
const origin = readTargetParam(body.origin);
if (!app === !origin) {
throw new HttpError(
400,
'Exactly one of `app` and `origin` is required',
{ legacyCode: 'bad_request' },
);
}
const message = body.message;
if (typeof message !== 'string' || message.length === 0) {
throw new HttpError(400, '`message` is required', {
legacyCode: 'bad_request',
});
}
if (message.length > RAW_MESSAGE_CAP) {
throw new HttpError(
400,
`\`message\` is too long (max ${AppFeedbackService.MESSAGE_MAX_LENGTH} characters)`,
{ legacyCode: 'bad_request' },
);
}
const sourceEnv =
body.context === 'app' || body.context === 'web'
? body.context
: undefined;
const userId = req.actor?.user?.id;
if (!userId) {
throw new HttpError(401, 'Unauthorized', {
legacyCode: 'unauthorized',
});
}
const service = this.services.appFeedback as AppFeedbackService;
await service.submit({
userId,
app,
origin,
message,
sourceEnv,
sourceOrigin: origin ?? null,
});
res.json({});
}
}
+14 -11
View File
@@ -21,7 +21,7 @@ import type { Request, Response } from 'express';
import type { Readable } from 'node:stream';
import { v4 as uuidv4 } from 'uuid';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import type { Actor } from '../../core/actor.js';
import { makeActor, type Actor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
@@ -927,7 +927,7 @@ describe('FSController.searchEntries', () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appUid = `app-search-${uuidv4()}`;
const appActor: Actor = { ...userActor, app: { uid: appUid } };
const appActor = makeActor({ ...userActor, app: { uid: appUid } });
const needle = `appneedle-${Math.random().toString(36).slice(2, 8)}`;
// User-owned entry outside AppData — must NOT appear for the app.
@@ -978,7 +978,7 @@ describe('FSController.searchEntries', () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appUid = `app-search-${uuidv4()}`;
const appActor: Actor = { ...userActor, app: { uid: appUid } };
const appActor = makeActor({ ...userActor, app: { uid: appUid } });
const needle = `appneedle-${Math.random().toString(36).slice(2, 8)}`;
// Only seed outside AppData — the app must not be able to find it.
@@ -1127,7 +1127,7 @@ describe('FSController.mkdirEntry', () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appUid = `app-mkdir-${uuidv4()}`;
const appActor: Actor = { ...userActor, app: { uid: appUid } };
const appActor = makeActor({ ...userActor, app: { uid: appUid } });
const target = `/${username}/AppData/${appUid}`;
await withActor(userActor, () =>
@@ -1889,10 +1889,10 @@ describe('FSController.readdirEntries recursive', () => {
it('masks denials for app-under-user actors as a 404 (legacy parity)', async () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appActor: Actor = {
const appActor = makeActor({
...userActor,
app: { uid: `app-readdir-${uuidv4()}` },
};
});
// The user's Documents is outside the app's AppData subtree, so the
// app can't list it. Legacy `/readdir` masks this as a 404
// subject_does_not_exist rather than leaking a 403.
@@ -2372,15 +2372,18 @@ describe('FSController.statEntry additional branches', () => {
// ── #getReportedCosts ───────────────────────────────────────────────
describe('FSController.getReportedCosts', () => {
it('mirrors every FS_COSTS entry as a per-byte line item', async () => {
const { FS_COSTS } = await import('./costs.js');
it('mirrors every storage-operation price as a per-operation line item', async () => {
const { STORAGE_OP_COSTS } =
await import('../../services/metering/costs.js');
const reported = controller.getReportedCosts();
expect(reported.length).toBe(Object.keys(FS_COSTS).length);
for (const [usageType, ucentsPerUnit] of Object.entries(FS_COSTS)) {
expect(reported.length).toBe(Object.keys(STORAGE_OP_COSTS).length);
for (const [usageType, ucentsPerUnit] of Object.entries(
STORAGE_OP_COSTS,
)) {
expect(reported).toContainEqual({
usageType,
ucentsPerUnit,
unit: 'byte',
unit: 'operation',
source: 'controller:fs',
});
}
+143 -57
View File
@@ -22,7 +22,6 @@ import type { Request, Response } from 'express';
import { posix as pathPosix } from 'node:path';
import { pipeline } from 'node:stream/promises';
import type { Actor } from '../../core/actor.js';
import { effectiveActorApp } from '../../core/actor.js';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { Controller, Get, Post } from '../../core/http/decorators.js';
@@ -39,7 +38,19 @@ import {
} from '../../util/concurrency.js';
import { applyInlineContentSecurity } from '../../util/inlineContentSecurity.js';
import { PuterController } from '../types.js';
import { FS_COSTS } from './costs.js';
import { STORAGE_OP_COSTS } from '../../services/metering/costs.js';
import {
FS_MULTIPART_LIMIT,
FS_MUTATE_LIMIT,
FS_READ_CONCURRENT,
FS_READ_LIMIT,
FS_READDIR_LIMIT,
FS_SEARCH_CONCURRENT,
FS_SEARCH_LIMIT,
FS_STAT_LIMIT,
FS_WRITE_CONCURRENT,
FS_WRITE_LIMIT,
} from './limits.js';
import {
assertAccess as assertLegacyAccess,
fsEntryMimeType,
@@ -106,16 +117,26 @@ const DEFAULT_BATCH_WRITE_SIDE_EFFECT_CONCURRENCY = 8;
@Controller('/fs')
export class FSController extends PuterController {
// Object-store requests are reported here because the filesystem is what
// makes them. Bytes leaving the server are not: they are metered for every
// response, so `MeteringService` prices them.
override getReportedCosts() {
return Object.entries(FS_COSTS).map(([usageType, ucentsPerUnit]) => ({
usageType,
ucentsPerUnit,
unit: 'byte',
source: 'controller:fs',
}));
return Object.entries(STORAGE_OP_COSTS).map(
([usageType, ucentsPerUnit]) => ({
usageType,
ucentsPerUnit,
unit: 'operation',
source: 'controller:fs',
}),
);
}
@Post('/startWrite', { subdomain: 'api', requireVerified: true })
@Post('/startWrite', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
})
async startWrite(
req: Request<RouteParams, null, SignedWriteRequest>,
res: Response<ClientSignedWriteResponse>,
@@ -173,7 +194,12 @@ export class FSController extends PuterController {
);
}
@Post('/startBatchWrite', { subdomain: 'api', requireVerified: true })
@Post('/startBatchWrite', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
})
async startBatchWrites(
req: Request<RouteParams, null, SignedWriteRequest[]>,
res: Response<ClientSignedWriteResponse[]>,
@@ -273,7 +299,12 @@ export class FSController extends PuterController {
);
}
@Post('/completeWrite', { subdomain: 'api', requireVerified: true })
@Post('/completeWrite', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
})
async completeWrite(
req: Request<RouteParams, null, CompleteWriteRequest>,
res: Response<ClientCompleteWriteResponse>,
@@ -304,7 +335,12 @@ export class FSController extends PuterController {
);
}
@Post('/completeBatchWrite', { subdomain: 'api', requireVerified: true })
@Post('/completeBatchWrite', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
})
async completeBatchWrites(
req: Request<RouteParams, null, CompleteWriteRequest[]>,
res: Response<ClientCompleteWriteResponse[]>,
@@ -347,7 +383,11 @@ export class FSController extends PuterController {
);
}
@Post('/abortWrite', { subdomain: 'api', requireVerified: true })
@Post('/abortWrite', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MULTIPART_LIMIT,
})
async abortWrite(
req: Request<RouteParams, null, AbortWriteRequest>,
res: Response<{ ok: true }>,
@@ -363,7 +403,12 @@ export class FSController extends PuterController {
res.json({ ok: true });
}
@Post('/signMultipartParts', { subdomain: 'api', requireVerified: true })
@Post('/signMultipartParts', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
})
async signMultipartParts(
req: Request<RouteParams, null, SignMultipartPartsRequest>,
res: Response<ClientSignMultipartPartsResponse>,
@@ -376,7 +421,13 @@ export class FSController extends PuterController {
res.json(this.#withoutStorageInternals(response));
}
@Post('/write', { subdomain: 'api', requireVerified: true })
@Post('/write', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_WRITE_LIMIT,
concurrent: FS_WRITE_CONCURRENT,
})
async write(
req: Request<RouteParams, null, WriteRequest>,
res: Response<ClientWriteResponse>,
@@ -422,7 +473,13 @@ export class FSController extends PuterController {
res.json(this.#withRequiredClientFsEntry(updatedResponse));
}
@Post('/batchWrite', { subdomain: 'api', requireVerified: true })
@Post('/batchWrite', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_WRITE_LIMIT,
concurrent: FS_WRITE_CONCURRENT,
})
async batchWrites(
req: Request<RouteParams, null, WriteRequest[]>,
res: Response<ClientWriteResponse[]>,
@@ -889,7 +946,11 @@ export class FSController extends PuterController {
// -- Read-side routes ------------------------------------------------
@Post('/stat', { subdomain: 'api', requireVerified: true })
@Post('/stat', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_STAT_LIMIT,
})
async statEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -1003,12 +1064,20 @@ export class FSController extends PuterController {
* without a JSON body. Every value arrives as a string, so parameter
* parsing goes through the same coercion helpers the POST path uses.
*/
@Get('/readdir', { subdomain: 'api', requireVerified: true })
@Get('/readdir', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_READDIR_LIMIT,
})
async readdirEntriesViaGet(req: Request, res: Response) {
return this.readdirEntries(req, res);
}
@Post('/readdir', { subdomain: 'api', requireVerified: true })
@Post('/readdir', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_READDIR_LIMIT,
})
async readdirEntries(req: Request, res: Response) {
const actor = this.#requireActor(req);
// GET carries its parameters in the query string; POST in the body.
@@ -1204,7 +1273,12 @@ export class FSController extends PuterController {
}
}
@Post('/search', { subdomain: 'api', requireVerified: true })
@Post('/search', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_SEARCH_LIMIT,
concurrent: FS_SEARCH_CONCURRENT,
})
async searchEntries(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -1230,7 +1304,13 @@ export class FSController extends PuterController {
res.json(results);
}
@Get('/read', { subdomain: 'api', requireVerified: true })
@Get('/read', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_READ_LIMIT,
concurrent: FS_READ_CONCURRENT,
})
async readEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const query = this.#toObjectRecord(req.query);
@@ -1274,15 +1354,6 @@ export class FSController extends PuterController {
);
res.status(range ? 206 : 200);
const metering = this.services.metering as
| {
batchIncrementUsages?: (
actor: unknown,
entries: unknown[],
) => void;
}
| undefined;
try {
await pipeline(download.body, res);
} catch {
@@ -1290,28 +1361,15 @@ export class FSController extends PuterController {
// tore down both ends. Response is partially sent; nothing to do.
return;
}
// Meter egress only on successful completion.
if (metering?.batchIncrementUsages && download.contentLength) {
try {
const bytes = download.contentLength;
metering.batchIncrementUsages(actor, [
{
usageType: 'filesystem:egress:bytes',
usageAmount: bytes,
costOverride:
FS_COSTS['filesystem:egress:bytes'] * bytes,
},
]);
} catch {
// ignore — metering is non-critical.
}
}
}
// -- Mutation routes ------------------------------------------------
@Post('/mkdir', { subdomain: 'api', requireVerified: true })
@Post('/mkdir', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
async mkdirEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -1352,7 +1410,11 @@ export class FSController extends PuterController {
res.json(this.#toClientEntry(entry));
}
@Post('/touch', { subdomain: 'api', requireVerified: true })
@Post('/touch', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
async touchEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -1388,7 +1450,11 @@ export class FSController extends PuterController {
res.json(this.#toClientEntry(entry));
}
@Post('/rename', { subdomain: 'api', requireVerified: true })
@Post('/rename', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
async renameEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const body = this.#toObjectRecord(req.body);
@@ -1406,7 +1472,11 @@ export class FSController extends PuterController {
res.json(this.#toClientEntry(renamed));
}
@Post('/delete', { subdomain: 'api', requireVerified: true })
@Post('/delete', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
async deleteEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -1424,7 +1494,11 @@ export class FSController extends PuterController {
res.json({ ok: true });
}
@Post('/move', { subdomain: 'api', requireVerified: true })
@Post('/move', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
async moveEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -1452,7 +1526,12 @@ export class FSController extends PuterController {
res.json(this.#toClientEntry(moved));
}
@Post('/copy', { subdomain: 'api', requireVerified: true })
@Post('/copy', {
subdomain: 'api',
requireVerified: true,
requireCredits: true,
rateLimit: FS_MUTATE_LIMIT,
})
async copyEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -1467,6 +1546,7 @@ export class FSController extends PuterController {
await this.#assertAccess(actor, source.path, 'read');
await this.#assertAccess(actor, destinationParent.path, 'write');
const storageAllowanceMax = this.#getStorageAllowanceMaxOverride(req);
const copy = await this.services.fs.copy(userId, {
source,
destinationParent,
@@ -1475,12 +1555,19 @@ export class FSController extends PuterController {
overwrite: this.#toBoolean(body.overwrite) ?? false,
dedupeName:
this.#toBoolean(body.dedupe_name ?? body.change_name) ?? true,
...(storageAllowanceMax !== undefined
? { storageAllowanceMax }
: {}),
});
this.#emitGuiItemAdded(copy);
res.json(this.#toClientEntry(copy));
}
@Post('/mkshortcut', { subdomain: 'api', requireVerified: true })
@Post('/mkshortcut', {
subdomain: 'api',
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
async mkshortcutEntry(req: Request, res: Response) {
const actor = this.#requireActor(req);
const userId = this.#getActorUserId(req);
@@ -2091,8 +2178,7 @@ export class FSController extends PuterController {
// the ActorUser type. Access via the escape hatch until a proper
// storage-quota mechanism is in place.
const actorUser = req.actor?.user as
| Record<string, unknown>
| undefined;
Record<string, unknown> | undefined;
const candidates = [
this.#toStorageCapacityCandidate(actorUser?.free_storage),
@@ -2437,7 +2523,7 @@ export class FSController extends PuterController {
// search results to this path so app actors can't see entries outside
// their AppData via `/fs/search`.
#appDataScopeForActor(actor: Actor): string | undefined {
const app = effectiveActorApp(actor);
const app = actor.effectiveApp;
if (!app) return undefined;
const username = actor.user?.username;
if (typeof username !== 'string' || username.length === 0)
@@ -21,7 +21,7 @@ import type { Request, RequestHandler, Response } from 'express';
import { Readable, Writable } from 'node:stream';
import { v4 as uuidv4 } from 'uuid';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import type { Actor } from '../../core/actor.js';
import { makeActor, type Actor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterServer } from '../../server.js';
@@ -377,7 +377,7 @@ describe('LegacyFSController.readdirSubdomains', () => {
'INSERT INTO `subdomains` (`uuid`, `subdomain`, `user_id`) VALUES (?, ?, ?)',
[uuidv4(), `sd-${Math.random().toString(36).slice(2, 8)}`, userId],
);
const appActor: Actor = { ...actor, app: { uid: `app-${uuidv4()}` } };
const appActor = makeActor({ ...actor, app: { uid: `app-${uuidv4()}` } });
const { res, captured } = makeRes();
await withActor(appActor, () =>
controller.readdirSubdomains(makeReq({ actor: appActor }), res),
@@ -21,7 +21,7 @@ import type { Request, Response } from 'express';
import { Readable } from 'node:stream';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import type { Actor } from '../../core/actor.js';
import { makeActor, type Actor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterServer } from '../../server.js';
@@ -250,7 +250,7 @@ describe('LegacyFSController.mkdir', () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appUid = `app-legacy-mkdir-${uuidv4()}`;
const appActor: Actor = { ...userActor, app: { uid: appUid } };
const appActor = makeActor({ ...userActor, app: { uid: appUid } });
const parent = `/${username}/AppData`;
await withActor(userActor, () =>
@@ -934,7 +934,7 @@ describe('LegacyFSController.copy', () => {
const username = actor.user!.username!;
const src = `/${username}/Documents/dup.txt`;
const existing = `/${username}/Pictures/dup.txt`;
for ( const p of [src, existing] ) {
for (const p of [src, existing]) {
await withActor(actor, () =>
controller.touch(
makeReq({ body: { path: p }, actor }),
@@ -964,9 +964,8 @@ describe('LegacyFSController.copy', () => {
fields: { entry_name: 'dup.txt' },
});
const replaced = (await server.stores.fsEntry.getEntryByPath(
existing,
))!;
const replaced =
(await server.stores.fsEntry.getEntryByPath(existing))!;
// With overwrite: the replaced entry rides along in the response
// (so the caller can drop its row) and item.removed tells every
@@ -1091,7 +1090,7 @@ describe('LegacyFSController.move', () => {
const username = actor.user!.username!;
const src = `/${username}/Documents/clash.txt`;
const existing = `/${username}/Pictures/clash.txt`;
for ( const p of [src, existing] ) {
for (const p of [src, existing]) {
await withActor(actor, () =>
controller.touch(
makeReq({ body: { path: p }, actor }),
@@ -1121,9 +1120,8 @@ describe('LegacyFSController.move', () => {
fields: { entry_name: 'clash.txt' },
});
const replaced = (await server.stores.fsEntry.getEntryByPath(
existing,
))!;
const replaced =
(await server.stores.fsEntry.getEntryByPath(existing))!;
// With overwrite: the replaced entry rides along in the response
// (so the caller can drop its row) and item.removed tells every
@@ -1214,7 +1212,7 @@ describe('LegacyFSController.search', () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appUid = `app-legacy-search-${uuidv4()}`;
const appActor: Actor = { ...userActor, app: { uid: appUid } };
const appActor = makeActor({ ...userActor, app: { uid: appUid } });
const needle = `appneedle-${Math.random().toString(36).slice(2, 8)}`;
await withActor(userActor, () =>
@@ -1419,10 +1417,10 @@ describe('LegacyFSController.sign', () => {
},
{ ownerUserId: userActor.user!.id! },
);
const attackerActor: Actor = {
const attackerActor = makeActor({
...userActor,
app: { uid: `attacker-${uuidv4()}` },
};
});
const { res } = makeRes();
await expect(
@@ -1456,7 +1454,7 @@ describe('LegacyFSController.sign', () => {
},
{ ownerUserId: userActor.user!.id! },
);
const appActor: Actor = { ...userActor, app: { uid: ownApp.uid } };
const appActor = makeActor({ ...userActor, app: { uid: ownApp.uid } });
const { res, captured } = makeRes();
await withActor(appActor, () =>
@@ -1736,6 +1734,185 @@ describe('LegacyFSController.openItem (write_url stripping)', () => {
});
});
// ── openItem: the grant is a user-authority action ──────────────────
//
// /open_item writes a user→app ACL row. An app actor reaching it could hand
// itself write on any file it can read — read-only access to a folder would
// silently widen into persistent write on the files inside it, with no second
// consent prompt. The user has to be the one opening the item, and the row it
// leaves can never exceed what that user proved on the entry.
describe('LegacyFSController.openItem (grant scope)', () => {
const makeApp = async (ownerUserId: number) => {
const name = `opener-${uuidv4()}`;
return (await server.stores.app.create(
{
name,
title: 'Opener test app',
index_url: `https://${name}.test/`,
},
{ ownerUserId },
)) as { id: number; uid: string };
};
it('is registered behind the user-session gate', () => {
const router = new PuterRouter();
(
controller as unknown as {
registerRoutes: (r: PuterRouter) => void;
}
).registerRoutes(router);
const route = router.routes.find(
(r) => r.method === 'post' && r.path === '/open_item',
);
expect(route?.options).toMatchObject({
requireUserActor: true,
allowFullAccessToken: true,
});
});
it('rejects an app actor, leaving no grant behind', async () => {
const { actor: userActor, userId } = await makeUser();
const username = userActor.user!.username!;
const target = `/${username}/Documents/app-opened.txt`;
await withActor(userActor, () =>
controller.touch(
makeReq({ body: { path: target }, actor: userActor }),
makeRes().res,
),
);
const entry = await server.stores.fsEntry.getEntryByPath(target);
// The app holds the read the user approved, and is its own default
// opener — the shape that would let it widen that read into write.
const app = await makeApp(userId);
await withActor(userActor, () =>
server.services.permission.grantUserAppPermission(
userActor,
app.uid,
`fs:${entry!.uuid}:read`,
),
);
const appActor = makeActor({
...userActor,
app: { uid: app.uid, id: app.id },
});
const spy = vi
.spyOn(server.services.suggestedApps, 'getSuggestedApps')
.mockResolvedValue([{ uuid: app.uid }] as never);
try {
await expect(
withActor(appActor, () =>
controller.openItem(
makeReq({
body: { uid: entry!.uuid },
actor: appActor,
}),
makeRes().res,
),
),
).rejects.toMatchObject({ statusCode: 403 });
} finally {
spy.mockRestore();
}
expect(
await server.stores.permission.hasUserAppPerm(
userId,
app.id,
`fs:${entry!.uuid}:write`,
),
).toBe(false);
});
it('grants the suggested app write when the caller has write', async () => {
const { actor, userId } = await makeUser();
const target = `/${actor.user!.username}/Documents/owned-open.txt`;
await withActor(actor, () =>
controller.touch(
makeReq({ body: { path: target }, actor }),
makeRes().res,
),
);
const entry = await server.stores.fsEntry.getEntryByPath(target);
const app = await makeApp(userId);
const spy = vi
.spyOn(server.services.suggestedApps, 'getSuggestedApps')
.mockResolvedValue([{ uuid: app.uid }] as never);
try {
await withActor(actor, () =>
controller.openItem(
makeReq({ body: { path: target }, actor }),
makeRes().res,
),
);
} finally {
spy.mockRestore();
}
expect(
await server.stores.permission.hasUserAppPerm(
userId,
app.id,
`fs:${entry!.uuid}:write`,
),
).toBe(true);
});
it('grants only read when the caller is a read-only sharee', async () => {
const victim = await makeUser();
const sharee = await makeUser();
const target = `/${victim.actor.user!.username}/Documents/ro-open.txt`;
await withActor(victim.actor, () =>
controller.touch(
makeReq({ body: { path: target }, actor: victim.actor }),
makeRes().res,
),
);
const entry = await server.stores.fsEntry.getEntryByPath(target);
await server.services.permission.grantUserUserPermission(
victim.actor,
sharee.actor.user!.username!,
`fs:${entry!.uuid}:read`,
{},
);
const app = await makeApp(sharee.userId);
const spy = vi
.spyOn(server.services.suggestedApps, 'getSuggestedApps')
.mockResolvedValue([{ uuid: app.uid }] as never);
try {
await withActor(sharee.actor, () =>
controller.openItem(
makeReq({
body: { uid: entry!.uuid },
actor: sharee.actor,
}),
makeRes().res,
),
);
} finally {
spy.mockRestore();
}
expect(
await server.stores.permission.hasUserAppPerm(
sharee.userId,
app.id,
`fs:${entry!.uuid}:write`,
),
).toBe(false);
expect(
await server.stores.permission.hasUserAppPerm(
sharee.userId,
app.id,
`fs:${entry!.uuid}:read`,
),
).toBe(true);
});
});
// ── requestAppRootDir ───────────────────────────────────────────────
describe('LegacyFSController.requestAppRootDir', () => {
@@ -1766,10 +1943,10 @@ describe('LegacyFSController.requestAppRootDir', () => {
it('rejects with 403 when the actor.app.uid differs from the requested app_uid', async () => {
const { actor } = await makeUser();
const { res } = makeRes();
const appActor = {
const appActor = makeActor({
...actor,
app: { uid: 'app-mismatch' },
} as unknown as Actor;
});
await expect(
withActor(appActor, () =>
controller.requestAppRootDir(
@@ -1787,10 +1964,10 @@ describe('LegacyFSController.requestAppRootDir', () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const appUid = 'app-self';
const appActor = {
const appActor = makeActor({
...actor,
app: { uid: appUid },
} as unknown as Actor;
});
const { res, captured } = makeRes();
await withActor(appActor, () =>
controller.requestAppRootDir(
@@ -2463,10 +2640,10 @@ describe('LegacyFSController.sign app sandbox + write downgrade', () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
// Build an app-under-user actor whose AppData root is the test app.
const appActor = {
const appActor = makeActor({
...actor,
app: { uid: 'sandbox-app' },
} as unknown as Actor;
});
// Create a file *outside* /Documents (anywhere outside AppData/<uid>).
const target = `/${username}/Documents/forbidden.txt`;
@@ -2730,10 +2907,10 @@ describe('LegacyFSController.suggestApps', () => {
it('refuses to look up entries an app actor cannot see', async () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appActor: Actor = {
const appActor = makeActor({
...userActor,
app: { uid: `app-suggest-${uuidv4()}` },
};
});
const target = `/${username}/Documents/probe.txt`;
await withActor(userActor, () =>
controller.touch(
@@ -2801,10 +2978,10 @@ describe('LegacyFSController.suggestApps', () => {
describe('LegacyFSController.readdirSubdomains', () => {
it('returns an empty array for app-under-user actors', async () => {
const { actor: userActor, userId } = await makeUser();
const appActor: Actor = {
const appActor = makeActor({
...userActor,
app: { uid: `app-subd-${uuidv4()}` },
};
});
await server.clients.db.write(
'INSERT INTO `subdomains` (`uuid`, `subdomain`, `user_id`) VALUES (?, ?, ?)',
[uuidv4(), `sd-${uuidv4().slice(0, 8)}`, userId],
@@ -2843,10 +3020,10 @@ describe('LegacyFSController.updateFsentryThumbnail', () => {
it('rejects an app actor probing entries outside AppData with 404', async () => {
const { actor: userActor } = await makeUser();
const username = userActor.user!.username!;
const appActor: Actor = {
const appActor = makeActor({
...userActor,
app: { uid: `app-thumb-${uuidv4()}` },
};
});
const target = `/${username}/Documents/thumbme.txt`;
await withActor(userActor, () =>
controller.touch(
@@ -2915,40 +3092,48 @@ describe('LegacyFSController.updateFsentryThumbnail', () => {
// object — an fs object's key is its fsentry uuid — and have the server
// read or destroy it. Only inline image data is accepted.
it.each([
['an s3:// pointer', 's3://puter-local/00000000-0000-4000-8000-000000000000'],
[
'an s3:// pointer',
's3://puter-local/00000000-0000-4000-8000-000000000000',
],
['an https URL', 'https://cdn.example.com/x.png'],
['a bare object key', 'thumbnails/whatever'],
])('rejects %s instead of storing it verbatim', async (_label, thumbnail) => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const target = `/${username}/Documents/thumbme3-${uuidv4()}.txt`;
await withActor(actor, () =>
controller.touch(
makeReq({
body: { path: target, set_modified_to_now: true },
actor,
}),
makeRes().res,
),
);
const entry = await server.stores.fsEntry.getEntryByPath(target);
const { res } = makeRes();
await expect(
withActor(actor, () =>
controller.updateFsentryThumbnail(
])(
'rejects %s instead of storing it verbatim',
async (_label, thumbnail) => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const target = `/${username}/Documents/thumbme3-${uuidv4()}.txt`;
await withActor(actor, () =>
controller.touch(
makeReq({
body: { uid: entry!.uuid, thumbnail },
body: { path: target, set_modified_to_now: true },
actor,
}),
res,
makeRes().res,
),
),
).rejects.toMatchObject({ statusCode: 400 });
);
const entry = await server.stores.fsEntry.getEntryByPath(target);
const after = await server.stores.fsEntry.getEntryByUuid(entry!.uuid);
expect(after?.thumbnail ?? null).toBeNull();
});
const { res } = makeRes();
await expect(
withActor(actor, () =>
controller.updateFsentryThumbnail(
makeReq({
body: { uid: entry!.uuid, thumbnail },
actor,
}),
res,
),
),
).rejects.toMatchObject({ statusCode: 400 });
const after = await server.stores.fsEntry.getEntryByUuid(
entry!.uuid,
);
expect(after?.thumbnail ?? null).toBeNull();
},
);
});
// ── GET /get-launch-apps ────────────────────────────────────────────
+270 -135
View File
@@ -21,17 +21,21 @@ import Busboy from 'busboy';
import type { NextFunction, Request, RequestHandler, Response } from 'express';
import { contentType as contentTypeFromMime } from 'mime-types';
import { posix as pathPosix } from 'node:path';
import type { Actor } from '../../core/actor.js';
import { effectiveActorApp, isAccessTokenActor } from '../../core/actor.js';
import {
assertResolvedActor,
isAccessTokenActor,
makeActor,
} from '../../core/actor.js';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { RouteOptions } from '../../core/http/index.js';
import {
assertNotSuspended,
assertVerifiedAccount,
} from '../../core/http/middleware/gates.js';
import { RouteOptions } from '../../core/http/index.js';
import type { PuterRouter } from '../../core/http/PuterRouter.js';
import type { ACLService } from '../../services/acl/ACLService.js';
import { assertActorHasCredits } from '../../services/metering/enforcement.js';
import type { SignedFile } from '../../util/fileSigning.js';
import { verifySignature } from '../../util/fileSigning.js';
import {
@@ -40,7 +44,24 @@ import {
} from '../../util/hostedAppBacking.js';
import { applyInlineContentSecurity } from '../../util/inlineContentSecurity.js';
import { PuterController } from '../types.js';
import { FS_COSTS } from './costs.js';
import {
FS_BATCH_CONCURRENT,
FS_BATCH_LIMIT,
FS_DF_LIMIT,
FS_HELPER_LIMIT,
FS_MUTATE_LIMIT,
FS_POLL_LIMIT,
FS_READ_CONCURRENT,
FS_READ_LIMIT,
FS_READDIR_LIMIT,
FS_SEARCH_CONCURRENT,
FS_SEARCH_LIMIT,
FS_SIGN_LIMIT,
FS_SIGNED_CONCURRENT,
FS_SIGNED_READ_LIMIT,
FS_SIGNED_WRITE_LIMIT,
FS_STAT_LIMIT,
} from './limits.js';
import {
asRecord,
assertAccess,
@@ -95,6 +116,14 @@ export class LegacyFSController extends PuterController {
subdomain: 'api',
requireVerified: true,
} as RouteOptions;
// Operations that move file content or make object-store requests on
// the caller's behalf, and so are refused to an account with nothing
// left of its budget. The metadata routes above deliberately aren't:
// an account that has run out still has to be able to look at what it
// has and delete it. The signature-authorised routes aren't either —
// they carry no session to answer for, and `/sign` (which does) is
// where the URL that reaches them is minted.
const spends = { ...apiOptions, requireCredits: true } as RouteOptions;
// Signed-URL routes: the handler validates the URL signature itself,
// so no auth gate is applied (matches v1, which mounted these routers
// with no middleware).
@@ -103,40 +132,118 @@ export class LegacyFSController extends PuterController {
} as RouteOptions;
// Core filesystem_api routes — direct handlers over the FS service.
router.post('/stat', apiOptions, this.stat);
router.post('/readdir', apiOptions, this.readdir);
router.post('/mkdir', apiOptions, this.mkdir);
router.post('/copy', apiOptions, this.copy);
router.post('/move', apiOptions, this.move);
router.post('/delete', apiOptions, this.delete);
router.post('/rename', apiOptions, this.rename);
router.post('/touch', apiOptions, this.touch);
router.post('/search', apiOptions, this.search);
router.get('/read', apiOptions, this.read);
// Limits come from `./limits` and carry an explicit `scope`, so these
// draw from the same per-user budget as their v2 counterparts rather
// than handing a caller a second allowance for the same operation.
const mutate = { ...apiOptions, rateLimit: FS_MUTATE_LIMIT };
router.post(
'/stat',
{ ...apiOptions, rateLimit: FS_STAT_LIMIT },
this.stat,
);
router.post(
'/readdir',
{ ...apiOptions, rateLimit: FS_READDIR_LIMIT },
this.readdir,
);
router.post('/mkdir', mutate, this.mkdir);
router.post('/copy', { ...mutate, requireCredits: true }, this.copy);
router.post('/move', mutate, this.move);
router.post('/delete', mutate, this.delete);
router.post('/rename', mutate, this.rename);
router.post('/touch', mutate, this.touch);
router.post(
'/search',
{
...apiOptions,
rateLimit: FS_SEARCH_LIMIT,
concurrent: FS_SEARCH_CONCURRENT,
},
this.search,
);
router.get(
'/read',
{
...spends,
rateLimit: FS_READ_LIMIT,
concurrent: FS_READ_CONCURRENT,
},
this.read,
);
router.get(
'/token-read',
{
subdomain: 'api',
requireVerified: false,
allowAccessToken: true,
// An access token may or may not carry a user, so this shares
// the network-keyed budget the other signed routes use.
rateLimit: FS_SIGNED_READ_LIMIT,
},
this.tokenRead,
);
router.post('/batch', apiOptions, this.batch);
router.post(
'/batch',
{
...spends,
rateLimit: FS_BATCH_LIMIT,
concurrent: FS_BATCH_CONCURRENT,
},
this.batch,
);
// Signed-URL + meta routes.
router.post('/sign', apiOptions, this.sign);
router.post('/writeFile', signedOptions, this.writeFile);
router.get('/file', signedOptions, this.file);
router.all('/df', apiOptions, this.df);
router.post('/open_item', apiOptions, this.openItem);
router.post(
'/sign',
// Gated even though it moves nothing itself: the URL it returns
// outlives the request and is served by a route with no session to
// check, so this is the last point at which the account is known.
{ ...spends, rateLimit: FS_SIGN_LIMIT },
this.sign,
);
router.post(
'/writeFile',
{
...signedOptions,
rateLimit: FS_SIGNED_WRITE_LIMIT,
concurrent: FS_SIGNED_CONCURRENT,
},
this.writeFile,
);
router.get(
'/file',
{
...signedOptions,
rateLimit: FS_SIGNED_READ_LIMIT,
concurrent: FS_SIGNED_CONCURRENT,
},
this.file,
);
router.all('/df', { ...apiOptions, rateLimit: FS_DF_LIMIT }, this.df);
router.post(
'/open_item',
{
...apiOptions,
// Opening an item grants an app access to a file on the user's
// behalf, so only the user's own credential may drive it — an
// app calling it for itself would be widening its own ACL.
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: FS_HELPER_LIMIT,
},
this.openItem,
);
router.post(
'/auth/request-app-root-dir',
apiOptions,
{ ...apiOptions, rateLimit: FS_SIGN_LIMIT },
this.requestAppRootDir,
);
router.post('/auth/check-app-acl', apiOptions, this.checkAppAcl);
router.post(
'/auth/check-app-acl',
{ ...apiOptions, rateLimit: FS_SIGN_LIMIT },
this.checkAppAcl,
);
// `/down` — session-auth'd file download. Unlike `/file` (signed URL)
// this accepts a path on the user's behalf and streams as attachment.
@@ -153,7 +260,10 @@ export class LegacyFSController extends PuterController {
// (CSRF can't forge a header-credentialed request).
allowFullAccessToken: true,
requireVerified: true,
requireCredits: true,
antiCsrf: true,
rateLimit: FS_READ_LIMIT,
concurrent: FS_READ_CONCURRENT,
},
this.down,
);
@@ -164,92 +274,102 @@ export class LegacyFSController extends PuterController {
});
});
router.get('/get-launch-apps', apiOptions, async (req, res) => {
const recommendedSvc = this.services.recommendedApps as unknown as
{ getRecommendedApps?: () => Promise<unknown[]> } | undefined;
const recommended = recommendedSvc?.getRecommendedApps
? await recommendedSvc.getRecommendedApps()
: [];
router.get(
'/get-launch-apps',
{ ...apiOptions, rateLimit: FS_HELPER_LIMIT },
async (req, res) => {
const recommendedSvc = this.services
.recommendedApps as unknown as
| { getRecommendedApps?: () => Promise<unknown[]> }
| undefined;
const recommended = recommendedSvc?.getRecommendedApps
? await recommendedSvc.getRecommendedApps()
: [];
let recent: unknown[] = [];
const userId = req.actor?.user?.id;
if (userId) {
const recentUids =
(await (
let recent: unknown[] = [];
const userId = req.actor?.user?.id;
if (userId) {
const recentUids =
(await (
this.stores.app as unknown as {
getRecentAppOpens?: (
id: number,
opts?: { limit?: number },
) => Promise<string[]>;
}
).getRecentAppOpens?.(userId, { limit: 10 })) ?? [];
// One batched read for the rows, then the backing checks
// concurrently. Serially awaiting a lookup per uid put ~2
// round trips of latency on every desktop boot.
const appsByUid = await (
this.stores.app as unknown as {
getRecentAppOpens?: (
id: number,
opts?: { limit?: number },
) => Promise<string[]>;
getByUids: (
uids: string[],
) => Promise<Map<string, Record<string, unknown>>>;
}
).getRecentAppOpens?.(userId, { limit: 10 })) ?? [];
// One batched read for the rows, then the backing checks
// concurrently. Serially awaiting a lookup per uid put ~2
// round trips of latency on every desktop boot.
const appsByUid = await (
this.stores.app as unknown as {
getByUids: (
uids: string[],
) => Promise<Map<string, Record<string, unknown>>>;
}
).getByUids(recentUids);
).getByUids(recentUids);
// `recentUids` is ordered most-recent-first; preserve it.
const orderedApps = recentUids
.map((uid) => appsByUid.get(uid))
.filter((app): app is Record<string, unknown> =>
Boolean(app),
// `recentUids` is ordered most-recent-first; preserve it.
const orderedApps = recentUids
.map((uid) => appsByUid.get(uid))
.filter((app): app is Record<string, unknown> =>
Boolean(app),
);
// Don't hand out an index_url whose puter-hosted backing is
// gone or reclaimed. The taskbar launches recents by name (so
// AppDriver's guard applies), but this list is a
// launch-metadata producer like any other — a future consumer
// reading index_url straight off it shouldn't inherit a stale
// origin.
const backingGoneFlags = await Promise.all(
orderedApps.map((app) =>
hostedIndexUrlBackingIsUnavailable({
app,
subdomainStore: this.stores.subdomain,
config: this.config,
}).catch(() => true),
),
);
// Don't hand out an index_url whose puter-hosted backing is
// gone or reclaimed. The taskbar launches recents by name (so
// AppDriver's guard applies), but this list is a
// launch-metadata producer like any other — a future consumer
// reading index_url straight off it shouldn't inherit a stale
// origin.
const backingGoneFlags = await Promise.all(
orderedApps.map((app) =>
hostedIndexUrlBackingIsUnavailable({
app,
subdomainStore: this.stores.subdomain,
config: this.config,
}).catch(() => true),
),
);
recent = orderedApps.map((app, index) => {
const backingGone = backingGoneFlags[index];
return {
uuid: app.uid,
name: app.name,
title: app.title,
icon: app.icon ?? null,
godmode: Boolean(app.godmode),
maximize_on_start: Boolean(app.maximize_on_start),
index_url: backingGone ? null : app.index_url,
...(backingGone
? { privateAccess: buildHostedBackingDenial() }
: {}),
// An app with no owner isn't owned by a Puter user —
// it's an "external" (origin-bootstrapped) app.
external:
app.owner_user_id == null ||
app.owner_user_id === '',
};
});
}
recent = orderedApps.map((app, index) => {
const backingGone = backingGoneFlags[index];
return {
uuid: app.uid,
name: app.name,
title: app.title,
icon: app.icon ?? null,
godmode: Boolean(app.godmode),
maximize_on_start: Boolean(app.maximize_on_start),
index_url: backingGone ? null : app.index_url,
...(backingGone
? { privateAccess: buildHostedBackingDenial() }
: {}),
// An app with no owner isn't owned by a Puter user —
// it's an "external" (origin-bootstrapped) app.
external:
app.owner_user_id == null ||
app.owner_user_id === '',
};
});
}
res.json({ recommended, recent });
},
);
res.json({ recommended, recent });
});
router.post('/suggest_apps', apiOptions, this.suggestApps);
router.post(
'/suggest_apps',
{ ...apiOptions, rateLimit: FS_HELPER_LIMIT },
this.suggestApps,
);
// puter-js polls this to decide whether to purge its in-memory FS
// cache. SocketService bumps a per-user Redis key on every
// `outer.gui.item.*` mutation — read it back here.
router.get(
'/cache/last-change-timestamp',
apiOptions,
{ ...apiOptions, rateLimit: FS_POLL_LIMIT },
async (req, res) => {
const userId = req.actor?.user?.id;
if (!userId) {
@@ -270,10 +390,14 @@ export class LegacyFSController extends PuterController {
},
);
router.post('/readdir-subdomains', apiOptions, this.readdirSubdomains);
router.post(
'/readdir-subdomains',
{ ...apiOptions, rateLimit: FS_HELPER_LIMIT },
this.readdirSubdomains,
);
router.post(
'/update-fsentry-thumbnail',
apiOptions,
{ ...apiOptions, rateLimit: FS_HELPER_LIMIT },
this.updateFsentryThumbnail,
);
@@ -908,7 +1032,7 @@ export class LegacyFSController extends PuterController {
const actor = this.#requireActor(req);
// Subdomain enumeration is a user-level concern; app actors would
// otherwise see root_dir uids pointing outside their AppData scope.
if (effectiveActorApp(actor)) {
if (actor.effectiveApp) {
res.json([]);
return;
}
@@ -985,7 +1109,7 @@ export class LegacyFSController extends PuterController {
// App-under-user actors only see entries within their AppData root;
// user actors are unscoped. Mirrors the ACL short-circuit in
// ACLService.check.
const app = effectiveActorApp(actor);
const app = actor.effectiveApp;
const username = actor.user?.username;
const pathScope =
app && typeof username === 'string' && username.length > 0
@@ -1070,32 +1194,6 @@ export class LegacyFSController extends PuterController {
);
res.status(range ? 206 : 200);
// Best-effort egress metering.
const metering = this.services.metering as
| {
batchIncrementUsages?: (
actor: unknown,
entries: unknown[],
) => void;
}
| undefined;
if (metering?.batchIncrementUsages && download.contentLength) {
download.body.once('end', () => {
try {
const bytes = download.contentLength!;
metering.batchIncrementUsages!(actor, [
{
usageType: 'filesystem:egress:bytes',
usageAmount: bytes,
costOverride:
FS_COSTS['filesystem:egress:bytes'] * bytes,
},
]);
} catch {
// ignore — non-critical.
}
});
}
download.body.on('error', (err) => {
res.destroy(err);
});
@@ -1125,9 +1223,20 @@ export class LegacyFSController extends PuterController {
assertNotSuspended(actor!.user);
assertVerifiedAccount(actor!.user);
req.actor = actor!;
req.actor = assertResolvedActor(actor!);
Context.set('actor', actor);
// And the budget gate the other read routes declare with
// `requireCredits`. The global auth probe only looks for `auth_token`,
// so `?token=` leaves `req.actor` unset for the whole gate chain and
// the declarative form would wave every request through — this streams
// file content like `/read` does, so it is refused on the same terms.
await assertActorHasCredits(
this.services.metering,
req.actor,
this.config,
);
// Forward back to regular read after setting actor
return this.read(req, res, undefined, { realMime: true });
};
@@ -1514,6 +1623,23 @@ export class LegacyFSController extends PuterController {
});
}
// Name who this response's bytes are billed to. A signature authorises
// access to a file; it says nothing about who is asking, so an
// unidentified caller is billed to the account whose file it is —
// otherwise a signed URL is a way to serve content for free. A caller
// who did identify themselves pays for what they fetch, as on a hosted
// site.
if (owner?.uuid) {
req.egressActor = req.actor ?? {
user: {
uuid: owner.uuid,
id: owner.id,
username: owner.username,
suspended: !!(owner as { suspended?: unknown }).suspended,
},
};
}
// Directory: return a signed listing of direct children.
// The caller only proved read access, so strip write_url from
// each child to prevent privilege escalation via /writeFile.
@@ -1581,16 +1707,25 @@ export class LegacyFSController extends PuterController {
};
/**
* POST /open_item — resolve an entry, grant the default suggested app write
* POST /open_item — resolve an entry, grant the default suggested app
* access to it, and return a signed URL + user-app token so the launched
* app can read/write the file via its app-under-user token.
*
* Matches v1 semantics: permission is always granted as `write` — the
* underlying user's permission check still caps the effective access
* (grantUserAppPermission doesn't escalate user privileges).
* This is a user-authority action: it writes a user→app ACL row. Two things
* keep an app from opening an item to widen its own access — the caller
* must be the user themselves (the route gate, re-checked here because
* extensions can reach handlers directly), and the grant is capped at the
* access the caller actually proved on the entry.
*/
openItem = async (req: Request, res: Response): Promise<void> => {
const actor = this.#requireActor(req);
if ((actor as { app?: unknown }).app) {
throw new HttpError(
403,
'This endpoint is only available to user sessions',
{ legacyCode: 'forbidden' },
);
}
const body = asRecord(req.body);
const entry = await resolveV1Selector(this.stores.fsEntry, body);
@@ -1602,8 +1737,8 @@ export class LegacyFSController extends PuterController {
'read',
);
// Downgrade the envelope when the caller only proved read.
// `/writeFile`'s ACL re-check would still block the write, but
// Downgrade the envelope and the grant when the caller only proved
// read. `/writeFile`'s ACL re-check would still block the write, but
// returning `write_url` to a read-only caller is the same
// privilege-leak shape that `/sign` and `/readdir` strip.
const writeOk = await this.services.acl.check(
@@ -1631,7 +1766,7 @@ export class LegacyFSController extends PuterController {
await this.services.permission.grantUserAppPermission(
actor,
defaultAppUid,
`fs:${entry.uuid}:write`,
`fs:${entry.uuid}:${writeOk ? 'write' : 'read'}`,
{},
{ reason: 'open_item' },
);
@@ -1727,10 +1862,10 @@ export class LegacyFSController extends PuterController {
});
// Build an actor-under-user shape for the check.
const actorForApp = {
user: (req.actor as { user?: unknown }).user,
const actorForApp = makeActor({
user: req.actor!.user,
app: { uid: (app as { uid: string }).uid },
} as unknown as Actor;
});
const descriptor = {
path: subject.path,
resolveAncestors: () =>
-30
View File
@@ -1,30 +0,0 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { toMicroCents } from '../../services/metering/utils.js';
// Microcents per byte. Egress roughly matches S3 data-transfer-out
// (~$0.12/GiB); cached egress is CloudFront-backed (~$0.10/GiB).
// Ingress and deletes are currently free.
export const FS_COSTS = {
'filesystem:ingress:bytes': 0,
'filesystem:delete:bytes': 0,
'filesystem:egress:bytes': toMicroCents(0.12 / 1024 / 1024 / 1024),
'filesystem:cached-egress:bytes': toMicroCents(0.1 / 1024 / 1024 / 1024),
} as const;
+163
View File
@@ -0,0 +1,163 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { describe, expect, it } from 'vitest';
import * as limits from './limits.js';
import {
DEFAULT_FREE_SUBSCRIPTION,
DEFAULT_TEMP_SUBSCRIPTION,
} from '../../services/metering/consts.js';
import type { RouteOptions, RouteRateLimit } from '../../core/http/types';
type Spec = RouteRateLimit | NonNullable<RouteOptions['concurrent']>;
const all = Object.entries(limits) as Array<[string, Spec | RouteRateLimit[]]>;
const flat: Array<[string, Spec]> = all.flatMap(([name, value]) =>
Array.isArray(value)
? value.map((v, i): [string, Spec] => [`${name}[${i}]`, v])
: [[name, value] as [string, Spec]],
);
const windows = flat.filter(([, s]) => 'window' in s) as Array<
[string, RouteRateLimit]
>;
const concurrents = flat.filter(([, s]) => !('window' in s));
describe('filesystem limit specs', () => {
// The whole point of this module is that the legacy and v2 controllers
// import the same specs. That only ties the counters together if every
// spec carries an explicit scope — without one the gate falls back to
// the route path, and the two controllers get separate budgets.
it.each(flat)('%s pins an explicit scope', (_name, spec) => {
expect(spec.scope).toBeTruthy();
});
it.each(windows)('%s has a positive limit and window', (_name, spec) => {
expect(spec.limit).toBeGreaterThan(0);
expect(spec.window).toBeGreaterThan(0);
});
// Base is the paid value; the free tiers are carved out beneath it.
// A free tier above the base would mean paying made you worse off.
it.each(windows)('%s never lets a free tier exceed paid', (_name, spec) => {
for (const n of Object.values(spec.bySubscription ?? {})) {
expect(n).toBeLessThanOrEqual(spec.limit);
}
});
it.each(windows)(
'%s caps temp at or below registered-free',
(_name, spec) => {
const free = spec.bySubscription?.[DEFAULT_FREE_SUBSCRIPTION];
const temp = spec.bySubscription?.[DEFAULT_TEMP_SUBSCRIPTION];
if (free === undefined || temp === undefined) return;
expect(temp).toBeLessThanOrEqual(free);
},
);
// A single in-flight slot turns incidental client parallelism into a
// spurious 429; paid tiers keep room to actually parallelise.
it.each(concurrents)(
'%s keeps concurrency at 5+ paid and 2+ for every tier',
(_name, spec) => {
expect(spec.limit).toBeGreaterThanOrEqual(5);
for (const n of Object.values(spec.bySubscription ?? {})) {
expect(n).toBeGreaterThanOrEqual(2);
}
},
);
it('gives search the tightest window of the read paths', () => {
expect(limits.FS_SEARCH_LIMIT.limit).toBeLessThan(
limits.FS_STAT_LIMIT.limit,
);
expect(limits.FS_SEARCH_LIMIT.limit).toBeLessThan(
limits.FS_READ_LIMIT.limit,
);
});
// The desktop deletes/moves one item per request with no batching and no
// pacing, so a single "empty trash" or "select all, delete" has to fit
// inside the minute window on every tier. The hour window is the abuse
// ceiling that a minute window this wide can no longer be.
it('gives mutations a bulk-sized minute window plus an hourly backstop', () => {
const [minute, hourly] = limits.FS_MUTATE_LIMIT;
expect(limits.FS_MUTATE_LIMIT).toHaveLength(2);
expect(minute.window).toBe(60_000);
expect(hourly.window).toBe(60 * 60_000);
const tiers = (spec: RouteRateLimit) => [
spec.limit,
spec.bySubscription![DEFAULT_FREE_SUBSCRIPTION],
spec.bySubscription![DEFAULT_TEMP_SUBSCRIPTION],
];
// A few hundred items clears in one pass, anonymous included.
for (const n of tiers(minute)) expect(n).toBeGreaterThanOrEqual(500);
// Anonymous stays meaningfully tighter than paid on both windows.
expect(tiers(minute)[2]).toBeLessThanOrEqual(minute.limit / 2);
expect(tiers(hourly)[2]).toBeLessThanOrEqual(hourly.limit / 2);
// The hour window has to bind rather than decorate: it allows a
// handful of bulk passes an hour, not sixty minutes' worth.
for (const [perMinute, perHour] of tiers(minute).map(
(n, i): [number, number] => [n, tiers(hourly)[i]],
)) {
expect(perHour).toBeGreaterThan(perMinute);
expect(perHour).toBeLessThanOrEqual(perMinute * 10);
}
});
// The DAV gate is consumed imperatively, before the request has an actor:
// it keys on the network fingerprint and reads only `limit` / `window`. A
// `key: 'user'` or a `bySubscription` map here would describe tiering that
// never happens.
it('shapes the DAV specs the way the DAV gate consumes them', () => {
expect(limits.DAV_LIMIT.key).toBe('fingerprint');
expect(limits.DAV_CONCURRENT.key).toBe('fingerprint');
expect(limits.DAV_LIMIT.bySubscription).toBeUndefined();
expect(limits.DAV_CONCURRENT.bySubscription).toBeUndefined();
});
// With no session to key on, the alternative is the bare address — and an
// address is a household, an office or a carrier gateway, so keying there
// makes one bucket serve everyone behind it and tighten as more real users
// arrive. The fingerprint separates clients within a network while still
// being something one client can't vary per request.
it('keys the signed-URL routes on the network fingerprint', () => {
expect(limits.FS_SIGNED_READ_LIMIT.key).toBe('fingerprint');
expect(limits.FS_SIGNED_WRITE_LIMIT.key).toBe('fingerprint');
expect(limits.FS_SIGNED_CONCURRENT.key).toBe('fingerprint');
});
// These serve page subresources — a gallery, an app's own assets — so the
// in-flight cap has to sit above what a browser opens to one origin at
// once, or a normal page load is what trips it.
it('leaves the signed-URL in-flight cap above a browser`s own parallelism', () => {
expect(limits.FS_SIGNED_CONCURRENT.limit).toBeGreaterThan(30);
});
it('uses distinct scopes so counters cannot collide', () => {
const scopes = flat
.filter(([, s]) => 'window' in s)
.map(([, s]) => s.scope);
expect(new Set(scopes).size).toBe(scopes.length);
});
});
+245
View File
@@ -0,0 +1,245 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import {
DEFAULT_FREE_SUBSCRIPTION,
DEFAULT_TEMP_SUBSCRIPTION,
} from '../../services/metering/consts.js';
import type { RouteOptions, RouteRateLimit } from '../../core/http/types';
// -- Shared filesystem limits ----------------------------------------
//
// The v2 controller and the legacy controller expose the same operations
// on separate route tables. Both import the specs below so a caller can't
// get two budgets for one operation by switching endpoints — the `scope`
// is what ties the counters together, and it only works if both sides
// pass the same one.
//
// The base `limit` is what subscribed tiers see; `bySubscription` carves
// out the free tiers beneath it. Any plan id not enumerated (a Stripe
// plan, the dev-only `unlimited`) falls through to the base, so new plans
// are generous by default rather than accidentally throttled.
//
// Storage quota already bounds total bytes, and egress and object-store
// requests are metered. These bound request *count*, which neither does:
// metadata reads cost database time while costing the caller almost
// nothing, and metering settles seconds behind the traffic, so a limit is
// what actually stops a runaway loop in the moment.
/** Per-user sliding window. Free tiers are carved out of the paid base. */
const userWindow = (
scope: string,
paid: number,
free: number,
temp: number,
window = 60_000,
): RouteRateLimit => ({
scope,
limit: paid,
window,
key: 'user',
bySubscription: {
[DEFAULT_FREE_SUBSCRIPTION]: free,
[DEFAULT_TEMP_SUBSCRIPTION]: temp,
},
});
/**
* In-flight cap. Nothing drops below 2 — a single slot turns any incidental
* parallelism in a client (two tabs, a prefetch alongside a user action) into a
* spurious 429 — and subscribed tiers keep enough headroom to actually
* parallelise.
*/
const userConcurrent = (
scope: string,
paid: number,
free: number,
temp: number,
): NonNullable<RouteOptions['concurrent']> => ({
scope,
limit: paid,
key: 'user',
bySubscription: {
[DEFAULT_FREE_SUBSCRIPTION]: free,
[DEFAULT_TEMP_SUBSCRIPTION]: temp,
},
});
/**
* Per-network window, for the signature-authenticated routes with no session.
*
* `fingerprint` rather than `ip`: without an actor the only alternative is the
* bare address, and an address is an aggregate — a household, an office, a
* mobile carrier's gateway. Keying on it means one bucket for everyone behind
* it, which is a shared cap that gets tighter the more real users are present.
* The fingerprint folds in the headers a client can't vary per request without
* also looking like a different client, so separate browsers behind one NAT get
* separate buckets while a single client still can't mint fresh ones per call.
*/
const networkWindow = (
scope: string,
limit: number,
window = 60_000,
): RouteRateLimit => ({ scope, limit, window, key: 'fingerprint' });
// -- Metadata reads --------------------------------------------------
/** Chattiest call in the GUI; high enough to only ever catch a loop. */
export const FS_STAT_LIMIT = userWindow('fs:stat', 1200, 600, 300);
/**
* Desktop boot fans out hard, so the minute budget is generous — the short
* second window is what actually catches a runaway loop before it has spent the
* whole minute's allowance.
*/
export const FS_READDIR_LIMIT: RouteRateLimit[] = [
userWindow('fs:readdir', 600, 300, 120),
userWindow('fs:readdir-burst', 120, 60, 30, 10_000),
];
/**
* Unindexed scan across the user's tree, and close to unmetered — a result set
* is a few hundred bytes of egress against an arbitrary amount of database
* work, so this is the cheapest way to occupy a connection. Tightest limit in
* the file.
*/
export const FS_SEARCH_LIMIT = userWindow('fs:search', 60, 30, 10);
export const FS_SEARCH_CONCURRENT = userConcurrent('fs:search', 5, 2, 2);
/** Aggregation over the whole tree; the GUI needs it rarely. */
export const FS_DF_LIMIT = userWindow('fs:df', 60, 30, 15);
// -- Content transfer ------------------------------------------------
/**
* Egress is billed, so cost is already bounded — the cap here is against
* connection exhaustion, which is why the concurrency slot matters more than
* the window.
*/
export const FS_READ_LIMIT = userWindow('fs:read', 600, 300, 120);
export const FS_READ_CONCURRENT = userConcurrent('fs:read', 10, 5, 3);
export const FS_WRITE_LIMIT = userWindow('fs:write', 300, 120, 30);
export const FS_WRITE_CONCURRENT = userConcurrent('fs:write', 15, 6, 3);
/**
* Multipart handshake — several calls per upload, so a multiple of the write
* budget rather than a peer of it. Signing is the half that costs us
* object-store calls.
*
* The multiple is what matters: a per-minute ceiling here divides down into a
* much smaller number of files, and selecting a folder's worth of them is one
* gesture. `FS_WRITE_LIMIT` and storage quota already bound what actually gets
* written, so this only has to stay out of the way of a real upload.
*/
export const FS_MULTIPART_LIMIT = userWindow('fs:multipart', 2400, 1200, 600);
// -- Metadata mutations ----------------------------------------------
/**
* Mkdir / touch / rename / delete / move / copy / mkshortcut.
*
* The desktop issues one call per item and does not pace them: emptying the
* trash, deleting a multi-selection, or dragging a folder's worth of files
* fires the whole set back to back. A minute budget in the low hundreds turns
* an ordinary "select all, delete" into a partial failure, so the minute window
* is sized to clear a bulk pass over a few hundred items on every tier — an
* anonymous session gets a real desktop, so it needs a real bulk allowance too,
* just a smaller one.
*
* The hour window is where the abuse ceiling actually lives. It allows a few of
* those bulk passes per hour, which is more than a person driving a file
* manager will ever need and well under what a script grinding metadata writes
* would want.
*/
export const FS_MUTATE_LIMIT: RouteRateLimit[] = [
userWindow('fs:mutate', 1200, 900, 600),
userWindow('fs:mutate-sustained', 6000, 3000, 1800, 60 * 60_000),
];
/** Mints a URL that outlives the request, so worth its own budget. */
export const FS_SIGN_LIMIT = userWindow('fs:sign', 300, 150, 60);
/** Low-frequency GUI helpers. */
export const FS_HELPER_LIMIT = userWindow('fs:helper', 120, 60, 30);
/**
* Puter.js polls this on a timer to decide whether to purge its FS cache. Same
* ceiling for free and paid — it is a single cache read.
*/
export const FS_POLL_LIMIT = userWindow('fs:poll', 240, 240, 120);
// -- Legacy multipart upload -----------------------------------------
/**
* `/batch` buffers every file fully into memory before any quota or storage
* check runs, up to BATCH_MAX_FILES × BATCH_MAX_FILE_SIZE. The concurrency slot
* is doing the real work here; the window is secondary, and sized to say so —
* one upload is one call, so a per-minute ceiling in the tens is a cap on how
* many files someone may upload rather than a bound on cost. What actually
* bounds the memory this route can tie up is how many run at once.
*/
export const FS_BATCH_LIMIT = userWindow('fs:batch', 600, 300, 300);
export const FS_BATCH_CONCURRENT = userConcurrent('fs:batch', 5, 2, 2);
// -- Signed-URL routes (no session to key on) ------------------------
//
// The URL's own signature is what authorizes these; the limits below only
// bound what an unsigned flood can cost. Sized for what the routes are
// actually used for, which is content the browser fetches as a subresource:
// a gallery, a document's images, an app loading its own assets. A page
// opening a few dozen of those at once is ordinary, and every rejection here
// is a broken image rather than a slow one — so the ceiling clears a burst of
// real page loads and only catches something looping.
export const FS_SIGNED_READ_LIMIT = networkWindow('fs:signed-read', 3_000);
export const FS_SIGNED_WRITE_LIMIT = networkWindow('fs:signed-write', 600);
export const FS_SIGNED_CONCURRENT: NonNullable<RouteOptions['concurrent']> = {
scope: 'fs:signed',
// Above what a browser will open to one origin at once, so the in-flight
// cap never decides the outcome for a single client — it is there for a
// client that opens connections without closing them.
limit: 60,
key: 'fingerprint',
};
// -- WebDAV ----------------------------------------------------------
/**
* One `router.use` fronts the whole DAV surface, so a single gate there covers
* every verb. Desktop DAV clients are bursty — a lower ceiling shows up as
* spurious failures in Finder / Explorer.
*
* Unlike everything above, the DAV gate runs before the request is
* authenticated: there is no actor yet, so there is no subscription to resolve
* a tier against and one ceiling applies to every caller. These are shaped to
* say that — keyed on the network fingerprint the gate buckets on, with no
* `bySubscription` map to imply a tiering that never happens.
*/
export const DAV_LIMIT: RouteRateLimit = {
scope: 'dav',
limit: 600,
window: 60_000,
key: 'fingerprint',
};
export const DAV_CONCURRENT: NonNullable<RouteOptions['concurrent']> = {
scope: 'dav',
limit: 10,
key: 'fingerprint',
};
@@ -19,6 +19,10 @@
import { HttpError } from '../../core/http/HttpError.js';
import { PuterController } from '../types.js';
import {
DEFAULT_FREE_SUBSCRIPTION,
DEFAULT_TEMP_SUBSCRIPTION,
} from '../../services/metering/consts.js';
/**
* Site hosting endpoints. Listing and create/update are not exposed as
@@ -46,6 +50,18 @@ export class HostingController extends PuterController {
requireUserActor: true,
allowFullAccessToken: true,
requireVerified: true,
// Destructive, and pairs with the `subdomains:create`
// budget on the driver side.
rateLimit: {
scope: 'delete-site',
limit: 60,
window: 60_000,
key: 'user',
bySubscription: {
[DEFAULT_FREE_SUBSCRIPTION]: 30,
[DEFAULT_TEMP_SUBSCRIPTION]: 10,
},
},
},
async (req, res) => {
const { site_uuid } = req.body ?? {};
+2
View File
@@ -18,6 +18,7 @@
*/
import { AppController } from './apps/AppController.js';
import { AppFeedbackController } from './feedback/AppFeedbackController.js';
import { AuthController } from './auth/AuthController.js';
import { BroadcastController } from './broadcast/BroadcastController.js';
import { DesktopController } from './desktop/DesktopController.js';
@@ -43,6 +44,7 @@ export const puterControllers = {
staticPages: StaticPagesController,
auth: AuthController,
apps: AppController,
appFeedback: AppFeedbackController,
desktop: DesktopController,
hosting: HostingController,
system: SystemController,
@@ -41,6 +41,14 @@ export class NotificationController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
// Fires per notification interaction, so the ceiling stays
// generous — it is here to catch a loop, not to pace a user.
rateLimit: {
scope: 'notification-mark',
limit: 300,
window: 60_000,
key: 'user',
},
})
async markAck(req: Request, res: Response): Promise<void> {
const uid = req.body?.uid;
@@ -56,8 +64,7 @@ export class NotificationController extends PuterController {
});
const notifService = this.services.notification as unknown as
| NotificationService
| undefined;
NotificationService | undefined;
if (notifService?.markAcknowledged) {
await notifService.markAcknowledged(uid, userId);
} else {
@@ -84,6 +91,14 @@ export class NotificationController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
// Fires per notification interaction, so the ceiling stays
// generous — it is here to catch a loop, not to pace a user.
rateLimit: {
scope: 'notification-mark',
limit: 300,
window: 60_000,
key: 'user',
},
})
async markRead(req: Request, res: Response): Promise<void> {
const uid = req.body?.uid;
@@ -99,8 +114,7 @@ export class NotificationController extends PuterController {
});
const notifService = this.services.notification as unknown as
| NotificationService
| undefined;
NotificationService | undefined;
if (notifService?.markShown) {
await notifService.markShown(uid, userId);
} else {
@@ -1746,3 +1746,35 @@ describe('OIDCController POST /auth/oidc/verify-popup-return', () => {
expect((await redeem(proof)).body).toMatchObject({ oidc_login: false });
});
});
// ── rate-limit scopes ───────────────────────────────────────────────
describe('OIDCController rate limits', () => {
const rateLimitOf = (method: string, path: string) => {
const route = router.routes.find(
(r) => r.method === method && r.path === path,
);
if (!route) throw new Error(`No ${method.toUpperCase()} ${path} route`);
return route.options.rateLimit as { scope: string; limit: number };
};
it('keeps the revalidate landing page off the identity-provider bucket', () => {
// `/auth/revalidate-done` serves a constant HTML page; the start and
// callback routes exchange codes with an identity provider. Sharing
// a scope made the cheap page inherit the flow routes' tight ceiling,
// which one shared address can exhaust on its own.
const done = rateLimitOf('get', '/auth/revalidate-done');
const start = rateLimitOf('get', '/auth/oidc/:provider/start');
expect(done.scope).not.toBe(start.scope);
expect(done.scope).toBe('oidc-revalidate-done');
expect(done.limit).toBeGreaterThan(start.limit);
});
it('sizes the public provider list for a shared address, not a fleet', () => {
// Unauthenticated and keyed on IP, so the bucket covers every client
// behind one NAT or campus, not one browser. It is still login
// surface, so it stays bounded well below the ceilings given to
// static reads like icons or version info.
expect(rateLimitOf('get', '/auth/oidc/providers').limit).toBe(1_200);
});
});
+34 -2
View File
@@ -228,10 +228,26 @@ export class OIDCController extends PuterController {
// -- GET /auth/oidc/providers --------------------------------
// Public — list enabled provider IDs for the frontend.
//
// Every render of a login form reads this, and with no actor the
// bucket is the address: one corporate egress, campus or carrier
// gateway stands for every person behind it. Size it for that — a
// large shared address is thousands of people, and a shift or class
// change bunches their sign-ins into the same minute — but no
// further. This is login surface, so the ceiling should still bound
// someone enumerating which identity providers a deployment accepts.
router.get(
'/auth/oidc/providers',
{ subdomain: 'api' },
{
subdomain: 'api',
rateLimit: {
scope: 'oidc-providers',
limit: 1_200,
window: 60_000,
key: 'ip',
},
},
async (_req: Request, res: Response) => {
const providers =
await this.services.oidc.getEnabledProviderIds();
@@ -584,10 +600,26 @@ export class OIDCController extends PuterController {
// -- GET /auth/revalidate-done -------------------------------
// Landing page after revalidation; posts to opener for popup flow.
//
// Deliberately not on the `oidc-general` bucket the start/callback
// routes share: those exchange codes with an identity provider, this
// one is a constant HTML page that touches nothing. Sharing a bucket
// meant everyone reachable through one address — an office, a school,
// a carrier gateway — competed for the same 30 popup closes a minute.
// The replacement is sized for the humans behind one such address
// re-validating at once, not for a fleet: it is still auth surface,
// and one page view per revalidation is a low-volume event.
router.get(
'/auth/revalidate-done',
{ subdomain: '' },
{
subdomain: '',
rateLimit: {
scope: 'oidc-revalidate-done',
limit: 600,
window: 60_000,
},
},
(_req: Request, res: Response) => {
const origin = this.config.origin ?? '';
res.set('Content-Type', 'text/html; charset=utf-8');
@@ -438,6 +438,7 @@ describe('PeerController TURN', () => {
id: created.id,
username: created.username,
},
effectiveApp: null,
});
} finally {
meterSpy.mockRestore();
+51 -6
View File
@@ -19,11 +19,15 @@
import { createHmac, randomBytes, timingSafeEqual } from 'node:crypto';
import type { Request, Response } from 'express';
import type { Actor } from '../../core/actor.js';
import { makeActor, type Actor } from '../../core/actor.js';
import { HttpError } from '../../core/http/HttpError.js';
import type { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterController } from '../types.js';
import { PEER_COSTS } from './costs.js';
import {
DEFAULT_FREE_SUBSCRIPTION,
DEFAULT_TEMP_SUBSCRIPTION,
} from '../../services/metering/consts.js';
/**
* Constant-time secret comparison for the internal-auth header. HMAC both sides
@@ -103,17 +107,58 @@ export class PeerController extends PuterController {
registerRoutes(router: PuterRouter): void {
router.get(
'/peer/signaller-info',
{ subdomain: 'api' },
{
subdomain: 'api',
// Public config read — the signaller URL and the fallback
// ICE list, both deploy constants. Unauthenticated, so the
// key is the address, and one address covers every client
// on that network; a peer session starts with this call, so
// the bucket has to hold a whole network's sessions. Nothing
// here is secret or expensive, so the ceiling only bounds a
// client stuck re-reading it.
rateLimit: {
scope: 'peer-signaller-info',
limit: 3_000,
window: 60_000,
key: 'ip',
},
},
this.#signallerInfo,
);
router.post(
'/peer/generate-turn',
{ subdomain: 'api', requireAuth: true },
{
subdomain: 'api',
requireAuth: true,
// Every call reaches the upstream TURN API and mints
// credentials against a paid allocation, so this is a
// spend limit as much as an abuse limit.
rateLimit: {
scope: 'peer-generate-turn',
limit: 30,
window: 60_000,
key: 'user',
bySubscription: {
[DEFAULT_FREE_SUBSCRIPTION]: 10,
[DEFAULT_TEMP_SUBSCRIPTION]: 5,
},
},
},
this.#generateTurn,
);
router.post(
'/turn/ingest-usage',
{ subdomain: 'api' },
{
subdomain: 'api',
// Shared-secret authenticated, so this only bounds how
// fast someone can guess the secret.
rateLimit: {
scope: 'turn-ingest',
limit: 60,
window: 60_000,
key: 'ip',
},
},
this.#ingestUsage,
);
}
@@ -217,13 +262,13 @@ export class PeerController extends PuterController {
if (!user) continue;
const costInMicrocents =
egressBytes * PEER_COSTS['turn:egress-bytes'];
const actor = {
const actor = makeActor({
user: {
uuid: user.uuid,
id: user.id,
username: user.username,
},
};
});
await this.services.metering.incrementUsage(
actor,
'turn:egress-bytes',
@@ -229,9 +229,18 @@ describe('PuterAIController.registerRoutes', () => {
const modelsRoute = calls.find(
(c) => c.path === '/puterai/chat/models',
);
// Unauthenticated, so the limit keys on IP rather than an actor —
// which makes the bucket an aggregate over every client behind that
// address, hence a ceiling sized for a network rather than a browser.
expect(modelsRoute?.opts).toEqual({
subdomain: 'api',
requireAuth: false,
rateLimit: {
scope: 'puterai-models',
limit: 3_000,
window: 60_000,
key: 'ip',
},
});
});
@@ -90,7 +90,23 @@ export class PuterAIController extends PuterController {
key: aiPolicyKey,
},
} as RouteOptions;
const publicOpts = { subdomain: 'api', requireAuth: false } as const;
// Model listings are unauthenticated, so the only key available is
// the address — which is an aggregate, not a user: a NAT, a school,
// a mobile carrier gateway or a server-side renderer all arrive as
// one address, and the SDK and GUI both fetch the catalogue on
// startup. The ceiling therefore has to cover a whole network's page
// loads. What it still protects is the serialisation cost of the
// catalogue under a client stuck in a fetch loop.
const publicOpts = {
subdomain: 'api',
requireAuth: false,
rateLimit: {
scope: 'puterai-models',
limit: 3_000,
window: 60_000,
key: 'ip',
},
} as RouteOptions;
// Every route below carries the `/puterai` prefix for wire
// compatibility with puter-js and existing API tests.
@@ -153,7 +169,23 @@ export class PuterAIController extends PuterController {
// URL itself is HMAC-signed, so no additional auth gate.
router.get(
'/puterai/video/proxy',
{ subdomain: 'api' },
{
subdomain: 'api',
// HMAC-signed but unauthenticated, and it streams provider
// bandwidth through us — so the in-flight cap matters as
// much as the window.
rateLimit: {
scope: 'puterai-video-proxy',
limit: 60,
window: 60_000,
key: 'ip',
},
concurrent: {
scope: 'puterai-video-proxy',
limit: 5,
key: 'ip',
},
},
this.#videoProxy,
);
}
@@ -542,8 +574,7 @@ export class PuterAIController extends PuterController {
text: extractTextContent(
(
messageResult.message as
| Record<string, unknown>
| undefined
Record<string, unknown> | undefined
)?.content,
),
index: 0,
@@ -32,6 +32,17 @@ import { promoteToVerifiedGroup } from '../../util/userProvisioning.js';
* All root-subdomain-only, all unauthenticated (the confirm/unsubscribe tokens
* in the query string are the auth).
*/
/**
* Unauthenticated pages reached from a token-bearing email link. Matches what
* the emailSend extension already applies to its own link pages.
*/
const TOKEN_LINK_LIMIT = {
scope: 'token-link-page',
limit: 30,
window: 60_000,
key: 'ip' as const,
};
export class StaticPagesController extends PuterController {
registerRoutes(router: PuterRouter) {
const origin = this.config.origin ?? '';
@@ -159,151 +170,178 @@ export class StaticPagesController extends PuterController {
});
// -- /sitemap.xml --------------------------------------------
router.get('/sitemap.xml', {}, async (req, res) => {
const domain = this.config.domain ?? req.hostname;
const origin = `${req.protocol}://${domain}`;
const apps = (await this.clients.db.read(
`SELECT \`name\` FROM \`apps\` WHERE \`approved_for_listing\` = ${this.clients.db.booleanLiteral(true)}`,
)) as Array<{ name: string }>;
const urls = [
`<url><loc>${req.protocol}://docs.${domain}/</loc></url>`,
...apps.map(
(a) => `<url><loc>${origin}/app/${a.name}</loc></url>`,
),
];
const body =
'<?xml version="1.0" encoding="UTF-8"?>' +
'<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">' +
urls.join('') +
'</urlset>';
res.type('application/xml').send(body);
});
router.get(
'/sitemap.xml',
{
// Unauthenticated and runs a full-table scan over approved
// apps on every request, with no response cache in front.
rateLimit: {
scope: 'sitemap',
limit: 10,
window: 60_000,
key: 'ip',
},
},
async (req, res) => {
const domain = this.config.domain ?? req.hostname;
const origin = `${req.protocol}://${domain}`;
const apps = (await this.clients.db.read(
`SELECT \`name\` FROM \`apps\` WHERE \`approved_for_listing\` = ${this.clients.db.booleanLiteral(true)}`,
)) as Array<{ name: string }>;
const urls = [
`<url><loc>${req.protocol}://docs.${domain}/</loc></url>`,
...apps.map(
(a) => `<url><loc>${origin}/app/${a.name}</loc></url>`,
),
];
const body =
'<?xml version="1.0" encoding="UTF-8"?>' +
'<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">' +
urls.join('') +
'</urlset>';
res.type('application/xml').send(body);
},
);
// -- /unsubscribe --------------------------------------------
router.get('/unsubscribe', {}, async (req, res) => {
const userUuid =
typeof req.query.user_uuid === 'string'
? req.query.user_uuid
: undefined;
if (!userUuid) {
res.send(err('user_uuid is required'));
return;
}
router.get(
'/unsubscribe',
{ rateLimit: TOKEN_LINK_LIMIT },
async (req, res) => {
const userUuid =
typeof req.query.user_uuid === 'string'
? req.query.user_uuid
: undefined;
if (!userUuid) {
res.send(err('user_uuid is required'));
return;
}
const user = await this.stores.user.getByUuid(userUuid);
if (!user) {
res.send(err('User not found.'));
return;
}
if (user.unsubscribed) {
res.send(ok('You are already unsubscribed.'));
return;
}
const user = await this.stores.user.getByUuid(userUuid);
if (!user) {
res.send(err('User not found.'));
return;
}
if (user.unsubscribed) {
res.send(ok('You are already unsubscribed.'));
return;
}
await this.stores.user.update(user.id, { unsubscribed: 1 });
res.send(ok('You have successfully unsubscribed from all emails.'));
});
await this.stores.user.update(user.id, { unsubscribed: 1 });
res.send(
ok('You have successfully unsubscribed from all emails.'),
);
},
);
// -- /confirm-email-by-token ---------------------------------
router.get('/confirm-email-by-token', {}, async (req, res) => {
const userUuid =
typeof req.query.user_uuid === 'string'
? req.query.user_uuid
: undefined;
const token =
typeof req.query.token === 'string'
? req.query.token
: undefined;
if (!userUuid) {
res.send(err('user_uuid is required'));
return;
}
if (!token) {
res.send(err('token is required'));
return;
}
router.get(
'/confirm-email-by-token',
{ rateLimit: TOKEN_LINK_LIMIT },
async (req, res) => {
const userUuid =
typeof req.query.user_uuid === 'string'
? req.query.user_uuid
: undefined;
const token =
typeof req.query.token === 'string'
? req.query.token
: undefined;
if (!userUuid) {
res.send(err('user_uuid is required'));
return;
}
if (!token) {
res.send(err('token is required'));
return;
}
const user = await this.stores.user.getByProperty(
'uuid',
userUuid,
{ force: true },
);
if (!user) {
res.send(err('user not found.'));
return;
}
if (user.email_confirmed) {
res.send(ok('Email already confirmed.'));
return;
}
if (user.email_confirm_token !== token) {
res.send(err('invalid token.'));
return;
}
const user = await this.stores.user.getByProperty(
'uuid',
userUuid,
{ force: true },
);
if (!user) {
res.send(err('user not found.'));
return;
}
if (user.email_confirmed) {
res.send(ok('Email already confirmed.'));
return;
}
if (user.email_confirm_token !== token) {
res.send(err('invalid token.'));
return;
}
// v2 writes `clean_email` at signup (lowercased email). Older rows
// that predate that may be null — fall back to email.lower().
const cleanEmail =
(user.clean_email as string | null | undefined) ??
String(user.email ?? '').toLowerCase();
// v2 writes `clean_email` at signup (lowercased email). Older rows
// that predate that may be null — fall back to email.lower().
const cleanEmail =
(user.clean_email as string | null | undefined) ??
String(user.email ?? '').toLowerCase();
const [dupe] = (await this.clients.db.read(
`SELECT EXISTS(
const [dupe] = (await this.clients.db.read(
`SELECT EXISTS(
SELECT 1 FROM \`user\` WHERE (\`email\` = ? OR \`clean_email\` = ?)
AND \`email_confirmed\` = ${this.clients.db.booleanLiteral(true)}
AND \`password\` IS NOT NULL
) AS email_exists`,
[user.email, cleanEmail],
)) as Array<{ email_exists: number }>;
if (dupe?.email_exists) {
res.send(
err('This email was confirmed on a different account.'),
[user.email, cleanEmail],
)) as Array<{ email_exists: number }>;
if (dupe?.email_exists) {
res.send(
err('This email was confirmed on a different account.'),
);
return;
}
// Revoke any other accounts' pending change-email slots targeting
// this address — they're no longer valid once someone confirms it.
await this.clients.db.write(
'UPDATE `user` SET `unconfirmed_change_email` = NULL, `change_email_confirm_token` = NULL WHERE `unconfirmed_change_email` = ?',
[user.email],
);
return;
}
// Revoke any other accounts' pending change-email slots targeting
// this address — they're no longer valid once someone confirms it.
await this.clients.db.write(
'UPDATE `user` SET `unconfirmed_change_email` = NULL, `change_email_confirm_token` = NULL WHERE `unconfirmed_change_email` = ?',
[user.email],
);
await this.stores.user.update(user.id, {
email_confirmed: 1,
requires_email_confirmation: 0,
email_confirm_code: null,
email_confirm_token: null,
});
await this.stores.user.update(user.id, {
email_confirmed: 1,
requires_email_confirmation: 0,
email_confirm_code: null,
email_confirm_token: null,
});
await promoteToVerifiedGroup(this.stores.group, this.config, user);
// Best-effort side-channels — don't fail the user-visible response
// if sockets or the event bus are unavailable.
try {
await this.services.socket.send(
{ room: user.id },
'user.email_confirmed',
{},
await promoteToVerifiedGroup(
this.stores.group,
this.config,
user,
);
} catch {
/* ignore */
}
try {
this.clients.event?.emit(
'user.email-confirmed',
{
user_id: user.id,
user_uid: user.uuid,
email: user.email,
},
{},
);
} catch {
/* ignore */
}
res.send(ok('Your email has been successfully confirmed.'));
});
// Best-effort side-channels — don't fail the user-visible response
// if sockets or the event bus are unavailable.
try {
await this.services.socket.send(
{ room: user.id },
'user.email_confirmed',
{},
);
} catch {
/* ignore */
}
try {
this.clients.event?.emit(
'user.email-confirmed',
{
user_id: user.id,
user_uid: user.uuid,
email: user.email,
},
{},
);
} catch {
/* ignore */
}
res.send(ok('Your email has been successfully confirmed.'));
},
);
}
}
@@ -25,6 +25,75 @@ import { PuterController } from '../types.js';
*
* These are all low-risk, authenticated or not, and mostly stateless.
*/
/**
* Liveness polling. The callers here are infrastructure, not people: a load
* balancer, an orchestrator and any external uptime prober all poll this, and
* they typically egress from a small set of addresses. A 429 here is read as an
* unhealthy node and takes the node out of rotation, so the ceiling is set
* where only a runaway loop can reach it. The handler itself reads a status
* snapshot refreshed on a background timer, so the per-request cost is close to
* nil.
*
* `memory` rather than the shared default, and that choice is load-bearing:
*
* - This route decides whether a node stays in rotation, so it must not depend on
* anything it isn't already reporting on. The default backend is redis, and
* the cluster is configured with an offline queue and no per-command timeout
* — so while redis is unreachable a gated request waits on it rather than
* failing fast. The gate does fail open, but only once the call rejects, and
* the ALB gives a target 4s per probe and evicts after two. A redis
* degradation could therefore empty every target group in every region, which
* is the outcome the `@dependencies` degrade rules in the health check query
* exist to prevent. Keeping the counter in-process removes redis from the
* liveness path entirely.
* - Per-node counting is also the more honest bucket here. The ceiling only ever
* needs to cover the pollers hitting _this_ node, not (pollers x fleet size)
* as a shared counter does.
*/
const HEALTHCHECK_LIMIT = {
scope: 'healthcheck',
limit: 30_000,
window: 60_000,
key: 'ip',
backend: 'memory',
};
/**
* Deploy-constant build info, polled by clients. One address is a NAT, a
* campus, a proxy or a server-side renderer, so this bucket aggregates every
* client behind it — sizing it for a single browser would throttle a whole
* office. The response is cached per-client for a minute, which bounds each
* client to roughly one hit per window; the ceiling is what is left to catch a
* client that ignores the cache.
*/
const VERSION_LIMIT = {
scope: 'version',
limit: 6_000,
window: 60_000,
key: 'ip',
};
/**
* Deploy-constant deployment identity, read once per page load to decide
* whether to offer signup. Same aggregation as `/version` — the bucket is a
* whole network's worth of clients — and the payload is four constants, so the
* limit only guards against an unbounded client loop.
*/
const WHOAREWE_LIMIT = {
scope: 'whoarewe',
limit: 6_000,
window: 60_000,
key: 'ip',
};
/** Static introspection output, read once at boot rather than in a loop. */
const LSMOD_LIMIT = {
scope: 'lsmod',
limit: 60,
window: 60_000,
key: 'user',
};
export class SystemController extends PuterController {
constructor(config, clients, stores, services, drivers) {
super(config, clients, stores, services, drivers);
@@ -44,7 +113,10 @@ export class SystemController extends PuterController {
// `?ignore=a,b` disregards the named checks for this request only.
// `?marked-degraded=a,b` demotes the named checks to a non-fatal
// `degraded` list: `ok` stays true but the response is 207 so the
// caller can tell the node is running in a degraded state.
// caller can tell the node is running in a degraded state. Either list
// accepts `@<group>` to stand for every check in a group — notably
// `@dependencies` for the backing-service probes — so a caller polling
// this route doesn't have to enumerate them.
const parseNames = (value) =>
typeof value === 'string'
? value
@@ -52,44 +124,53 @@ export class SystemController extends PuterController {
.map((name) => name.trim())
.filter(Boolean)
: [];
router.get('/healthcheck', { subdomain: '*' }, async (req, res) => {
const health = this.services.health;
if (!health || typeof health.getStatus !== 'function') {
// Fallback for boot ordering / missing service.
return res.send('ok');
}
const status = await health.getStatus({
ignore: parseNames(req.query.ignore),
degrade: parseNames(req.query['marked-degraded']),
});
if (!status.ok) return res.status(503).json(status);
if (status.degraded?.length) return res.status(207).json(status);
return res.json(status);
});
router.get(
'/healthcheck',
{ subdomain: '*', rateLimit: HEALTHCHECK_LIMIT },
async (req, res) => {
const health = this.services.health;
if (!health || typeof health.getStatus !== 'function') {
// Fallback for boot ordering / missing service.
return res.send('ok');
}
const status = await health.getStatus({
ignore: parseNames(req.query.ignore),
degrade: parseNames(req.query['marked-degraded']),
});
if (!status.ok) return res.status(503).json(status);
if (status.degraded?.length)
return res.status(207).json(status);
return res.json(status);
},
);
// -- Version -------------------------------------------------
router.get('/version', { subdomain: '*' }, (_req, res) => {
const version =
this.config.version ??
process.env.npm_package_version ??
'unknown';
const parts = String(version).split('.');
// Deploy-constant, and callers poll it. Cache per-client only:
// a shared cache could pin one region's `location` for everyone,
// and the short window still bounds how long a client can miss a
// new deploy.
res.setHeader('Cache-Control', 'private, max-age=60');
res.json({
version,
major: parts[0] ? Number(parts[0]) : null,
minor: parts[1] ? Number(parts[1]) : null,
patch: parts[2] ? Number(parts[2]) : null,
environment: this.config.env ?? 'prod',
location: this.config.serverId ?? null,
deploy_timestamp: this.bootTime,
});
});
router.get(
'/version',
{ subdomain: '*', rateLimit: VERSION_LIMIT },
(_req, res) => {
const version =
this.config.version ??
process.env.npm_package_version ??
'unknown';
const parts = String(version).split('.');
// Deploy-constant, and callers poll it. Cache per-client only:
// a shared cache could pin one region's `location` for everyone,
// and the short window still bounds how long a client can miss a
// new deploy.
res.setHeader('Cache-Control', 'private, max-age=60');
res.json({
version,
major: parts[0] ? Number(parts[0]) : null,
minor: parts[1] ? Number(parts[1]) : null,
patch: parts[2] ? Number(parts[2]) : null,
environment: this.config.env ?? 'prod',
location: this.config.serverId ?? null,
deploy_timestamp: this.bootTime,
});
},
);
// -- Contact us ----------------------------------------------
@@ -153,7 +234,7 @@ export class SystemController extends PuterController {
// -- GET /whoarewe -------------------------------------------
router.get('/whoarewe', {}, (_req, res) => {
router.get('/whoarewe', { rateLimit: WHOAREWE_LIMIT }, (_req, res) => {
res.json({
name: 'Puter',
version: this.config.version ?? null,
@@ -181,8 +262,16 @@ export class SystemController extends PuterController {
}
res.json({ interfaces });
};
router.get('/lsmod', { subdomain: 'api', requireAuth: true }, lsmod);
router.post('/lsmod', { subdomain: 'api', requireAuth: true }, lsmod);
router.get(
'/lsmod',
{ subdomain: 'api', requireAuth: true, rateLimit: LSMOD_LIMIT },
lsmod,
);
router.post(
'/lsmod',
{ subdomain: 'api', requireAuth: true, rateLimit: LSMOD_LIMIT },
lsmod,
);
}
onServerStart() {}
@@ -141,6 +141,19 @@ const callRoute = async (
// ── /healthcheck ────────────────────────────────────────────────────
describe('SystemController GET /healthcheck', () => {
// This route decides whether a node stays in rotation, so its rate limit
// must not reach for a backing service. The default backend is redis,
// whose client queues rather than fails fast while it is unreachable —
// enough to push a probe past the 4s the load balancer allows and evict
// every target during a redis degradation. In-process counting keeps the
// liveness path free of anything it is itself reporting on.
it('counts in-process, so liveness never waits on redis', () => {
const route = router.routes.find(
(r) => r.method === 'get' && r.path === '/healthcheck',
);
expect(route?.options.rateLimit?.backend).toBe('memory');
});
it('returns the live ServerHealthService status payload', async () => {
const { res, captured } = makeRes();
await callRoute('get', '/healthcheck', makeReq({}), res);
@@ -461,6 +474,52 @@ describe('SystemController GET /lsmod', () => {
});
});
// ── rate-limit scopes ───────────────────────────────────────────────
describe('SystemController public route rate limits', () => {
const rateLimitOf = (path: string) => {
const route = router.routes.find(
(r) => r.method === 'get' && r.path === path,
);
if (!route) throw new Error(`No GET ${path} route`);
return route.options.rateLimit as {
scope: string;
limit: number;
window: number;
key: string;
};
};
it('gives /healthcheck, /version and /whoarewe separate buckets', () => {
// These once shared one scope, which meant clients polling /version
// could exhaust the budget that liveness probes depend on. Keep them
// apart: a 429 on /healthcheck is read as an unhealthy node.
const scopes = [
rateLimitOf('/healthcheck').scope,
rateLimitOf('/version').scope,
rateLimitOf('/whoarewe').scope,
];
expect(new Set(scopes).size).toBe(3);
expect(scopes).toEqual(['healthcheck', 'version', 'whoarewe']);
});
it('sizes the unauthenticated buckets for a shared address, not one client', () => {
// All three key on IP, and an IP is a NAT, a campus or a carrier
// gateway — the bucket aggregates everyone behind it, and the
// limiter counts region-wide rather than per process.
for (const path of ['/healthcheck', '/version', '/whoarewe']) {
const limit = rateLimitOf(path);
expect(limit.key).toBe('ip');
expect(limit.window).toBe(60_000);
expect(limit.limit).toBeGreaterThanOrEqual(6_000);
}
// Liveness polling is the most generous of the three by design.
expect(rateLimitOf('/healthcheck').limit).toBe(30_000);
expect(rateLimitOf('/version').limit).toBe(6_000);
expect(rateLimitOf('/whoarewe').limit).toBe(6_000);
});
});
// ── lifecycle ───────────────────────────────────────────────────────
describe('SystemController.onServerPrepareShutdown', () => {
@@ -64,7 +64,20 @@ const makeRes = () => {
headers: {},
ended: false,
};
const listeners: Record<string, Array<() => void>> = {};
const res = {
// The DAV mount holds a concurrency slot for the life of the request
// and releases it on `finish` / `close`, so the stub has to behave
// like an emitter or every dispatch throws.
once: vi.fn((event: string, fn: () => void) => {
(listeners[event] ??= []).push(fn);
return res;
}),
emit: vi.fn((event: string) => {
const fns = listeners[event] ?? [];
listeners[event] = [];
for (const fn of fns) fn();
}),
json: vi.fn((value: unknown) => {
captured.body = value;
return res;
@@ -89,10 +102,12 @@ const makeRes = () => {
}),
send: vi.fn((value: unknown) => {
captured.body = value;
res.emit('finish');
return res;
}),
end: vi.fn(() => {
captured.ended = true;
res.emit('finish');
return res;
}),
headersSent: false,
@@ -962,6 +977,11 @@ describe('WebDAVController verbs', () => {
send: (value: unknown) => {
captured.body = value;
captured.ended = true;
// End the underlying Writable so `finish` fires, as it does
// on a real response. The DAV mount releases its concurrency
// slot on that event — without it every `send()` path would
// leak a slot and later requests would 429.
if (!sink.writableEnded) sink.end();
return res;
},
headersSent: false,
@@ -21,7 +21,7 @@ import { compare as bcryptCompare } from 'bcrypt';
import type { Request, Response } from 'express';
import { posix as pathPosix } from 'node:path';
import { EventMap } from '../../clients/event/types.js';
import type { Actor } from '../../core/actor.js';
import { makeActor, type Actor } from '../../core/actor.js';
import { HttpError } from '../../core/http/HttpError.js';
import {
assertNotSuspended,
@@ -42,6 +42,13 @@ import {
hasWritePermission,
refreshLock,
} from './locks.js';
import { DAV_CONCURRENT, DAV_LIMIT } from '../fs/limits.js';
import {
acquireConcurrent,
checkRateLimit,
computeNetworkFingerprint,
} from '../../core/http/middleware/rateLimit.js';
import { assertActorHasCredits } from '../../services/metering/enforcement.js';
const DAV_HEADERS = {
DAV: '1, 2, ordered-collections',
@@ -54,6 +61,13 @@ const ALLOW_METHODS =
// macOS creates these files; reject them to keep the FS clean.
const MACOS_JUNK_REGEX = /(?:^\.DS_Store$|^\._)/;
/**
* Verbs that move file content or duplicate it in the object store, and so are
* refused to an account with nothing left of its budget. HEAD is here with GET
* because a client asking for headers is a client about to fetch the body.
*/
const CREDIT_GATED_DAV_METHODS = new Set(['GET', 'HEAD', 'PUT', 'COPY']);
/**
* WebDAV controller — full RFC 4918 surface on the `dav.*` subdomain.
*
@@ -70,10 +84,17 @@ export class WebDAVController extends PuterController {
// Single catch-all on the `dav` subdomain. We dispatch by req.method
// inside the handler because WebDAV uses non-standard HTTP verbs that
// Express doesn't have first-class router methods for in all versions.
//
// The rate limit is applied inside the handler rather than through
// `RouteOptions`. For a `use` mount the subdomain check lives in the
// handler wrapper, not in the middleware chain — so a `rateLimit`
// here would run for every request on every subdomain and count
// non-DAV traffic against the DAV budget.
router.use(
{ subdomain: 'dav' },
async (req: Request, res: Response, _next) => {
try {
if (!(await this.#admit(req, res))) return;
await this.#dispatch(req, res);
} catch (err) {
if (err instanceof HttpError) {
@@ -88,6 +109,41 @@ export class WebDAVController extends PuterController {
);
}
/**
* Rate + concurrency gate for the whole DAV surface. Returns false when the
* request was rejected (429 already sent).
*
* Runs before `#dispatch` authenticates, so it keys on the network
* fingerprint rather than an actor. That is the coarser bucket, but a DAV
* client sends credentials on every request anyway — there is no
* unauthenticated browsing phase to protect a per-user key from.
*/
async #admit(req: Request, res: Response): Promise<boolean> {
const key = computeNetworkFingerprint(req);
if (
!(await checkRateLimit(
`${DAV_LIMIT.scope}:${key}`,
DAV_LIMIT.limit,
DAV_LIMIT.window,
))
) {
res.status(429).send('Too many requests.');
return false;
}
const slot = await acquireConcurrent(
`${DAV_CONCURRENT.scope}:${key}`,
DAV_CONCURRENT.limit,
);
if (!slot.ok) {
res.status(429).send('Too many concurrent requests.');
return false;
}
// `finish` and `close` can both fire; release is once-only.
res.once('finish', () => void slot.release());
res.once('close', () => void slot.release());
return true;
}
async #dispatch(req: Request, res: Response): Promise<void> {
// Authenticate
const actor = await this.#resolveActor(req, res);
@@ -104,6 +160,19 @@ export class WebDAVController extends PuterController {
assertNotSuspended(actor.user);
assertVerifiedAccount(actor.user);
// And the same budget gate the FS routes declare with
// `requireCredits`, for the verbs that move content — DAV serves the
// same files over a metered host, so leaving it out would make mounting
// the drive the way around enforcement. The verbs that only describe or
// remove things stay open, as they do over HTTP.
if (CREDIT_GATED_DAV_METHODS.has(req.method.toUpperCase())) {
await assertActorHasCredits(
this.services.metering,
actor,
this.config,
);
}
// Expand `~`/`~/...` against the authenticated actor's username.
// WebDAV doesn't standardize `~`, but some clients do — and the
// pre-existing behaviour silently expanded it via the FS store.
@@ -238,7 +307,7 @@ export class WebDAVController extends PuterController {
}
// Build a session-less actor for the user
return {
return makeActor({
user: {
id: user.id,
uuid: user.uuid,
@@ -258,7 +327,7 @@ export class WebDAVController extends PuterController {
requires_card_verification:
user.requires_card_verification ?? false,
},
};
});
}
// -- OPTIONS ------------------------------------------------------
+33 -2
View File
@@ -21,6 +21,10 @@ import type { Request, Response } from 'express';
import { HttpError } from '../../core/http/HttpError.js';
import type { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterController } from '../types.js';
import {
DEFAULT_FREE_SUBSCRIPTION,
DEFAULT_TEMP_SUBSCRIPTION,
} from '../../services/metering/consts.js';
/**
* WISP relay token controller — create and verify short-lived JWT tokens for
@@ -32,12 +36,39 @@ export class WispController extends PuterController {
registerRoutes(router: PuterRouter): void {
router.post(
'/wisp/relay-token/create',
{ subdomain: 'api', requireAuth: true },
{
subdomain: 'api',
requireAuth: true,
// Auth is optional in practice (the handler tolerates an
// anonymous actor), so the key falls back to a fingerprint
// when there is no user to key on.
rateLimit: {
scope: 'wisp-token-create',
limit: 60,
window: 60_000,
key: 'user',
bySubscription: {
[DEFAULT_FREE_SUBSCRIPTION]: 30,
[DEFAULT_TEMP_SUBSCRIPTION]: 10,
},
},
},
this.#create,
);
router.post(
'/wisp/relay-token/verify',
{ subdomain: 'api', requireAuth: false },
{
subdomain: 'api',
requireAuth: false,
// Unauthenticated by design, which makes it a token-guessing
// oracle without a ceiling.
rateLimit: {
scope: 'wisp-token-verify',
limit: 300,
window: 60_000,
key: 'ip',
},
},
this.#verify,
);
}
+104
View File
@@ -0,0 +1,104 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { describe, expect, it } from 'vitest';
import {
assertResolvedActor,
makeActor,
SYSTEM_ACTOR,
userRelatedActor,
type Actor,
} from './actor';
describe('makeActor / effectiveApp', () => {
const user = { uuid: 'u-1', id: 1, username: 'u' };
it('resolves a plain user actor to no app', () => {
expect(makeActor({ user }).effectiveApp).toBeNull();
});
it("resolves an app-under-user actor to its own app", () => {
expect(makeActor({ user, app: { uid: 'app-1' } }).effectiveApp).toEqual({
uid: 'app-1',
});
});
it("resolves a token to the app that issued it", () => {
// The whole point: a token actor carries no `app` of its own, so
// anything reading `app` sees a bare user token and skips app gating.
const issuer = makeActor({ user, app: { uid: 'app-1' } });
const token = makeActor({
user,
accessToken: { uid: 'tok-1', issuer },
});
expect(token.app).toBeUndefined();
expect(token.effectiveApp).toEqual({ uid: 'app-1' });
});
it('collapses a chain of tokens in one hop', () => {
const issuer = makeActor({ user, app: { uid: 'app-1' } });
const inner = makeActor({ user, accessToken: { uid: 't1', issuer } });
const outer = makeActor({
user,
accessToken: { uid: 't2', issuer: inner },
});
expect(outer.effectiveApp).toEqual({ uid: 'app-1' });
});
it('resolves a user-issued token to no app', () => {
const issuer = makeActor({ user });
const token = makeActor({
user,
accessToken: { uid: 'tok-1', issuer, fullAccess: true },
});
expect(token.effectiveApp).toBeNull();
});
it('drops the app when narrowing to the underlying user', () => {
const app = makeActor({ user, app: { uid: 'app-1' } });
expect(userRelatedActor(app).effectiveApp).toBeNull();
});
it('resolves the system actor', () => {
expect(SYSTEM_ACTOR.effectiveApp).toBeNull();
});
});
describe('assertResolvedActor', () => {
it('passes a resolved actor through unchanged', () => {
const actor = makeActor({ user: { uuid: 'u-1' } });
expect(assertResolvedActor(actor)).toBe(actor);
// `null` is a resolved answer, not a missing one.
expect(assertResolvedActor({ user: {}, effectiveApp: null })).toEqual({
user: {},
effectiveApp: null,
});
});
it('throws on an actor that skipped makeActor', () => {
// The field is optional so pre-existing literals still compile, which
// means an unresolved one can reach a gate. Fail loudly at the edge
// rather than let a gate read `undefined` as "no app" and wave it
// through — an app-under-user actor is the dangerous case.
const unresolved = { user: { uuid: 'u-1' }, app: { uid: 'app-1' } };
expect(() => assertResolvedActor(unresolved as Actor)).toThrow(
/effectiveApp/,
);
});
});
+53 -11
View File
@@ -47,6 +47,26 @@ export interface ActorAccessToken {
export interface Actor {
user: Partial<UserRow>;
app?: ActorApp | null;
/**
* The app this actor ultimately acts as: its own `app`, or failing that the
* app of whoever issued its access token.
*
* Read this — not `app` — in any gate asking "which app is doing this?". An
* access-token actor carries no `app` of its own, so `app` alone reads as
* "no app" even for a token an app minted, and a gate keyed off it fails
* open exactly where it must not.
*
* `null` and `undefined` are not the same thing here:
*
* - `null` — resolved, and this actor is not acting as any app.
* - Absent — never resolved, because the actor skipped `makeActor`.
*
* A gate must not read the second as the first: that is the fail-open this
* field exists to prevent. Optional only so an actor literal that predates
* the field still compiles; `assertResolvedActor` is what keeps the request
* path honest, and `makeActor` is the one place the derivation lives.
*/
effectiveApp?: ActorApp | null;
/** True for the system actor; skips metering / quota tracking. */
system?: boolean;
accessToken?: ActorAccessToken | null;
@@ -67,9 +87,41 @@ export const SYSTEM_ACTOR_UUID = '5d4adce0-a381-4982-9c02-6e2540026238';
/** The default system actor used when no actor is supplied. */
export const SYSTEM_ACTOR: Actor = {
user: { uuid: SYSTEM_ACTOR_UUID, username: 'system' },
effectiveApp: null,
system: true,
};
/**
* Build an actor, deriving `effectiveApp` from its own app and, failing that,
* from the app of whoever issued its access token.
*
* The issuer was itself built here, so its chain is already collapsed — one hop
* is enough, and no caller has to walk anything.
*/
export const makeActor = (actor: Omit<Actor, 'effectiveApp'>): Actor => ({
...actor,
effectiveApp: actor.app ?? actor.accessToken?.issuer.effectiveApp ?? null,
});
/**
* Fail closed on an actor whose `effectiveApp` was never derived.
*
* Every actor on the request path is built by `AuthService` through
* `makeActor`, so this cannot fire in production — which is the point. It turns
* a future actor literal that skips the builder into a loud 500 at the edge
* rather than a silent bypass deep inside a gate that read `undefined` as "no
* app". Call it once, where the request actor is established.
*/
export const assertResolvedActor = (actor: Actor): Actor => {
if (actor.effectiveApp === undefined) {
throw new Error(
'actor was built without `makeActor`: `effectiveApp` is unresolved, ' +
'and app-scoped gates would read that as "no app"',
);
}
return actor;
};
export const isSystemActor = (actor: Actor | undefined | null): boolean => {
return !!actor?.system || actor?.user?.uuid === SYSTEM_ACTOR_UUID;
};
@@ -106,15 +158,5 @@ export const actorUid = (actor: Actor): string => {
*/
export const userRelatedActor = (actor: Actor): Actor => {
if (!actor.app && !actor.accessToken) return actor;
return { user: actor.user };
};
/**
* Walk the access-token issuer chain and return the first app identity found,
* or null if no app appears anywhere in the chain.
*/
export const effectiveActorApp = (actor: Actor): ActorApp | null => {
if (actor.app) return actor.app;
if (actor.accessToken) return effectiveActorApp(actor.accessToken.issuer);
return null;
return { user: actor.user, effectiveApp: null };
};
@@ -18,6 +18,7 @@
*/
import type { Actor } from '../actor';
import type { StorageOpCounts } from '../storageOps';
import type { TokenSource } from './types';
/**
@@ -103,6 +104,22 @@ declare global {
* middleware.
*/
cookies?: Record<string, string>;
/**
* Who this response's egress is billed to, when that isn't the
* actor who made the request. Set by handlers that serve one
* account's bytes to an unidentified caller — a hosted site's
* visitor being the case that matters. Takes precedence over
* `actor` in the egress middleware.
*/
egressActor?: Actor;
/**
* Object-store requests made while serving this request, by class.
* Tallied through `recordStorageOps` and billed when the response
* ends.
*/
storageOps?: StorageOpCounts;
}
}
}
@@ -21,7 +21,7 @@ import type { Request, Response } from 'express';
import jwt from 'jsonwebtoken';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import type { Actor } from '../../actor';
import { makeActor, type Actor } from '../../actor';
import type { AuthService } from '../../../services/auth/AuthService';
import { PuterServer } from '../../../server';
import { setupTestServer } from '../../../testUtil';
@@ -59,7 +59,16 @@ const makeStubAuth = (defaultActor: Actor | null = null): StubAuth => {
authenticate: async (token: string) => {
seenTokens.push(token);
if (nextResult === 'throw') throw new Error('verify failed');
return nextResult;
// The real service builds every actor through `makeActor`, and the
// probe asserts that contract — so the stub has to honour it too,
// rather than handing back a literal the probe rejects.
if (!('actor' in nextResult)) return nextResult;
// Resolve only if the fixture didn't: the probe asserts the
// `makeActor` contract the real service satisfies, but tests that
// check actor identity need the same object back.
return nextResult.actor.effectiveApp === undefined
? { ...nextResult, actor: makeActor(nextResult.actor) }
: nextResult;
},
// Back-compat wrapper for callers that still want Actor | null.
authenticateFromToken: async (token: string) => {
@@ -273,7 +282,7 @@ describe('createAuthProbe — tokenSource', () => {
['handshake', { handshakeQuery: { auth_token: 'tok' } }],
];
const actor: Actor = { user: { uuid: 'u-1' } };
const actor: Actor = makeActor({ user: { uuid: 'u-1' } });
it.each(cases)('records %s', async (expected, init) => {
const stub = makeStubAuth(actor);
@@ -523,7 +532,7 @@ describe('createAuthProbe — cookie reading', () => {
describe('createAuthProbe — actor attachment + failure tracking', () => {
it('attaches actor + token on a successful authenticate', async () => {
const actor: Actor = { user: { uuid: 'u-1' } };
const actor: Actor = makeActor({ user: { uuid: 'u-1' } });
const stub = makeStubAuth(actor);
const probe = createAuthProbe({ authService: stub.service });
const { req } = await runProbe(
@@ -22,6 +22,7 @@ import type {
AuthService,
ReauthReason,
} from '../../../services/auth/AuthService';
import { assertResolvedActor } from '../../actor';
import type { TokenSource } from '../types';
// Ensure the `Request.actor` / `Request.token` augmentation is in scope
@@ -155,7 +156,7 @@ export const createAuthProbe = (opts: AuthProbeOptions): RequestHandler => {
}
if (result.actor) {
req.actor = result.actor;
req.actor = assertResolvedActor(result.actor);
req.token = token;
req.tokenSource = source;
} else if (result.invalid) {
@@ -0,0 +1,50 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { RequestHandler } from 'express';
import type { IConfig } from '../../../types';
import {
assertActorHasCredits,
type CreditMeteringLike,
} from '../../../services/metering/enforcement.js';
import '../expressAugmentation';
/**
* Reject an authenticated caller with nothing left of their budget, for routes
* that opt in with `requireCredits`.
*
* Anonymous callers pass: the signed-URL routes authorize on the URL rather
* than a session, and there is no account to charge or turn away. So do worker
* sessions, unless configured otherwise — see `creditEnforcementExempt`.
*
* The answer comes from a short-lived per-actor cache in the metering service,
* so this normally costs nothing beyond a map lookup. That is what makes it
* affordable on routes that are called hundreds of times a minute.
*/
export const requireCreditsGate = (
metering: CreditMeteringLike | undefined,
config: IConfig,
): RequestHandler => {
return (req, _res, next) => {
assertActorHasCredits(metering, req.actor, config).then(
() => next(),
(err) => next(err),
);
};
};
@@ -0,0 +1,176 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import type { Actor } from '../../actor';
import { PERIOD_ESCAPE } from '../../../services/metering/consts.js';
import type { UsageByType } from '../../../services/metering/types';
import { setupPuterTestEnv, type PuterTestEnv } from '../../../testUtil.js';
/**
* Egress metering over real HTTP. The unit tests drive the middleware with a
* response double; only a listening server proves the byte counter survives the
* middleware stack it is installed under (compression included) and that the
* actor is resolvable by the time the response closes.
*/
describe('egress metering over HTTP', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const escape = (usageType: string) =>
usageType.replace(/\./g, PERIOD_ESCAPE);
const usageFor = async (actor: Actor): Promise<UsageByType> => {
await env.server.services.metering.flushBufferedUsages();
const { usage } =
await env.server.services.metering.getActorCurrentMonthUsageDetails(
actor,
);
return usage;
};
const actorFor = async (username: string): Promise<Actor> => {
const user = await env.server.stores.user.getByUsername(username);
return { user: user! } as Actor;
};
it('bills a file read to the reader, bytes and object-store request alike', async () => {
const { username, token } = env.users.user;
const actor = await actorFor(username);
const body = Buffer.from('x'.repeat(4096));
await env.server.services.fs.write(actor.user.id!, {
fileMetadata: {
path: `/${username}/Desktop/egress.txt`,
size: body.byteLength,
contentType: 'text/plain',
},
fileContent: body,
});
const before = await usageFor(actor);
const beforeEgress =
(before[escape('egress:bytes')] as { units?: number } | undefined)
?.units ?? 0;
const readUrl = new URL('/fs/read', env.apiOrigin);
readUrl.searchParams.set('path', `/${username}/Desktop/egress.txt`);
const read = await fetch(readUrl, {
headers: { Authorization: `Bearer ${token}` },
});
expect(read.status).toBe(200);
expect(await read.text()).toHaveLength(body.byteLength);
const after = await usageFor(actor);
const egress = after[escape('egress:bytes')] as {
units: number;
cost: number;
};
// Compression may shrink the payload on the wire, so the floor is the
// headers rather than the file — what matters is that the read was
// counted at all, and that it cost something.
expect(egress.units).toBeGreaterThan(beforeEgress);
expect(egress.cost).toBeGreaterThan(0);
const reads = after[escape('storage:read:ops')] as {
units: number;
};
expect(reads.units).toBeGreaterThanOrEqual(1);
});
it('bills a signed-URL read to the account whose file it is', async () => {
const { username, token } = env.users.other;
const actor = await actorFor(username);
const body = Buffer.from('y'.repeat(4096));
const path = `/${username}/Desktop/signed-egress.txt`;
await env.server.services.fs.write(actor.user.id!, {
fileMetadata: {
path,
size: body.byteLength,
contentType: 'text/plain',
},
fileContent: body,
});
const signed = await fetch(new URL('/sign', env.apiOrigin), {
method: 'POST',
headers: {
Authorization: `Bearer ${token}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({ items: [{ path, action: 'read' }] }),
});
expect(signed.status).toBe(200);
const readUrl = new URL(
(await signed.json()).signatures[0].read_url as string,
);
const before = await usageFor(actor);
const beforeEgress =
(before[escape('egress:bytes')] as { units?: number } | undefined)
?.units ?? 0;
// A signature proves access to the file, not who is asking — so this
// fetch carries no credential, and the owner is the only account there
// is to bill.
const fetched = await fetch(
new URL(`${readUrl.pathname}${readUrl.search}`, env.apiOrigin),
);
expect(fetched.status).toBe(200);
expect(await fetched.text()).toHaveLength(body.byteLength);
const after = await usageFor(actor);
const egress = after[escape('egress:bytes')] as {
units: number;
cost: number;
};
expect(egress.units).toBeGreaterThan(beforeEgress);
expect(egress.cost).toBeGreaterThan(0);
});
it('leaves root-origin asset traffic out of the actor’s usage', async () => {
const { username, token } = env.users.user;
const actor = await actorFor(username);
const before = await usageFor(actor);
const beforeEgress =
(before[escape('egress:bytes')] as { units?: number } | undefined)
?.units ?? 0;
const sdk = await fetch(new URL('/puter.js/v2', env.origin), {
headers: { Authorization: `Bearer ${token}` },
});
expect(sdk.status).toBe(200);
expect((await sdk.text()).length).toBeGreaterThan(1000);
const after = await usageFor(actor);
const afterEgress =
(after[escape('egress:bytes')] as { units?: number } | undefined)
?.units ?? 0;
expect(afterEgress).toBe(beforeEgress);
});
});
@@ -0,0 +1,254 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { EventEmitter } from 'node:events';
import type { Request, Response } from 'express';
import { describe, expect, it, vi } from 'vitest';
import type { Actor } from '../../actor';
import { SYSTEM_ACTOR } from '../../actor';
import {
EGRESS_COSTS,
STORAGE_OP_COSTS,
} from '../../../services/metering/costs';
import type { UsageInput } from '../../../services/metering/types';
import { createEgressMeteringMiddleware } from './egressMetering';
const actor = (uuid = 'user-1'): Actor =>
({ user: { uuid, username: 'u' } }) as Actor;
/**
* Response double that behaves like the parts of `http.ServerResponse` the
* middleware touches: writes go somewhere, `close` is emitted once the response
* is over, and headers are readable at that point.
*/
const makeRes = (headers: Record<string, string> = {}) => {
const emitter = new EventEmitter();
const written: unknown[] = [];
const res = Object.assign(emitter, {
write: vi.fn((chunk: unknown) => {
written.push(chunk);
return true;
}),
end: vi.fn((chunk?: unknown) => {
if (chunk !== undefined && typeof chunk !== 'function')
written.push(chunk);
return res;
}),
getHeaders: () => headers,
}) as unknown as Response & { written: unknown[] };
return Object.assign(res, { written });
};
const run = (
reqPartial: Partial<Request> & Record<string, unknown> = {},
headers: Record<string, string> = {},
) => {
const buffered: Array<{ actor: Actor; usages: UsageInput[] }> = [];
const bufferIncrementUsages = vi.fn((a: Actor, usages: UsageInput[]) => {
buffered.push({ actor: a, usages });
});
const middleware = createEgressMeteringMiddleware({
services: { metering: { bufferIncrementUsages } },
});
const req = {
subdomains: ['api'],
actor: actor(),
...reqPartial,
} as unknown as Request;
const res = makeRes(headers);
const next = vi.fn();
middleware(req, res, next);
return { req, res, next, buffered, bufferIncrementUsages };
};
const finish = (res: Response) => res.emit('close');
const usageOf = (usages: UsageInput[], usageType: string) =>
usages.find((u) => u.usageType === usageType);
describe('createEgressMeteringMiddleware', () => {
it('passes the request straight through', () => {
const { next, res } = run();
expect(next).toHaveBeenCalledOnce();
// The write hooks must not swallow the payload.
res.write(Buffer.from('abc'));
res.end('de');
expect((res as Response & { written: unknown[] }).written).toEqual([
Buffer.from('abc'),
'de',
]);
});
it('bills every byte written, plus the headers, at the response cost', () => {
const { res, buffered } = run({}, { 'content-type': 'text/plain' });
res.write(Buffer.alloc(1000));
res.end(Buffer.alloc(24));
finish(res);
expect(buffered).toHaveLength(1);
const egress = usageOf(buffered[0]!.usages, 'egress:bytes')!;
// Header estimate is small but non-zero, so the total is a floor.
expect(egress.usageAmount).toBeGreaterThan(1024);
expect(egress.usageAmount).toBeLessThan(1100);
expect(egress.costOverride).toBeCloseTo(
EGRESS_COSTS['egress:bytes'] * egress.usageAmount,
10,
);
});
it('counts string chunks by their encoded length, not their character count', () => {
const empty = run();
finish(empty.res);
const headerBytes = usageOf(
empty.buffered[0]!.usages,
'egress:bytes',
)!.usageAmount;
const { res, buffered } = run();
res.end('déjà');
finish(res);
const withBody = usageOf(
buffered[0]!.usages,
'egress:bytes',
)!.usageAmount;
expect(withBody - headerBytes).toBe(Buffer.byteLength('déjà'));
});
it('meters a response that died mid-stream for what it managed to send', () => {
const { res, buffered } = run();
res.write(Buffer.alloc(500));
// No end() — the connection dropped.
finish(res);
expect(
usageOf(buffered[0]!.usages, 'egress:bytes')!.usageAmount,
).toBeGreaterThan(500);
});
it('bills the object-store requests made while serving the response', () => {
const { req, res, buffered } = run();
(req as Request).storageOps = { write: 3, read: 2, delete: 5 };
res.end('x');
finish(res);
const usages = buffered[0]!.usages;
expect(usageOf(usages, 'storage:write:ops')).toMatchObject({
usageAmount: 3,
costOverride: STORAGE_OP_COSTS['storage:write:ops'] * 3,
});
expect(usageOf(usages, 'storage:read:ops')).toMatchObject({
usageAmount: 2,
});
// Removals are counted but free.
expect(usageOf(usages, 'storage:delete:ops')).toMatchObject({
usageAmount: 5,
costOverride: 0,
});
});
it('bills `egressActor` ahead of the requesting actor', () => {
const owner = actor('owner-1');
const { res, buffered } = run({
subdomains: ['some-site'],
actor: undefined,
egressActor: owner,
});
res.end('hello');
finish(res);
expect(buffered[0]!.actor).toBe(owner);
});
it('meters a host it otherwise ignores once a billing target is named', () => {
const { res, buffered } = run({
subdomains: [],
egressActor: actor('owner-1'),
});
res.end('hello');
finish(res);
expect(buffered).toHaveLength(1);
});
it('leaves first-party asset traffic unmetered', () => {
for (const subdomains of [[], ['js'], ['docs']]) {
const { res, bufferIncrementUsages } = run({ subdomains });
res.end(Buffer.alloc(5_000_000));
finish(res);
expect(bufferIncrementUsages).not.toHaveBeenCalled();
}
});
it('meters the dav surface alongside the api', () => {
const { res, bufferIncrementUsages } = run({ subdomains: ['dav'] });
res.end('hello');
finish(res);
expect(bufferIncrementUsages).toHaveBeenCalledOnce();
});
it('skips requests with no actor and the system actor', () => {
for (const req of [
{ actor: undefined },
{ actor: { user: {} } as Actor },
{ actor: SYSTEM_ACTOR },
]) {
const { res, bufferIncrementUsages } = run(req);
res.end('hello');
finish(res);
expect(bufferIncrementUsages).not.toHaveBeenCalled();
}
});
it('records nothing when metering is not installed', () => {
const middleware = createEgressMeteringMiddleware({ services: {} });
const req = {
subdomains: ['api'],
actor: actor(),
} as unknown as Request;
const res = makeRes();
const next = vi.fn();
middleware(req, res, next);
res.end('hello');
expect(() => finish(res)).not.toThrow();
expect(next).toHaveBeenCalledOnce();
});
it('never lets a metering failure escape into the response path', () => {
const bufferIncrementUsages = vi.fn(() => {
throw new Error('metering down');
});
const middleware = createEgressMeteringMiddleware({
services: { metering: { bufferIncrementUsages } },
});
const req = {
subdomains: ['api'],
actor: actor(),
} as unknown as Request;
const res = makeRes();
middleware(req, res, vi.fn());
res.end('hello');
expect(() => finish(res)).not.toThrow();
});
});
@@ -0,0 +1,227 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, RequestHandler, Response } from 'express';
import type { Actor } from '../../actor';
import { isSystemActor } from '../../actor';
import type { StorageOpClass } from '../../storageOps';
import {
EGRESS_COSTS,
STORAGE_OP_COSTS,
STORAGE_OP_USAGE_TYPES,
} from '../../../services/metering/costs.js';
import type { UsageInput } from '../../../services/metering/types';
import '../expressAugmentation';
/**
* Subset of the metering service this middleware needs. Metering is optional
* from here — a deployment without it still serves traffic.
*/
interface MeteringLike {
bufferIncrementUsages?: (actor: Actor, usages: UsageInput[]) => void;
}
interface Layers {
services: { metering?: MeteringLike };
}
/**
* Hosts whose responses are a user's own data or an account's app traffic, and
* so are billed to that account. Everything else the origin serves — the
* desktop shell, the SDK, the homepage, static assets — is Puter's own cost of
* being reachable and is deliberately not charged to whoever happens to be
* signed in while loading it.
*
* Hosted sites are not in here: they arrive on a per-site subdomain and opt in
* by naming who to bill (`req.egressActor`).
*/
const METERED_SUBDOMAINS = new Set(['api', 'dav']);
/**
* Rough size of the status line and headers, which never reach `res.write` and
* so can only be estimated. Sized as `NAME: value\r\n` per header plus the
* status line and the blank line that ends the block. Small next to any real
* payload, but responses that are almost all headers (a 204, a redirect) are
* common enough that ignoring it would under-count a chatty client by a wide
* margin.
*/
const estimateHeaderBytes = (res: Response): number => {
// "HTTP/1.1 200 OK\r\n" plus the "\r\n" that terminates the block.
let bytes = 19;
let headers: ReturnType<Response['getHeaders']>;
try {
headers = res.getHeaders();
} catch {
return bytes;
}
for (const [name, value] of Object.entries(headers)) {
if (value === undefined) continue;
const rendered = Array.isArray(value)
? value.join(', ')
: String(value);
bytes += name.length + rendered.length + 4;
}
return bytes;
};
const chunkBytes = (chunk: unknown, encoding: unknown): number => {
if (typeof chunk === 'string') {
return Buffer.byteLength(
chunk,
typeof encoding === 'string'
? (encoding as BufferEncoding)
: 'utf8',
);
}
if (chunk instanceof Uint8Array || Buffer.isBuffer(chunk)) {
return chunk.byteLength;
}
return 0;
};
/** Who the response's bytes are billed to, or undefined if nobody. */
const resolveEgressActor = (req: Request): Actor | undefined => {
const actor = req.egressActor ?? req.actor;
if (!actor?.user?.uuid) return undefined;
if (isSystemActor(actor)) return undefined;
return actor;
};
const isMeteredHost = (req: Request): boolean => {
// An explicit billing target is the opt-in for hosts that aren't metered
// by default, so honour it whatever the subdomain says.
if (req.egressActor) return true;
const subdomain = req.subdomains?.[req.subdomains.length - 1];
return !!subdomain && METERED_SUBDOMAINS.has(subdomain);
};
const storageOpUsages = (req: Request): UsageInput[] => {
const ops = req.storageOps;
if (!ops) return [];
const usages: UsageInput[] = [];
for (const [opClass, count] of Object.entries(ops)) {
if (!count || count <= 0) continue;
const usageType = STORAGE_OP_USAGE_TYPES[opClass as StorageOpClass];
if (!usageType) continue;
usages.push({
usageType,
usageAmount: count,
costOverride: STORAGE_OP_COSTS[usageType] * count,
});
}
return usages;
};
/**
* Meters what a request actually costs to serve: every byte written back to the
* client, plus the object-store requests made on its behalf.
*
* This is the only place egress is counted. Handlers that stream file content
* used to meter their own bytes, which measured the payload they handed to
* express rather than what left the process, missed every other response, and
* charged an increment per download. Counting here instead means one rule for
* all traffic, and bytes counted after compression has had its say.
*
* Install FIRST, ahead of the compression middleware: middleware that wraps
* `res.write` later ends up wrapping this one, so anything installed after
* compression sees the payload before it is compressed. The actor is read at
* the end of the response rather than here, by which time the auth probe has
* run.
*
* Never rejects, never delays the response: increments are handed to metering
* once the response is over, and metering buffers them rather than writing per
* request.
*/
export const createEgressMeteringMiddleware = (
layers: Layers,
): RequestHandler => {
return (req, res, next) => {
const metering = layers.services.metering;
if (!metering?.bufferIncrementUsages) {
next();
return;
}
let bodyBytes = 0;
const write = res.write.bind(res);
const end = res.end.bind(res);
res.write = ((
chunk: unknown,
encoding?: unknown,
callback?: unknown,
) => {
bodyBytes += chunkBytes(chunk, encoding);
return (write as (...args: unknown[]) => boolean)(
chunk,
encoding,
callback,
);
}) as Response['write'];
res.end = ((
chunk?: unknown,
encoding?: unknown,
callback?: unknown,
) => {
// `end()` also takes a callback in the first or second slot.
if (typeof chunk !== 'function') {
bodyBytes += chunkBytes(chunk, encoding);
}
return (end as (...args: unknown[]) => Response)(
chunk,
encoding,
callback,
);
}) as Response['end'];
// 'close' rather than 'finish': it fires for a response that completed
// AND for one whose connection died mid-stream, and an aborted download
// still sent whatever it sent.
res.once('close', () => {
try {
if (!isMeteredHost(req)) return;
const actor = resolveEgressActor(req);
if (!actor) return;
const usages = storageOpUsages(req);
const bytes = bodyBytes + estimateHeaderBytes(res);
if (bytes > 0) {
usages.push({
usageType: 'egress:bytes',
usageAmount: bytes,
costOverride: EGRESS_COSTS['egress:bytes'] * bytes,
});
}
if (usages.length === 0) return;
metering.bufferIncrementUsages!(actor, usages);
} catch (e) {
// Metering is never worth failing a request that already
// succeeded over.
console.warn(
`[metering] egress metering failed: ${(e as Error).message}`,
);
}
});
next();
};
};
@@ -19,6 +19,7 @@
import type { Request, Response } from 'express';
import { describe, expect, it } from 'vitest';
import { makeActor, type Actor } from '../../actor';
import { HttpError, isHttpError } from '../HttpError';
import {
DEFAULT_ADMIN_USERNAMES,
@@ -42,6 +43,23 @@ import {
type NextArg = undefined | 'route' | HttpError | unknown;
/**
* Rebuild an actor literal through `makeActor`, issuer-first, so the derived
* `effectiveApp` is present at every level of the token chain.
*/
const reviveActor = (actor: Actor): Actor =>
makeActor({
...actor,
...(actor.accessToken
? {
accessToken: {
...actor.accessToken,
issuer: reviveActor(actor.accessToken.issuer),
},
}
: {}),
});
const runGate = (
gate: (
req: Request,
@@ -50,6 +68,12 @@ const runGate = (
) => unknown,
req: Partial<Request>,
): NextArg => {
// Normalise the actor the way AuthService does before any gate sees one,
// so `effectiveApp` is derived here rather than spelled out on every
// literal below. Issuers too: a gate reading the chain reads the derived
// field, not `issuer.app`.
if (req.actor) req = { ...req, actor: reviveActor(req.actor) };
let captured: NextArg = undefined;
let called = false;
gate(req as Request, {} as Response, (arg?: unknown) => {
+2 -3
View File
@@ -18,7 +18,6 @@
*/
import type { Request, RequestHandler } from 'express';
import { effectiveActorApp } from '../../actor';
import type { Actor } from '../../actor';
import { HttpError } from '../HttpError';
import { assertVerifiedEmail } from '../verifiedEmail';
@@ -238,7 +237,7 @@ export const DEFAULT_ADMIN_USERNAMES = ['admin', 'system'] as const;
* pair, not a replacement for it.
*
* Also requires a _root token_ — an actor with no app anywhere in its token
* chain (see `effectiveActorApp`) — so a third-party app an admin has
* chain (see `Actor.effectiveApp`) — so a third-party app an admin has
* authorized can't reach admin endpoints on the admin's behalf. The one
* exception is `appGated`: on a route that is also appId-gated
* (`allowedAppIds`), a direct app-under-user actor is deferred to
@@ -278,7 +277,7 @@ export const adminOnlyGate = (
// through an app. A direct app-under-user actor is deferred to
// `allowedAppIdsGate` when the route is appId-gated; chain-only apps
// are rejected even then, since that gate can't see them.
const chainApp = req.actor ? effectiveActorApp(req.actor) : null;
const chainApp = req.actor?.effectiveApp ?? null;
if (chainApp && !(opts.appGated && req.actor?.app?.uid)) {
next(
new HttpError(403, 'Only admins may request this resource', {
+14 -40
View File
@@ -21,7 +21,6 @@ import type { RequestHandler } from 'express';
import { contentType as contentTypeFromMime } from 'mime-types';
import { posix as pathPosix } from 'node:path';
import type { puterClients } from '../../../clients';
import { FS_COSTS } from '../../../controllers/fs/costs';
import type { puterServices } from '../../../services';
import type { puterStores } from '../../../stores';
import type { IConfig, LayerInstances } from '../../../types';
@@ -592,45 +591,20 @@ export const createPuterSiteMiddleware = (
res.setHeader('Access-Control-Allow-Origin', '*');
res.status(statusOverride ?? (range ? 206 : 200));
// Best-effort egress metering against the site owner. The request
// itself is unauthenticated (public site visitor), so we can't use
// req.actor — charge the account that hosts the file. Same cost
// key as FS read egress (`filesystem:egress:bytes`). Fires once
// the body stream ends so we only meter bytes actually delivered
// (not aborted mid-stream).
const metering = layers.services.metering as unknown as
| {
batchIncrementUsages?: (
actor: unknown,
entries: unknown[],
) => void;
}
| undefined;
if (metering?.batchIncrementUsages && download.contentLength) {
const ownerActor = {
user: {
uuid: owner.uuid,
id: owner.id,
username: owner.username,
suspended: !!owner.suspended,
},
};
download.body.once('end', () => {
try {
const bytes = download.contentLength!;
metering.batchIncrementUsages!(ownerActor, [
{
usageType: 'filesystem:egress:bytes',
usageAmount: bytes,
costOverride:
FS_COSTS['filesystem:egress:bytes'] * bytes,
},
]);
} catch {
// ignore — non-critical.
}
});
}
// Name who this response's bytes are billed to; the egress middleware
// does the metering. A visitor carrying a token pays for what they
// fetch, and the account hosting the site covers everyone else —
// hosting is unauthenticated by design, so most visitors are nobody in
// particular. Set unconditionally because hosting subdomains are not
// metered by default: this is what opts the response in.
req.egressActor = req.actor ?? {
user: {
uuid: owner.uuid,
id: owner.id,
username: owner.username,
suspended: !!owner.suspended,
},
};
req.on('close', () => download.body.destroy());
download.body.on('error', (err) => res.destroy(err));
+148 -25
View File
@@ -57,32 +57,57 @@ export const RATE_LIMIT_BACKENDS = ['memory', 'redis', 'kv'];
const MEMORY_MAX_KEYS = 10_000;
const MEMORY_MAX_RETAIN_MS = 60 * 60_000;
/**
* Key → `{ ts, windowMs }`. The window is kept alongside the timestamps because
* the sweep has to know it: collecting on `MEMORY_MAX_RETAIN_MS` alone would
* drop the state of any limit whose window is longer than the retention floor,
* which silently shortens that limit to the floor. Day-scale windows are a real
* shape — a "few per day" grant, for one — and under `memory` they were being
* reset every hour. Retention is therefore whichever is longer, and the key cap
* below is what actually bounds memory.
*/
const memoryWindows = new Map();
/**
* Drop memory buckets that can no longer affect a decision. Runs on a timer
* below; exported as a test seam because the timer isn't drivable from a test
* (it's created at module load, before any fake clock is installed).
*/
export function sweepMemoryWindows() {
const now = Date.now();
for (const [k, entry] of memoryWindows) {
const retainMs = Math.max(MEMORY_MAX_RETAIN_MS, entry.windowMs);
if (
entry.ts.length === 0 ||
entry.ts[entry.ts.length - 1] < now - retainMs
)
memoryWindows.delete(k);
}
}
{
const sweep = setInterval(() => {
const cutoff = Date.now() - MEMORY_MAX_RETAIN_MS;
for (const [k, ts] of memoryWindows) {
if (ts.length === 0 || ts[ts.length - 1] < cutoff)
memoryWindows.delete(k);
}
}, 60_000);
const sweep = setInterval(sweepMemoryWindows, 60_000);
sweep.unref?.();
}
async function checkMemory(key, limit, windowMs) {
const now = Date.now();
const cutoff = now - windowMs;
let timestamps = memoryWindows.get(key);
if (!timestamps) {
let entry = memoryWindows.get(key);
if (!entry) {
// Map preserves insertion order; FIFO-evict before adding so a
// unique-key flood between sweep ticks can't blow up memory.
if (memoryWindows.size >= MEMORY_MAX_KEYS) {
const oldest = memoryWindows.keys().next().value;
memoryWindows.delete(oldest);
}
timestamps = [];
memoryWindows.set(key, timestamps);
entry = { ts: [], windowMs };
memoryWindows.set(key, entry);
} else {
// A scope's window can change across a deploy; the live value wins.
entry.windowMs = windowMs;
}
const timestamps = entry.ts;
while (timestamps.length > 0 && timestamps[0] < cutoff) timestamps.shift();
if (timestamps.length >= limit) return false;
timestamps.push(now);
@@ -185,37 +210,57 @@ async function acquireMemoryConcurrent(key, limit) {
if (c <= 1) memoryConcurrentCounts.delete(key);
else memoryConcurrentCounts.set(key, c - 1);
},
// Nothing expires a memory slot but the process holding it, so there
// is no staleness to renew away.
renew: async () => {},
};
}
async function acquireRedisConcurrent(redis, key, limit) {
const redisKey = `concurrent:${key}`;
// Atomic INCR + EXPIRE. If the new count exceeds the limit we DECR
// ourselves back out; the brief over-count is invisible to other
// callers because INCR is atomic per-key. EXPIRE is a safety net for
// process death between acquire and release — slots eventually clear
// on their own so a crashed worker can't pin the bucket.
const member = `${Date.now()}-${crypto.randomUUID()}`;
// One sorted-set member per held slot, scored by acquire time — the same
// shape `checkRedis` uses for windows, and for the same reason: expiry has
// to be per-slot, not per-key.
//
// A counter with a key-wide TTL cannot express that. Whoever touches the
// key last decides when *every* slot on it expires, so a rejected acquire
// extends the life of the slots that rejected it — and a client that
// retries on rejection (a websocket reconnect loop is the pointed case)
// holds a leaked bucket open forever, locking its owner out of a resource
// nobody is actually using. Here the sweep below drops each slot on its own
// age, so a leak drains on schedule no matter how hard anyone retries.
const now = Date.now();
const results = await redis
.multi()
.incr(redisKey)
// Slots older than the orphan window belonged to a process that died
// before releasing; drop them before counting.
.zremrangebyscore(redisKey, 0, now - ORPHAN_SAFETY_TTL_MS)
.zadd(redisKey, now, member)
.zcard(redisKey)
// Key-level TTL is only garbage collection for a bucket that goes
// quiet — the per-member sweep above is what bounds a live one.
.expire(redisKey, ORPHAN_SAFETY_TTL_SEC)
.exec();
const count = Number(
Array.isArray(results[0]) ? results[0][1] : results[0],
Array.isArray(results[2]) ? results[2][1] : results[2],
);
if (count > limit) {
await redis.decr(redisKey);
await redis.zrem(redisKey, member);
return { ok: false };
}
return {
ok: true,
release: async () => {
// DECR can race below zero if the TTL fired between acquire
// and release (slot cleared, counter resets, we DECR to -1).
// Clamp on the next observation; it costs an extra read only
// on the rare TTL race.
const after = await redis.decr(redisKey);
if (after < 0) await redis.set(redisKey, 0);
await redis.zrem(redisKey, member);
},
renew: async () => {
// Re-score in place so a slot held longer than the orphan window
// isn't mistaken for one whose owner died. `ZADD XX` only touches
// a member that's still there, so renewing after release (or after
// a sweep) can't resurrect the slot.
await redis.zadd(redisKey, 'XX', Date.now(), member);
await redis.expire(redisKey, ORPHAN_SAFETY_TTL_SEC);
},
};
}
@@ -246,6 +291,15 @@ async function acquireKvConcurrent(kv, key, limit) {
release: async () => {
await kv.del({ key: slotKey });
},
renew: async () => {
// Push the row's own TTL out; a slot that outlives the orphan
// window is held, not abandoned.
await kv.set({
key: slotKey,
value: 1,
expireAt: Math.ceil((Date.now() + ORPHAN_SAFETY_TTL_MS) / 1000),
});
},
};
}
@@ -559,6 +613,75 @@ export async function checkRateLimit(key, limit, windowMs, backend) {
}
}
/**
* Imperative concurrency acquire — the `acquire` twin to `checkRateLimit`, for
* long-lived things that aren't a request/response pair and so can't use
* `concurrencyGate`. The websocket handshake is the motivating case: the slot
* has to be held for the life of the connection, not the life of a response.
*
* Caller MUST invoke `release()` exactly once when the thing being counted ends
* (`ok: false` still returns a no-op `release`, so callers can release
* unconditionally). Fails open on backend error.
*
* `release()` returns a promise that settles once the slot is actually back —
* await it when the next observation has to see the freed slot. Fire-and-forget
* is fine for the usual case (an event handler on connection close), which is
* why it never rejects.
*
* A holder that can outlive `ORPHAN_SAFETY_TTL_MS` must call `renew()` on a
* timer, or the orphan sweep will reclaim its slot as abandoned and the cap
* stops counting it. Anything that finishes in seconds can ignore it.
*/
export async function acquireConcurrent(key, limit, backend) {
const bk = resolveBackend(backend);
try {
const result = await bk.acquire(key, limit);
if (!result.ok)
return {
ok: false,
release: async () => {},
renew: async () => {},
};
let released = false;
return {
ok: true,
release: async () => {
if (released) return;
released = true;
try {
await result.release();
} catch (err) {
console.error(
'[concurrent] imperative release failed:',
err,
);
}
},
renew: async () => {
if (released) return;
try {
await result.renew?.();
} catch (err) {
console.error('[concurrent] imperative renew failed:', err);
}
},
};
} catch (err) {
console.error(
'[concurrent] imperative acquire failed, failing open:',
err,
);
return { ok: true, release: async () => {}, renew: async () => {} };
}
}
/**
* How long a held slot stays valid without a `renew()`. Exported so a
* long-lived holder can pick a renewal cadence from it rather than hardcoding
* one that drifts out of step.
*/
export const CONCURRENT_SLOT_TTL_MS = ORPHAN_SAFETY_TTL_MS;
// -- Subscription-aware limit resolution -----------------------------
/**
@@ -32,6 +32,8 @@ import { EventEmitter } from 'node:events';
import { isHttpError } from '../HttpError.js';
import { setupTestServer } from '../../../testUtil.ts';
import {
CONCURRENT_SLOT_TTL_MS,
acquireConcurrent,
acquireDriverConcurrent,
checkDriverRateLimit,
checkRateLimit,
@@ -39,6 +41,7 @@ import {
configureRateLimit,
listConfiguredRateLimitBackends,
rateLimitGate,
sweepMemoryWindows,
} from './rateLimit.js';
// The rate-limit module is configured once at boot in production. In tests
@@ -848,7 +851,7 @@ describe('concurrencyGate — redis backend', () => {
await redis.flushall();
});
it('uses INCR/EXPIRE to coordinate slots and releases on finish', async () => {
it('coordinates slots across callers and releases on finish', async () => {
const opts = {
limit: 1,
key: 'ip',
@@ -879,6 +882,85 @@ describe('concurrencyGate — redis backend', () => {
});
});
// ── concurrency: orphaned slots ─────────────────────────────────────
//
// A slot whose holder died without releasing has to age out on its own. The
// pointed case is a caller that retries on rejection — a reconnecting socket —
// where a per-key expiry gets refreshed by the very attempts it is rejecting
// and the bucket never drains.
describe('acquireConcurrent — orphan recovery (redis)', () => {
let redis;
beforeAll(() => {
redis = new RedisMock();
configureRateLimit({ default: 'redis', redis });
});
afterAll(async () => {
vi.useRealTimers();
await redis?.quit?.();
configureRateLimit();
});
beforeEach(async () => {
vi.useFakeTimers();
vi.setSystemTime(new Date('2026-01-01T00:00:00Z'));
await redis.flushall();
});
it('frees a leaked slot after the orphan window even while callers retry', async () => {
const key = 'orphan-retry';
const start = Date.now();
const step = 5 * 60_000;
// Holder takes the only slot and dies — never releases.
expect((await acquireConcurrent(key, 1)).ok).toBe(true);
expect((await acquireConcurrent(key, 1)).ok).toBe(false);
// Retry throughout the orphan window, the way a reconnecting client
// does. Each rejection must leave the leaked slot's own age untouched.
for (
let elapsed = step;
elapsed < CONCURRENT_SLOT_TTL_MS;
elapsed += step
) {
vi.setSystemTime(new Date(start + elapsed));
expect((await acquireConcurrent(key, 1)).ok).toBe(false);
}
vi.setSystemTime(new Date(start + CONCURRENT_SLOT_TTL_MS + step));
expect((await acquireConcurrent(key, 1)).ok).toBe(true);
});
it('keeps a renewed slot held past the orphan window', async () => {
const key = 'orphan-renew';
const slot = await acquireConcurrent(key, 1);
expect(slot.ok).toBe(true);
// A live long-lived holder renews rather than letting the sweep
// mistake it for an abandoned one.
for (
let elapsed = 0;
elapsed < 2 * CONCURRENT_SLOT_TTL_MS;
elapsed += 10 * 60_000
) {
vi.setSystemTime(new Date(Date.now() + 10 * 60_000));
await slot.renew();
}
expect((await acquireConcurrent(key, 1)).ok).toBe(false);
await slot.release();
expect((await acquireConcurrent(key, 1)).ok).toBe(true);
});
it('does not resurrect a slot renewed after release', async () => {
const key = 'orphan-renew-after-release';
const slot = await acquireConcurrent(key, 1);
await slot.release();
await slot.renew();
expect((await acquireConcurrent(key, 1)).ok).toBe(true);
});
});
// ── concurrency: subscription-based limits ──────────────────────────
describe('concurrencyGate — bySubscription overrides', () => {
@@ -1184,6 +1266,31 @@ describe('driver helpers — backend failures', () => {
const req = { ip: '7.7.7.7', headers: {}, socket: {} };
// Minimal redis whose concurrent-acquire always reports one held slot, so
// a test can substitute its own failing `zrem` and exercise release alone.
const stubConcurrentRedis = () => ({
multi: () => ({
zremrangebyscore: function () {
return this;
},
zadd: function () {
return this;
},
zcard: function () {
return this;
},
expire: function () {
return this;
},
exec: async () => [
[null, 0],
[null, 1],
[null, 1],
[null, 1],
],
}),
});
it('checkDriverRateLimit fails open when the backend throws', async () => {
configureRateLimit({
default: 'redis',
@@ -1225,20 +1332,11 @@ describe('driver helpers — backend failures', () => {
configureRateLimit({
default: 'redis',
redis: {
multi: () => ({
incr: function () {
return this;
},
expire: function () {
return this;
},
exec: async () => [[null, 1]],
}),
decr: async () => {
if (failRelease) throw new Error('decr exploded');
return 0;
...stubConcurrentRedis(),
zrem: async () => {
if (failRelease) throw new Error('zrem exploded');
return 1;
},
set: async () => 'OK',
},
});
const handle = await acquireDriverConcurrent(req, 'iface', 'm', {
@@ -1261,19 +1359,10 @@ describe('driver helpers — backend failures', () => {
configureRateLimit({
default: 'redis',
redis: {
multi: () => ({
incr: function () {
return this;
},
expire: function () {
return this;
},
exec: async () => [[null, 1]],
}),
decr: async () => {
throw new Error('decr exploded');
...stubConcurrentRedis(),
zrem: async () => {
throw new Error('zrem exploded');
},
set: async () => 'OK',
},
});
const res = new EventEmitter();
@@ -1295,25 +1384,26 @@ describe('driver helpers — backend failures', () => {
spy.mockRestore();
});
it('clamps a redis concurrent counter that went negative after a TTL race', async () => {
it('absorbs a release whose slot was already swept away', async () => {
const redis = new RedisMock();
await redis.flushall();
configureRateLimit({ default: 'redis', redis });
const handle = await acquireDriverConcurrent(req, 'iface', 'clamp', {
limit: 2,
const handle = await acquireDriverConcurrent(req, 'iface', 'sweep', {
limit: 1,
});
expect(handle.ok).toBe(true);
// Simulate the orphan TTL firing between acquire and release.
const keys = await redis.keys('concurrent:*');
for (const k of keys) await redis.del(k);
// The orphan sweep collected the whole bucket before release ran.
for (const k of await redis.keys('concurrent:*')) await redis.del(k);
await expect(handle.release()).resolves.toBeUndefined();
await handle.release();
const remaining = await redis.keys('concurrent:*');
for (const k of remaining) {
expect(Number(await redis.get(k))).toBe(0);
}
// Releasing a slot that is already gone must not leave the bucket
// owing anything — the next caller gets a clean one.
const next = await acquireDriverConcurrent(req, 'iface', 'sweep', {
limit: 1,
});
expect(next.ok).toBe(true);
await redis.quit?.();
});
});
@@ -1342,3 +1432,112 @@ describe('memory backend key cap', () => {
expect(await checkRateLimit(victim, 1, 60_000, 'memory')).toBe(true);
});
});
describe('memory backend sweep retention', () => {
beforeAll(() => configureRateLimit());
afterAll(() => {
vi.useRealTimers();
configureRateLimit();
});
// The sweep's retention floor is an hour. A window longer than that has
// to survive it, or the limit is silently shortened to the floor — a
// day-scale "few per day" grant would reset hourly.
it('keeps a bucket whose window outlives the retention floor', async () => {
vi.useFakeTimers();
const day = 24 * 60 * 60_000;
const key = `long-window-${Math.random()}`;
vi.setSystemTime(new Date('2026-01-01T00:00:00Z'));
expect(await checkRateLimit(key, 1, day, 'memory')).toBe(true);
expect(await checkRateLimit(key, 1, day, 'memory')).toBe(false);
// Two hours on: past the retention floor, far short of the window.
vi.setSystemTime(new Date('2026-01-01T02:00:00Z'));
sweepMemoryWindows();
expect(await checkRateLimit(key, 1, day, 'memory')).toBe(false);
// Past the window itself, the bucket is collectable again.
vi.setSystemTime(new Date('2026-01-02T01:00:00Z'));
sweepMemoryWindows();
expect(await checkRateLimit(key, 1, day, 'memory')).toBe(true);
});
it('still collects a short-window bucket at the retention floor', async () => {
vi.useFakeTimers();
const key = `short-window-${Math.random()}`;
vi.setSystemTime(new Date('2026-01-01T00:00:00Z'));
expect(await checkRateLimit(key, 1, 60_000, 'memory')).toBe(true);
vi.setSystemTime(new Date('2026-01-01T02:00:00Z'));
sweepMemoryWindows();
// Nothing asserts the delete directly; a fresh bucket is the
// observable consequence of having been collected.
expect(await checkRateLimit(key, 1, 60_000, 'memory')).toBe(true);
});
});
describe('acquireConcurrent (imperative)', () => {
beforeAll(() => configureRateLimit());
afterAll(() => configureRateLimit());
// The websocket handshake is the motivating case: the slot is held for
// the life of the connection, not the life of a response, so it cannot
// go through `concurrencyGate`.
it('admits up to the limit and rejects past it', async () => {
const key = `imperative-${Math.random()}`;
const a = await acquireConcurrent(key, 2, 'memory');
const b = await acquireConcurrent(key, 2, 'memory');
const c = await acquireConcurrent(key, 2, 'memory');
expect(a.ok).toBe(true);
expect(b.ok).toBe(true);
expect(c.ok).toBe(false);
await a.release();
expect((await acquireConcurrent(key, 2, 'memory')).ok).toBe(true);
});
it('returns a no-op release on rejection so callers can release blindly', async () => {
const key = `imperative-noop-${Math.random()}`;
const held = await acquireConcurrent(key, 1, 'memory');
const denied = await acquireConcurrent(key, 1, 'memory');
expect(denied.ok).toBe(false);
// Releasing a slot we never got must not free the one we did.
await denied.release();
expect((await acquireConcurrent(key, 1, 'memory')).ok).toBe(false);
await held.release();
expect((await acquireConcurrent(key, 1, 'memory')).ok).toBe(true);
});
it('releases at most once even if called repeatedly', async () => {
const key = `imperative-once-${Math.random()}`;
const a = await acquireConcurrent(key, 1, 'memory');
await a.release();
await a.release();
await a.release();
// A double release would have driven the counter negative and
// handed out more slots than the limit allows.
expect((await acquireConcurrent(key, 1, 'memory')).ok).toBe(true);
expect((await acquireConcurrent(key, 1, 'memory')).ok).toBe(false);
});
it('fails open when the backend throws', async () => {
const err = new Error('backend down');
const redis = {
multi: () => ({
incr: () => {
throw err;
},
}),
};
configureRateLimit({ redis });
const result = await acquireConcurrent('any', 1, 'redis');
expect(result.ok).toBe(true);
configureRateLimit();
});
});
+20 -8
View File
@@ -26,12 +26,7 @@ import type { Actor } from '../actor';
* it instead of accepting any token that authenticates.
*/
export type TokenSource =
| 'body'
| 'header'
| 'x-api-key'
| 'cookie'
| 'query'
| 'handshake';
'body' | 'header' | 'x-api-key' | 'cookie' | 'query' | 'handshake';
/**
* Every route method PuterRouter exposes. Mirrors the express router surface
@@ -72,8 +67,8 @@ export type RoutePath = string | RegExp | Array<string | RegExp>;
* middleware chain in this order:
*
* subdomain → requireAuth (+ suspended) → emailConfirmed →
* requireUserActor → adminOnly → allowedAppIds →
* caller `middleware: []` → handler
* requireUserActor → adminOnly → allowedAppIds → rateLimit →
* requireCredits → concurrent → caller `middleware: []` → handler
*
* `requireUserActor`, `adminOnly`, and `allowedAppIds` all imply `requireAuth`;
* the materializer dedupes so only one auth gate ends up in the chain.
@@ -316,6 +311,23 @@ export interface RouteOptions {
backend?: 'memory' | 'redis' | 'kv';
};
/**
* Reject an account with nothing left of its usage budget with 402
* `insufficient_funds`.
*
* For routes that spend metered resources on the caller's behalf — moving
* file content, making object-store requests. Not for the routes that show
* an account what it has or let it delete things: an account that has run
* out still needs to be able to see its files, clear space, and reach its
* billing pages, and blocking that leaves no way out other than paying.
*
* Anonymous callers and worker sessions pass; see `requireCreditsGate`.
* Rate limits remain the bound on request _count_ — this bounds spend, and
* lags the traffic that produced it by the metering buffer window, so it
* stops sustained usage rather than a burst.
*/
requireCredits?: boolean;
// Reserved — wire as the corresponding features/services land:
// bodyFiles?: string[]; // multer-style multipart fields
// responseTimeout?: number;
+2
View File
@@ -28,5 +28,7 @@ export {
isAppActor,
isAccessTokenActor,
actorUid,
assertResolvedActor,
makeActor,
userRelatedActor,
} from './actor';
+62
View File
@@ -0,0 +1,62 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request } from 'express';
import { describe, expect, it } from 'vitest';
import { runWithContext } from './context';
import { recordStorageOps } from './storageOps';
const withRequest = (fn: () => void): Request => {
const req = {} as Request;
runWithContext({ req }, fn);
return req;
};
describe('recordStorageOps', () => {
it('tallies each class on the request in scope', () => {
const req = withRequest(() => {
recordStorageOps('write');
recordStorageOps('write', 3);
recordStorageOps('read');
recordStorageOps('delete', 2);
});
expect(req.storageOps).toEqual({ write: 4, read: 1, delete: 2 });
});
it('ignores counts that are not a positive number', () => {
const req = withRequest(() => {
recordStorageOps('write', 0);
recordStorageOps('write', -1);
recordStorageOps('write', Number.NaN);
});
expect(req.storageOps).toBeUndefined();
});
it('does nothing outside a request', () => {
expect(() => recordStorageOps('write')).not.toThrow();
});
it('does nothing when the context carries no request', () => {
expect(() =>
runWithContext({}, () => recordStorageOps('write')),
).not.toThrow();
});
});
+51
View File
@@ -0,0 +1,51 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { Context } from './context';
import './http/expressAugmentation';
/**
* Object-store request classes. They are priced apart because they cost
* different amounts per request: uploads, copies and multipart parts are the
* expensive class, fetches and metadata lookups an order of magnitude cheaper,
* and removals are not charged for at all.
*/
export type StorageOpClass = 'write' | 'read' | 'delete';
export type StorageOpCounts = Partial<Record<StorageOpClass, number>>;
/**
* Tally object-store requests against the request that caused them.
*
* The counts ride on the express request rather than going straight to
* metering, so that the whole cost of serving a request — its response bytes
* and the object-store calls behind them — settles as one write when the
* response ends. Work with no request in scope (boot, background sweeps)
* tallies nothing, which is why this never throws when called outside one.
*/
export const recordStorageOps = (
opClass: StorageOpClass,
count: number = 1,
): void => {
if (!Number.isFinite(count) || count <= 0) return;
const req = Context.get('req');
if (!req) return;
const ops = (req.storageOps ??= {});
ops[opClass] = (ops[opClass] ?? 0) + count;
};
+13 -46
View File
@@ -88,55 +88,22 @@ const implicit_user_app_permissions = [
},
];
// Pre-v2 each entry below carried a `policy: { 'rate-limit': { max, period } }`
// block attached by a `policyPerm('temp.kv')` / `policyPerm('user.es')` helper,
// and a `driverPolicies` table held the actual `{max, period}` values. Nothing
// ever read those policy blocks in v2 — rate limiting is now declared
// directly on each driver via `@Driver({ rateLimit })` / `readonly rateLimit`
// (see KVStoreDriver, AppDriver, SubdomainDriver, NotificationDriver), with
// per-subscription overrides via `bySubscription`. The grants themselves
// stay so the permission scan still emits a path for each group/permission;
// only the dead policy plumbing was removed.
const hardcoded_user_group_permissions = {
system: {
'ca342a5e-b13d-4dee-9048-58b11a57cc55': {
driver: {},
service: {},
feature: {},
'local-terminal:access': {},
},
'b7220104-7905-4985-b996-649fdcdb3c8f': {
driver: {},
service: {},
'service:hello-world:ii:hello-world': {},
'service:puter-kvstore:ii:puter-kvstore': {},
'driver:puter-kvstore': {},
'service:puter-notifications:ii:crud-q': {},
'service:puter-apps:ii:crud-q': {},
'service:puter-subdomains:ii:crud-q': {},
'service:apps:ii:crud-q': {},
'service:es\\Cnotification:ii:crud-q': {},
'service:es\\Capp:ii:crud-q': {},
'service:app:ii:crud-q': {},
'service:es\\Csubdomain:ii:crud-q': {},
},
'78b1b1dd-c959-44d2-b02c-8735671f9997': {
driver: {},
service: {},
'service:hello-world:ii:hello-world': {},
'service:puter-kvstore:ii:puter-kvstore': {},
'driver:puter-kvstore': {},
'service:es\\Cnotification:ii:crud-q': {},
'service:es\\Capp:ii:crud-q': {},
'service:app:ii:crud-q': {},
'service:es\\Csubdomain:ii:crud-q': {},
'service:apps:ii:crud-q': {},
},
},
// Permissions every user actor holds, regardless of group membership.
//
// Roots only: a grant subsumes everything beneath it, so `service` already
// answers `service:puter-kvstore:ii:puter-kvstore`. Listing descendants adds
// entries no check can ever need.
//
// A floor, not a ceiling — anything that depends on *who* the user is belongs
// in the ACL as a `user_to_group_permissions` / `user_to_user_permissions`
// row. Adding a root here grants it to every user, temp accounts included.
const default_user_permissions = {
driver: {},
service: {},
};
module.exports = {
implicit_user_app_permissions,
default_implicit_user_app_permissions,
hardcoded_user_group_permissions,
default_user_permissions,
};
@@ -48,6 +48,7 @@ import { ChatCompletionDriver } from './ChatCompletionDriver.js';
import { AzureChatProvider } from './providers/azure/AzureChatProvider.js';
import { FakeChatProvider } from './providers/FakeChatProvider.js';
import { InfronProvider } from './providers/infron/InfronProvider.js';
import { NeuralwattProvider } from './providers/neuralwatt/NeuralwattProvider.js';
import { OpenAiChatProvider } from './providers/openai/OpenAiChatCompletionsProvider.js';
import { OpenRouterProvider } from './providers/openrouter/OpenRouterProvider.js';
import { TogetherAIProvider } from './providers/together/TogetherAIProvider.js';
@@ -72,6 +73,7 @@ const FULL_PROVIDER_CONFIG = {
'together-ai': { apiKey: 'k' },
openrouter: { apiKey: 'k', apiBaseUrl: 'https://openrouter.test' },
infron: { apiKey: 'k' },
neuralwatt: { apiKey: 'k' },
// Suppress auto-discovery of a developer's local Ollama.
ollama: { enabled: false },
},
@@ -120,6 +122,9 @@ beforeAll(async () => {
vi.spyOn(InfronProvider.prototype, 'models').mockResolvedValue(
aggregatorCatalog('infron-only-model') as never,
);
vi.spyOn(NeuralwattProvider.prototype, 'models').mockResolvedValue(
aggregatorCatalog('neuralwatt-only-model') as never,
);
fullDriver = await makeDriver(FULL_PROVIDER_CONFIG);
fakeOnlyDriver = await makeDriver({
providers: { ollama: { enabled: false } },
@@ -176,6 +181,7 @@ describe('ChatCompletionDriver provider registration', () => {
'together-ai',
'openrouter',
'infron',
'neuralwatt',
'fake-chat',
]) {
expect(providers).toContain(expected);
@@ -0,0 +1,329 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify it under the
* terms of the GNU Affero General Public License as published by the Free
* Software Foundation, either version 3 of the License, or (at your option) any
* later version.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
* details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
*/
/**
* Which provider actually serves a model that several providers advertise.
*
* Lives apart from ChatCompletionDriver.test.ts because `vi.mock` is
* file-scoped and hoisted — mocking the OpenAI SDK and axios here would
* otherwise leak into every test in that file. Both mocks sit at the real
* network egress points, so the driver's registration, model-map build and
* resolution all run for real.
*/
import {
afterAll,
beforeAll,
beforeEach,
describe,
expect,
it,
vi,
type MockInstance,
} from 'vitest';
import { HttpError } from '../../core/http/HttpError.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { kv } from '../../util/kvSingleton.js';
import { withTestActor } from '../integrationTestUtil.js';
import { ChatCompletionDriver } from './ChatCompletionDriver.js';
import {
clearUnhealthyRoutes,
markRouteUnhealthy,
} from './utils/providerHealth.js';
// -- OpenAI SDK mock ------------------------------------------------
// Gemini reaches Google through `new openai.OpenAI()` (default export) and
// the gateway through the named one, so both must resolve to the same ctor.
const { createMock } = vi.hoisted(() => ({ createMock: vi.fn() }));
vi.mock('openai', () => {
const OpenAICtor = vi.fn().mockImplementation(function (
this: Record<string, unknown>,
) {
this.chat = { completions: { create: createMock } };
});
return { OpenAI: OpenAICtor, default: { OpenAI: OpenAICtor } };
});
// -- axios mock (gateway model catalog) -----------------------------
const { axiosRequestMock } = vi.hoisted(() => ({ axiosRequestMock: vi.fn() }));
vi.mock('axios', () => ({
default: { request: axiosRequestMock },
request: axiosRequestMock,
}));
// -- Harness --------------------------------------------------------
let server: PuterServer;
let driver: ChatCompletionDriver;
const INFRON_KV_KEY = 'infronChat:models';
const OPENROUTER_KV_KEY = 'openrouterChat:models';
// Google lists gemini-2.5-flash input at $0.30/MTok. The gateway quotes a
// floor price across its upstream routes, so it undercuts — which is exactly
// the condition that used to hand it the traffic.
const GATEWAY_CATALOG = [
{
id: 'google/gemini-2.5-flash',
display_name: 'Google: Gemini 2.5 Flash',
category_type: 'LLM',
supported_endpoint_types: ['openai'],
context_length: 1_048_576,
max_output_tokens: 65_536,
min_prompt_price: 0.15,
min_completion_price: 1.0,
},
{
// Only the gateway carries this one — no first-party counterpart.
id: 'google/gemini-2.5-flash-image-preview',
display_name: 'Google: Gemini 2.5 Flash Image Preview',
category_type: 'LLM',
supported_endpoint_types: ['openai'],
context_length: 32_768,
max_output_tokens: 8_192,
min_prompt_price: 0.3,
min_completion_price: 2.5,
},
{
// A vendor we integrate with directly that isn't Google — the case
// resold duplicates used to be dropped for.
id: 'deepseek/deepseek-v4-pro',
display_name: 'DeepSeek: V4 Pro',
category_type: 'LLM',
supported_endpoint_types: ['openai'],
context_length: 1_000_000,
max_output_tokens: 65_536,
min_prompt_price: 0.1,
min_completion_price: 0.5,
},
];
// OpenRouter's catalog is shaped differently, and it carries the same model
// under two upstream orgs — four routes total for deepseek-v4-pro once the
// vendor and Infron are counted.
const OPENROUTER_CATALOG = [
'deepseek/deepseek-v4-pro',
'deepseek-ai/deepseek-v4-pro',
'google/gemini-2.5-flash',
].map((id) => ({
id,
name: `${id} (via OpenRouter)`,
pricing: { prompt: '0.0000001', completion: '0.0000005' },
context_length: 1_000_000,
top_provider: { max_completion_tokens: 65_536 },
created: 1_700_000_000,
}));
beforeAll(async () => {
server = await setupTestServer();
kv.del?.(INFRON_KV_KEY);
kv.del?.(OPENROUTER_KV_KEY);
axiosRequestMock.mockImplementation(({ url }: { url: string }) => ({
data: {
data: url.includes('openrouter')
? OPENROUTER_CATALOG
: GATEWAY_CATALOG,
},
}));
// Built once, not per-test: `#buildModelMap` mutates the catalogs
// providers hand back (lowercasing ids, pushing `puterId` onto the
// shared `aliases` array), and GeminiChatProvider returns its
// module-level GEMINI_MODELS by reference.
driver = new ChatCompletionDriver(
{
providers: {
gemini: { apiKey: 'test-key' },
deepseek: { apiKey: 'test-key' },
infron: { apiKey: 'test-key' },
openrouter: { apiKey: 'test-key' },
ollama: { enabled: false },
},
} as never,
server.clients,
server.stores,
server.services,
);
driver.onServerStart();
// `onServerStart` doesn't await `#buildModelMap`, and the gateway
// catalogs resolve on a microtask — poll until both gateways land.
for (let i = 0; i < 200; i++) {
const ids = await driver.list();
if (
ids.some((id) => id.startsWith('infron:')) &&
ids.some((id) => id.startsWith('openrouter:'))
) {
break;
}
await new Promise((r) => setTimeout(r, 5));
}
});
// Every failure in this file marks the route it hit. Without this the first
// test would decide where the second one starts.
beforeEach(() => clearUnhealthyRoutes());
afterAll(async () => {
await server?.shutdown();
clearUnhealthyRoutes();
});
/**
* Route a request and report who was tried, in order. Forcing the upstream to
* reject is what makes the whole chain observable: the driver records every
* attempt on the thrown error, and `attempts[0]` is who it chose first.
*/
const attemptsFor = async (model: string) => {
createMock.mockRejectedValue(new Error('upstream down'));
let caught: HttpError | undefined;
try {
await withTestActor(() =>
driver.complete({
model,
messages: [{ role: 'user', content: 'hi' }],
}),
);
} catch (e) {
caught = e as HttpError;
}
expect(caught).toBeInstanceOf(HttpError);
return (
caught as unknown as {
fields: { attempts: { model: string; provider: string }[] };
}
).fields.attempts;
};
describe('ChatCompletionDriver gemini routing', () => {
it('serves gemini models from Google, with the gateway only as fallback', async () => {
const attempts = await attemptsFor('gemini-2.5-flash');
expect(attempts[0]).toMatchObject({
provider: 'gemini',
model: 'gemini-2.5-flash',
});
expect(attempts[1]).toMatchObject({
provider: 'infron',
model: 'infron:google/gemini-2.5-flash',
});
});
it('routes the prefixed and puterId forms to Google too', async () => {
for (const alias of [
'google/gemini-2.5-flash',
'google:google/gemini-2.5-flash',
]) {
const attempts = await attemptsFor(alias);
expect(attempts[0].provider).toBe('gemini');
}
});
it('still routes models only the gateway carries to the gateway', async () => {
const attempts = await attemptsFor(
'google/gemini-2.5-flash-image-preview',
);
expect(attempts[0]).toMatchObject({ provider: 'infron' });
});
});
describe('ChatCompletionDriver duplicate-model fallback', () => {
// deepseek-v4-pro is served directly by DeepSeek, by Infron, and twice by
// OpenRouter (two upstream orgs) — four routes in one bucket.
const SHARED = 'deepseek-v4-pro';
it('keeps a reseller duplicate of any vendor, not just Google', async () => {
const attempts = await attemptsFor(SHARED);
expect(attempts[0]).toMatchObject({ provider: 'deepseek' });
expect(attempts.map((a) => a.provider)).toContain('infron');
});
it('leaves openrouter below the other resellers in the chain', async () => {
const attempts = await attemptsFor(SHARED);
const providers = attempts.map((a) => a.provider);
expect(providers.indexOf('infron')).toBeLessThan(
providers.indexOf('openrouter'),
);
});
it('stops after three attempts even with a fourth route available', async () => {
const attempts = await attemptsFor(SHARED);
expect(attempts).toHaveLength(3);
// Proof the cap is what stopped the chain rather than the bucket
// running dry: take the three just tried out of contention and a
// fourth route is still there to be served.
const burned = attempts.map((a) => `${a.provider}:${a.model}`);
for (const a of attempts) markRouteUnhealthy(a.provider, a.model);
const next = await attemptsFor(SHARED);
expect(burned).not.toContain(`${next[0].provider}:${next[0].model}`);
});
it('never tries the same provider-and-model pair twice', async () => {
const attempts = await attemptsFor(SHARED);
const routes = attempts.map((a) => `${a.provider}:${a.model}`);
expect(new Set(routes).size).toBe(routes.length);
});
});
describe('ChatCompletionDriver unhealthy-route skipping', () => {
it('skips a route marked by an earlier failure and serves the next one', async () => {
markRouteUnhealthy('deepseek', 'deepseek-v4-pro');
const attempts = await attemptsFor('deepseek-v4-pro');
expect(attempts[0].provider).toBe('infron');
expect(attempts.map((a) => a.provider)).not.toContain('deepseek');
});
it('marks the routes a failing request burned through', async () => {
await attemptsFor('deepseek-v4-pro');
// The marks the first request left behind push the second one past
// everything that just failed.
const next = await attemptsFor('deepseek-v4-pro');
expect(next[0].provider).not.toBe('deepseek');
});
it('still serves a marked route when it is the only one left', async () => {
// gemini-2.5-flash-image-preview has a single route; marking it must
// degrade to trying it anyway rather than failing with no attempt.
markRouteUnhealthy(
'infron',
'infron:google/gemini-2.5-flash-image-preview',
);
const attempts = await attemptsFor(
'google/gemini-2.5-flash-image-preview',
);
expect(attempts[0]).toMatchObject({ provider: 'infron' });
});
});
@@ -587,6 +587,64 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
expect(completeSpy).not.toHaveBeenCalled();
});
// A provider that can't report a model's output ceiling used to make the
// cap arithmetic go negative — `null - approxTokens` is negative, not NaN
// — so a funded account was told it had insufficient funds.
for (const [label, ceiling] of [
['null', null],
['zero', 0],
['undefined', undefined],
] as const) {
it(`serves models whose output ceiling is ${label}`, async () => {
vi.spyOn(
FakeChatProvider.prototype,
'models',
).mockResolvedValueOnce([
{
id: 'nocap',
aliases: [],
costs_currency: 'usd-cents',
costs: { input_tokens: 1000, output_tokens: 2000 },
max_tokens: ceiling,
},
] as never);
const d = await makeDriver();
vi.spyOn(
server.services.metering,
'hasEnoughCredits',
).mockResolvedValue(true);
vi.spyOn(
server.services.metering,
'getRemainingUsage',
).mockResolvedValue(100_000);
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockResolvedValueOnce({
message: {
role: 'assistant',
content: [{ type: 'text', text: 'ok' }],
},
usage: { input_tokens: 1, output_tokens: 1 },
finish_reason: 'stop',
} as never);
await withTestActor(() =>
d.complete({
model: 'nocap',
messages: [{ role: 'user', content: 'hi' }],
}),
);
// Credits still bound the request; only the unknown model
// ceiling is ignored.
const passed = completeSpy.mock.calls[0]![0] as ICompleteArguments;
expect(passed.max_tokens!).toBeGreaterThan(0);
expect(passed.max_tokens!).toBeLessThanOrEqual(50);
});
}
it('rejects subscriber-only models for the default free subscription', async () => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValueOnce([
{
@@ -41,6 +41,7 @@ import { InfronProvider } from './providers/infron/InfronProvider.js';
import { MiniMaxProvider } from './providers/minimax/MiniMaxProvider.js';
import { MistralAIProvider } from './providers/mistral/MistralAiProvider.js';
import { MoonshotProvider } from './providers/moonshot/MoonshotProvider.js';
import { NeuralwattProvider } from './providers/neuralwatt/NeuralwattProvider.js';
import { OllamaChatProvider } from './providers/ollama/OllamaProvider.js';
import { OpenAiChatProvider } from './providers/openai/OpenAiChatCompletionsProvider.js';
import { OpenAiResponsesChatProvider } from './providers/openai/OpenAiChatResponsesProvider.js';
@@ -60,9 +61,18 @@ import {
normalize_messages,
normalize_single_message,
} from './utils/Messages.js';
import {
compareModelPreference,
isIdentityKey,
normalizeModelKey,
} from './utils/modelRouting.js';
import {
isRouteUnhealthy,
markRouteUnhealthy,
} from './utils/providerHealth.js';
import { AIChatStream } from './utils/Streaming.js';
const MAX_FALLBACKS = 4; // includes first attempt
const MAX_ATTEMPTS = 3; // the first attempt plus two fallbacks
type ProviderAttempt = {
model: string;
@@ -123,6 +133,25 @@ const isUpstream5xx = (a: ProviderAttempt) =>
a.error,
);
/**
* Whether a failure indicts the route rather than the request.
*
* Outages, rate limits and bad credentials will hit the next caller too, so the
* route is worth marking. A 4xx the upstream returned on the request's own
* merits (malformed tools, oversized prompt) says nothing about the route and
* must not take it out of rotation for everyone else. Attempts with no status
* at all are transport failures — treat them as route problems.
*/
const isRouteLevelFailure = (a: ProviderAttempt) =>
a.status === undefined ||
isRateLimit(a) ||
isAuthFailure(a) ||
isUpstream5xx(a);
// One bucket can hold the same model id under several providers *and* several
// ids under one provider, so only the pair identifies an attempt.
const routeId = (provider: string, modelId: string) => `${provider}:${modelId}`;
/**
* Map an exhausted fallback chain to a single user-facing HttpError.
*
@@ -402,11 +431,19 @@ export class ChatCompletionDriver extends PuterDriver {
remainingCredits - approximateInputCost;
const maxAllowedOutputTokens =
maxAllowedOutputUcents / outputTokenCost;
// A provider may not know a model's output ceiling. Drop the
// term rather than let a missing value drive the cap: `null`
// coerces to 0, so the subtraction goes negative instead of
// NaN and the user is told they're out of credits.
const modelOutputCeiling =
Number.isFinite(model.max_tokens) && model.max_tokens > 0
? model.max_tokens - approximateTokenCount
: Number.POSITIVE_INFINITY;
const cap = Math.floor(
Math.min(
args.max_tokens ?? Number.POSITIVE_INFINITY,
maxAllowedOutputTokens,
model.max_tokens - approximateTokenCount,
modelOutputCeiling,
),
);
// `cap` is the credit-bounded ceiling on output tokens. When
@@ -437,6 +474,20 @@ export class ChatCompletionDriver extends PuterDriver {
const attempts: ProviderAttempt[] = [];
let res: IChatCompleteResult | undefined;
// A failed route is remembered briefly so the next request skips it
// rather than paying its timeout again.
const recordFailure = (
modelId: string,
providerId: string,
err: unknown,
) => {
const attempt = toAttempt(modelId, providerId, err);
attempts.push(attempt);
if (isRouteLevelFailure(attempt)) {
markRouteUnhealthy(providerId, modelId);
}
};
try {
res = await provider.complete({
...args,
@@ -444,19 +495,17 @@ export class ChatCompletionDriver extends PuterDriver {
provider: model.provider,
});
} catch (e) {
attempts.push(toAttempt(model.id, model.provider!, e));
recordFailure(model.id, model.provider!, e);
// Fallback loop
const tried = [model.id];
const triedProviders = [model.provider!];
// Fallback loop — the bucket holds every provider that serves this
// model, ranked by `compareModelPreference`, so each miss walks one
// step down that order.
const bucketKey = model.id;
const tried = new Set([routeId(model.provider!, model.id)]);
let lastError: Error | null = e as Error;
while (lastError && tried.length < MAX_FALLBACKS) {
const fallback = this.#findFallback(
model.id,
tried,
triedProviders,
);
while (lastError && attempts.length < MAX_ATTEMPTS) {
const fallback = this.#findFallback(bucketKey, tried);
if (!fallback) break;
const fbProvider = this.#providers[fallback.provider!];
@@ -473,8 +522,7 @@ export class ChatCompletionDriver extends PuterDriver {
});
}
tried.push(fallback.id);
triedProviders.push(fallback.provider!);
tried.add(routeId(fallback.provider!, fallback.id));
try {
res = await fbProvider.complete({
@@ -486,9 +534,7 @@ export class ChatCompletionDriver extends PuterDriver {
lastError = null;
} catch (fbErr) {
lastError = fbErr as Error;
attempts.push(
toAttempt(fallback.id, fallback.provider!, fbErr),
);
recordFailure(fallback.id, fallback.provider!, fbErr);
}
}
}
@@ -999,127 +1045,97 @@ export class ChatCompletionDriver extends PuterDriver {
);
}
const neuralwatt = providers['neuralwatt'];
const neuralwattKey = readKey(neuralwatt);
if (neuralwattKey) {
this.#providers['neuralwatt'] = new NeuralwattProvider(
{
apiKey: neuralwattKey,
apiBaseUrl: neuralwatt?.apiBaseUrl as string | undefined,
},
metering,
);
}
// Fake provider — always available for testing
this.#providers['fake-chat'] = new FakeChatProvider();
}
// -- Model map ---------------------------------------------------
/**
* Group every provider's catalog into per-model buckets.
*
* A bucket is the set of routes to one model: the vendor we integrate with
* directly plus every reseller carrying it. They are deliberately _not_
* deduplicated — the duplicates are what the fallback loop walks when a
* route fails. `compareModelPreference` decides who serves first, so a
* reseller only takes traffic once the vendor has actually failed.
*
* Entries join a bucket by identity key (see `isIdentityKey`); display
* names remain addressable but never merge two providers' entries.
*/
async #buildModelMap() {
const AGGREGATORS = new Set(['together-ai', 'openrouter', 'infron']);
for (const providerName in this.#providers) {
const provider = this.#providers[providerName];
const isAggregator = AGGREGATORS.has(providerName);
for (const model of await provider.models()) {
model.id = model.id.trim().toLowerCase();
if (!this.#modelIdMap[model.id]) {
this.#modelIdMap[model.id] = [];
}
this.#modelIdMap[model.id].push({
...model,
provider: providerName,
});
model.id = normalizeModelKey(model.id);
if (model.puterId) {
if (model.aliases) {
model.aliases.push(model.puterId);
} else {
model.aliases = [model.puterId];
}
model.aliases = model.aliases
? [...model.aliases, model.puterId]
: [model.puterId];
}
if (isAggregator && model.aliases) {
let skip = false;
for (const rawAlias of model.aliases) {
const alias = rawAlias.trim().toLowerCase();
const existing = this.#modelIdMap[alias];
if (
existing &&
existing !== this.#modelIdMap[model.id]
) {
if (existing.some((m) => m.provider === 'gemini')) {
// Gemini exception — let the aggregator
// entry through.
continue;
}
skip = true;
break;
}
}
if (skip) {
// Remove the entry we just pushed; leave the bucket
// intact for other providers.
const bucket = this.#modelIdMap[model.id];
bucket.pop();
if (bucket.length === 0) {
delete this.#modelIdMap[model.id];
}
continue;
}
// Catalogs derive an alias by stripping the vendor org off the
// id, which yields '' for ids that carry no org. Drop those —
// an empty key would pool unrelated models together.
const keys = [model.id, ...(model.aliases ?? [])]
.map(normalizeModelKey)
.filter((key) => key.length > 0);
const bucket =
keys
.filter(isIdentityKey)
.map((key) => this.#modelIdMap[key])
.find(Boolean) ?? [];
bucket.push({ ...model, provider: providerName });
// First registration owns a key: a name already claimed by
// another model keeps pointing where it did.
for (const key of keys) {
this.#modelIdMap[key] ??= bucket;
}
if (model.aliases) {
for (let alias of model.aliases) {
alias = alias.trim().toLowerCase();
if (!this.#modelIdMap[alias]) {
this.#modelIdMap[alias] =
this.#modelIdMap[model.id];
} else if (
this.#modelIdMap[alias] !==
this.#modelIdMap[model.id]
) {
this.#modelIdMap[alias].push({
...model,
provider: providerName,
});
this.#modelIdMap[model.id] =
this.#modelIdMap[alias];
}
}
}
// Sort: together-ai always last; then cheapest input-cost
// first; ties break by shorter id (usually the official
// name over a long aggregator-qualified one).
this.#modelIdMap[model.id].sort((a, b) => {
const aAgg = a.provider === 'together-ai';
const bAgg = b.provider === 'together-ai';
if (aAgg !== bAgg) return aAgg ? 1 : -1;
const aCost = a.costs[
(a.input_cost_key as string) || 'input_tokens'
] as number;
const bCost = b.costs[
(b.input_cost_key as string) || 'input_tokens'
] as number;
if (aCost === bCost) return a.id.length - b.id.length;
return aCost - bCost;
});
bucket.sort(compareModelPreference);
}
}
}
#resolveModel(modelId: string, provider?: string): IChatModel | null {
const models = this.#modelIdMap[modelId?.trim().toLowerCase()];
const models = this.#modelIdMap[normalizeModelKey(modelId ?? '')];
if (!models || models.length === 0) return null;
if (!provider) return models[0];
return models.find((m) => m.provider === provider) ?? models[0];
// An explicitly requested provider is honoured even if its route is
// marked — the caller asked for that one, not for the cheapest hop.
if (provider) {
const pinned = models.find((m) => m.provider === provider);
if (pinned) return pinned;
}
return this.#preferHealthy(models) ?? models[0];
}
#findFallback(
modelId: string,
tried: string[],
triedProviders: string[],
): IChatModel | null {
#findFallback(modelId: string, tried: Set<string>): IChatModel | null {
const models = this.#modelIdMap[modelId];
if (!models) return null;
return (
models.find(
(m) =>
!tried.includes(m.id) ||
!triedProviders.includes(m.provider!),
) ?? null
const untried = models.filter(
(m) => !tried.has(routeId(m.provider!, m.id)),
);
// Degrade to a marked route rather than to no route at all: the marks
// are a hint about recent failures, not a quota.
return this.#preferHealthy(untried) ?? untried[0] ?? null;
}
#preferHealthy(models: IChatModel[]): IChatModel | undefined {
return models.find((m) => !isRouteUnhealthy(m.provider!, m.id));
}
}
@@ -314,28 +314,4 @@ export const CLAUDE_MODELS: IChatModel[] = [
context: 200000,
max_tokens: 64000,
},
{
puterId: 'anthropic:anthropic/claude-opus-4-1',
id: 'claude-opus-4-1-20250805',
modalities: { input: ['text', 'image', 'pdf'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-03-31',
release_date: '2025-08-05',
aliases: ['claude-opus-4-1', 'anthropic/claude-opus-4-1'],
name: 'Claude Opus 4.1',
costs_currency: 'usd-cents',
input_cost_key: 'input_tokens',
output_cost_key: 'output_tokens',
costs: {
tokens: 1_000_000,
input_tokens: 1500,
ephemeral_5m_input_tokens: 1500 * 1.25,
ephemeral_1h_input_tokens: 1500 * 2,
cache_read_input_tokens: 1500 * 0.1,
output_tokens: 7500,
},
context: 200000,
max_tokens: 32000,
},
];
@@ -0,0 +1,699 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Offline unit tests for NeuralwattProvider.
*
* Boots a real PuterServer and constructs NeuralwattProvider against the
* live MeteringService. The OpenAI SDK and axios (catalog + quota) are
* mocked at their module boundaries.
*/
import { Writable } from 'node:stream';
import {
afterAll,
afterEach,
beforeAll,
beforeEach,
describe,
expect,
it,
vi,
type MockInstance,
} from 'vitest';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import { PuterServer } from '../../../../server.js';
import { setupTestServer } from '../../../../testUtil.js';
import { kv } from '../../../../util/kvSingleton.js';
import { withTestActor } from '../../../integrationTestUtil.js';
import { AIChatStream } from '../../utils/Streaming.js';
import {
mapNeuralwattApiModel,
NEURALWATT_DEFAULT_MODEL,
stripNeuralwattPrefix,
} from './models.js';
import { NeuralwattProvider } from './NeuralwattProvider.js';
// ── OpenAI SDK mock ─────────────────────────────────────────────────
const { createMock, openAICtor } = vi.hoisted(() => ({
createMock: vi.fn(),
openAICtor: vi.fn(),
}));
vi.mock('openai', () => {
const OpenAICtor = vi.fn().mockImplementation(function (
this: Record<string, unknown>,
opts: unknown,
) {
openAICtor(opts);
this.chat = { completions: { create: createMock } };
});
return { OpenAI: OpenAICtor, default: { OpenAI: OpenAICtor } };
});
// ── axios mock (models + quota) ─────────────────────────────────────
const { axiosRequestMock } = vi.hoisted(() => ({
axiosRequestMock: vi.fn(),
}));
vi.mock('axios', () => ({
default: { request: axiosRequestMock },
request: axiosRequestMock,
}));
// ── Test harness ────────────────────────────────────────────────────
let server: PuterServer;
let recordSpy: MockInstance<MeteringService['utilRecordUsageObject']>;
const KV_MODELS_KEY = 'neuralwattChat:models';
const KV_QUOTA_KEY = 'neuralwattChat:quota';
const SAMPLE_API_MODELS = [
{
id: 'deepseek-v4-flash',
created: 1_700_000_000,
max_model_len: 1_000_000,
metadata: {
display_name: 'DeepSeek V4 Flash',
description: 'Fast tool-calling model',
pricing: {
input_per_million: 0.14,
output_per_million: 0.28,
cached_input_per_million: 0.014,
pricing_tbd: false,
},
capabilities: {
tools: true,
vision: false,
streaming: true,
},
limits: {
max_context_length: 1_000_000,
max_output_tokens: 384_000,
},
},
},
{
id: 'zai-org/GLM-5.1-FP8',
metadata: {
display_name: 'GLM 5.1',
pricing: {
input_per_million: 0.35,
output_per_million: 1.38,
cached_input_per_million: 0.035,
pricing_tbd: false,
},
capabilities: {
tools: true,
vision: false,
reasoning: true,
},
limits: {
max_context_length: 202_752,
max_output_tokens: 16_384,
},
},
},
{
id: 'coming-soon-model',
metadata: {
display_name: 'Coming Soon',
pricing: {
input_per_million: 0,
output_per_million: 0,
pricing_tbd: true,
},
capabilities: { tools: true },
limits: { max_context_length: 8_000, max_output_tokens: 1_000 },
},
},
{
id: 'deprecated-model',
metadata: {
display_name: 'Old',
deprecated: true,
pricing: {
input_per_million: 1,
output_per_million: 2,
pricing_tbd: false,
},
capabilities: { tools: true },
limits: { max_context_length: 8_000, max_output_tokens: 1_000 },
},
},
];
const mockCatalogAndQuota = (
accountingMethod: 'energy' | 'token' = 'energy',
) => {
axiosRequestMock.mockImplementation(async (opts: { url?: string }) => {
if (opts.url?.endsWith('/quota')) {
return {
data: {
balance: { accounting_method: accountingMethod },
},
};
}
return { data: { data: SAMPLE_API_MODELS } };
});
};
beforeAll(async () => {
server = await setupTestServer();
});
afterAll(async () => {
await server?.shutdown();
});
const makeProvider = (config?: { apiBaseUrl?: string }) => {
const provider = new NeuralwattProvider(
{ apiKey: 'test-key', ...config },
server.services.metering,
);
return { provider };
};
const asAsyncIterable = <T>(items: T[]): AsyncIterable<T> => ({
async *[Symbol.asyncIterator]() {
for (const item of items) {
yield item;
}
},
});
const makeCapturingChatStream = () => {
const chunks: string[] = [];
const sink = new Writable({
write(chunk, _enc, cb) {
chunks.push(chunk.toString('utf8'));
cb();
},
});
const chatStream = new AIChatStream({ stream: sink });
return {
chatStream,
events: () =>
chunks
.join('')
.split('\n')
.filter(Boolean)
.map((line) => JSON.parse(line)),
};
};
beforeEach(() => {
createMock.mockReset();
openAICtor.mockReset();
axiosRequestMock.mockReset();
mockCatalogAndQuota('energy');
kv.del(KV_MODELS_KEY);
kv.del(KV_QUOTA_KEY);
recordSpy = vi.spyOn(server.services.metering, 'utilRecordUsageObject');
});
afterEach(() => {
vi.restoreAllMocks();
kv.del(KV_MODELS_KEY);
kv.del(KV_QUOTA_KEY);
});
// ── Mapping helpers ─────────────────────────────────────────────────
describe('Neuralwatt model mapping helpers', () => {
it('strips the neuralwatt: prefix for upstream ids', () => {
expect(stripNeuralwattPrefix('neuralwatt:deepseek-v4-flash')).toBe(
'deepseek-v4-flash',
);
expect(stripNeuralwattPrefix('deepseek-v4-flash')).toBe(
'deepseek-v4-flash',
);
});
it('maps catalog pricing into usd-cents cost keys and skips pricing_tbd', () => {
const mapped = mapNeuralwattApiModel(SAMPLE_API_MODELS[0]!);
expect(mapped).toMatchObject({
id: 'neuralwatt:deepseek-v4-flash',
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 0.14 * 100,
completion_tokens: 0.28 * 100,
cached_tokens: 0.014 * 100,
},
tool_call: true,
max_tokens: 384_000,
modalities: { input: ['text'], output: ['text'] },
});
expect(mapNeuralwattApiModel(SAMPLE_API_MODELS[2]!)).toBeNull();
});
it('marks vision models from capabilities.vision', () => {
const mapped = mapNeuralwattApiModel({
id: 'gemma-4-31b',
metadata: {
display_name: 'Gemma 4 31B',
pricing: {
input_per_million: 0.1,
output_per_million: 0.2,
pricing_tbd: false,
},
capabilities: { tools: true, vision: true },
limits: {
max_context_length: 256_000,
max_output_tokens: 16_384,
max_images: 8,
},
},
});
expect(mapped).toMatchObject({
id: 'neuralwatt:gemma-4-31b',
modalities: { input: ['text', 'image'], output: ['text'] },
max_images: 8,
});
});
});
// ── Construction ────────────────────────────────────────────────────
describe('NeuralwattProvider construction', () => {
it('points the OpenAI SDK at the Neuralwatt base URL with the configured key', () => {
makeProvider();
expect(openAICtor).toHaveBeenCalledTimes(1);
expect(openAICtor).toHaveBeenCalledWith({
apiKey: 'test-key',
baseURL: 'https://api.neuralwatt.com/v1',
});
});
it('honours an apiBaseUrl override', () => {
makeProvider({ apiBaseUrl: 'https://custom.neuralwatt.example/v1' });
expect(openAICtor).toHaveBeenLastCalledWith({
apiKey: 'test-key',
baseURL: 'https://custom.neuralwatt.example/v1',
});
});
});
// ── Model catalog + quota ───────────────────────────────────────────
describe('NeuralwattProvider model catalog', () => {
it('returns the neuralwatt-prefixed default model id', () => {
const { provider } = makeProvider();
expect(provider.getDefaultModel()).toBe(NEURALWATT_DEFAULT_MODEL);
});
it('sends the API key as a bearer token on the catalog fetch', async () => {
const { provider } = makeProvider();
await provider.models();
const modelsCall = axiosRequestMock.mock.calls.find(
([args]) =>
typeof args?.url === 'string' && args.url.endsWith('/models'),
);
expect(modelsCall?.[0]).toMatchObject({
url: 'https://api.neuralwatt.com/v1/models',
headers: { Authorization: 'Bearer test-key' },
});
});
it('list() prefixes ids and skips deprecated / pricing_tbd entries', async () => {
const { provider } = makeProvider();
const ids = await provider.list();
expect(ids).toContain('neuralwatt:deepseek-v4-flash');
expect(ids).toContain('neuralwatt:zai-org/GLM-5.1-FP8');
expect(ids).toContain('GLM-5.1-FP8');
expect(ids).not.toContain('neuralwatt:coming-soon-model');
expect(ids).not.toContain('neuralwatt:deprecated-model');
});
it('caches the model list in kv after the first axios round-trip', async () => {
const { provider } = makeProvider();
await provider.models();
await provider.models();
const modelsCalls = axiosRequestMock.mock.calls.filter(
([args]) =>
typeof args?.url === 'string' && args.url.endsWith('/models'),
);
expect(modelsCalls).toHaveLength(1);
});
it('caches accounting_method from /quota', async () => {
const { provider } = makeProvider();
await expect(provider.getAccountingMethod()).resolves.toBe('energy');
await expect(provider.getAccountingMethod()).resolves.toBe('energy');
const quotaCalls = axiosRequestMock.mock.calls.filter(
([args]) =>
typeof args?.url === 'string' && args.url.endsWith('/quota'),
);
expect(quotaCalls).toHaveLength(1);
});
});
// ── Request shape ───────────────────────────────────────────────────
describe('NeuralwattProvider.complete request shape', () => {
const baseCompletion = {
choices: [
{
message: { content: 'hi', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 1, completion_tokens: 1 },
cost: { request_cost_usd: 0 },
energy: {
energy_kwh: 0.000001,
energy_joules: 3.6,
measurement_available: true,
},
};
it('strips the neuralwatt: prefix from the wire model id', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'neuralwatt:deepseek-v4-flash',
messages: [{ role: 'user', content: 'hello' }],
}),
);
const [args] = createMock.mock.calls[0]!;
expect(args.model).toBe('deepseek-v4-flash');
});
it('only sets stream_options.include_usage when streaming', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'neuralwatt:deepseek-v4-flash',
messages: [{ role: 'user', content: 'hi' }],
stream: false,
}),
);
expect(createMock.mock.calls[0]![0].stream).toBe(false);
expect('stream_options' in createMock.mock.calls[0]![0]).toBe(false);
createMock.mockReturnValueOnce(asAsyncIterable([]));
await withTestActor(() =>
provider.complete({
model: 'neuralwatt:deepseek-v4-flash',
messages: [{ role: 'user', content: 'hi' }],
stream: true,
}),
);
expect(createMock.mock.calls[1]![0].stream_options).toEqual({
include_usage: true,
});
});
it('rejects image content on a text-only catalog model', async () => {
const { provider } = makeProvider();
await expect(
withTestActor(() =>
provider.complete({
model: 'neuralwatt:deepseek-v4-flash',
messages: [
{
role: 'user',
content: [
{
type: 'image_url',
image_url: {
url: 'https://example.com/cat.png',
},
},
],
},
],
}),
),
).rejects.toMatchObject({
statusCode: 400,
message: expect.stringContaining('does not support image input'),
});
expect(createMock).not.toHaveBeenCalled();
});
it('prefers a vision catalog model when the prompt has images and no model was named', async () => {
// Seed a vision model into the catalog payload.
axiosRequestMock.mockImplementation(async (opts: { url?: string }) => {
if (opts.url?.endsWith('/quota')) {
return {
data: { balance: { accounting_method: 'energy' } },
};
}
return {
data: {
data: [
...SAMPLE_API_MODELS,
{
id: 'gemma-4-31b',
metadata: {
display_name: 'Gemma 4 31B',
pricing: {
input_per_million: 0.1,
output_per_million: 0.2,
pricing_tbd: false,
},
capabilities: {
tools: true,
vision: true,
},
limits: {
max_context_length: 256_000,
max_output_tokens: 16_384,
},
},
},
],
},
};
});
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'a cat', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 1, completion_tokens: 1 },
cost: { request_cost_usd: 0 },
});
await withTestActor(() =>
provider.complete({
model: '',
messages: [
{
role: 'user',
content: [
{
type: 'image_url',
image_url: {
url: 'data:image/png;base64,abc',
},
},
],
},
],
}),
);
expect(createMock.mock.calls[0]![0].model).toBe('gemma-4-31b');
});
});
// ── Non-stream metering ─────────────────────────────────────────────
describe('NeuralwattProvider.complete non-stream output', () => {
it('bills from cost.request_cost_usd and records energy units at zero cost', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'hi there', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: {
prompt_tokens: 100,
completion_tokens: 50,
prompt_tokens_details: { cached_tokens: 10 },
},
cost: { request_cost_usd: 0.0001 },
energy: {
energy_kwh: 0.00000145,
energy_joules: 5.23,
measurement_available: true,
},
});
const result = (await withTestActor(() =>
provider.complete({
model: 'neuralwatt:deepseek-v4-flash',
messages: [{ role: 'user', content: 'hi' }],
}),
)) as { usage: Record<string, number | string> };
expect(recordSpy).toHaveBeenCalledTimes(1);
const [usage, , prefix, overrides] = recordSpy.mock.calls[0]!;
expect(prefix).toBe('neuralwatt:deepseek-v4-flash');
expect(usage).toMatchObject({
prompt_tokens: 100,
completion_tokens: 50,
cached_tokens: 10,
energy_kwh: 0.00000145,
energy_joules: 5.23,
billedUsage: 1,
});
expect(overrides).toMatchObject({
prompt_tokens: 0,
completion_tokens: 0,
cached_tokens: 0,
energy_kwh: 0,
energy_joules: 0,
billedUsage: 0.0001 * 100_000_000,
});
expect(result.usage.usd_cents).toBe(0.0001 * 100);
expect(result.usage.accounting_method).toBe('energy');
});
it('falls back to catalog token pricing when request_cost_usd is absent', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'ok', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: {
prompt_tokens: 100,
completion_tokens: 50,
prompt_tokens_details: { cached_tokens: 10 },
},
});
await withTestActor(() =>
provider.complete({
model: 'neuralwatt:deepseek-v4-flash',
messages: [{ role: 'user', content: 'hi' }],
}),
);
// usdPerMToken: $0.14/M → 14 cents per MTok unit in costs map
const promptRate = 0.14 * 100;
const completionRate = 0.28 * 100;
const cachedRate = 0.014 * 100;
const [usage, , , overrides] = recordSpy.mock.calls[0]!;
expect(usage).toMatchObject({
prompt_tokens: 100,
completion_tokens: 50,
cached_tokens: 10,
});
expect(overrides).toMatchObject({
prompt_tokens: 100 * promptRate,
completion_tokens: 50 * completionRate,
cached_tokens: 10 * cachedRate,
});
});
});
// ── Streaming ───────────────────────────────────────────────────────
describe('NeuralwattProvider.complete streaming', () => {
it('streams text deltas and meters final-chunk cost + energy', async () => {
const { provider } = makeProvider();
createMock.mockReturnValueOnce(
asAsyncIterable([
{
choices: [
{
delta: { content: 'Hello' },
finish_reason: null,
},
],
},
{
choices: [
{
delta: { content: '!' },
finish_reason: 'stop',
},
],
usage: {
prompt_tokens: 10,
completion_tokens: 2,
},
cost: { request_cost_usd: 0.00005 },
energy: {
energy_kwh: 0.000002,
energy_joules: 7.2,
measurement_available: true,
},
},
]),
);
const result = (await withTestActor(() =>
provider.complete({
model: 'neuralwatt:deepseek-v4-flash',
messages: [{ role: 'user', content: 'hi' }],
stream: true,
}),
)) as {
stream: true;
init_chat_stream: (p: {
chatStream: AIChatStream;
}) => Promise<void>;
};
const { chatStream, events } = makeCapturingChatStream();
await result.init_chat_stream({ chatStream });
const textEvents = events().filter((e) => e.type === 'text');
expect(textEvents.map((e) => e.text).join('')).toBe('Hello!');
expect(recordSpy).toHaveBeenCalledTimes(1);
const [usage, , , overrides] = recordSpy.mock.calls[0]!;
expect(usage).toMatchObject({
prompt_tokens: 10,
completion_tokens: 2,
energy_kwh: 0.000002,
energy_joules: 7.2,
billedUsage: 1,
});
expect(overrides).toMatchObject({
billedUsage: 0.00005 * 100_000_000,
energy_kwh: 0,
});
});
});
@@ -0,0 +1,384 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import axios from 'axios';
import { OpenAI } from 'openai';
import { ChatCompletionCreateParams } from 'openai/resources';
import { Context } from '../../../../core/context.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import { kv } from '../../../../util/kvSingleton.js';
import * as OpenAIUtil from '../../utils/OpenAIUtil.js';
import type {
IChatModel,
IChatProvider,
IChatCompleteResult,
ICompleteArguments,
} from '../../types.js';
import { inlineHttpImageUrls } from '../moonshot/imageHandling.js';
import {
mapNeuralwattApiModel,
messagesHaveImageContent,
modelSupportsVision,
NEURALWATT_DEFAULT_MODEL,
NEURALWATT_ID_PREFIX,
stripNeuralwattPrefix,
type NeuralwattAccountingMethod,
type NeuralwattApiModel,
type NeuralwattCost,
type NeuralwattEnergy,
} from './models.js';
const DEFAULT_API_BASE_URL = 'https://api.neuralwatt.com/v1';
const KV_MODELS_KEY = 'neuralwattChat:models';
const KV_QUOTA_KEY = 'neuralwattChat:quota';
const CACHE_TTL_SEC = 15 * 60;
type NeuralwattUsage = OpenAI.Completions.CompletionUsage & {
request_cost_usd?: number;
energy_kwh?: number;
energy_joules?: number;
measurement_available?: boolean;
};
export class NeuralwattProvider implements IChatProvider {
#meteringService: MeteringService;
#openai: OpenAI;
#apiKey: string;
#apiBaseUrl: string = DEFAULT_API_BASE_URL;
constructor(
config: { apiBaseUrl?: string; apiKey: string },
meteringService: MeteringService,
) {
this.#apiBaseUrl = config.apiBaseUrl || DEFAULT_API_BASE_URL;
this.#apiKey = config.apiKey;
this.#openai = new OpenAI({
apiKey: config.apiKey,
baseURL: this.#apiBaseUrl,
});
this.#meteringService = meteringService;
}
getDefaultModel() {
return NEURALWATT_DEFAULT_MODEL;
}
async list() {
const models = await this.models();
const modelNames: string[] = [];
for (const model of models) {
modelNames.push(model.id);
if (model.aliases) modelNames.push(...model.aliases);
}
return modelNames;
}
async models(): Promise<IChatModel[]> {
let apiModels = kv.get(KV_MODELS_KEY) as
| NeuralwattApiModel[]
| undefined;
if (!apiModels) {
try {
const resp = await axios.request({
method: 'GET',
url: `${this.#apiBaseUrl}/models`,
headers: {
Authorization: `Bearer ${this.#apiKey}`,
},
});
apiModels = resp.data.data ?? [];
kv.set(KV_MODELS_KEY, apiModels, { EX: CACHE_TTL_SEC });
} catch (e) {
console.error(
'Failed to fetch Neuralwatt models:',
(e as Error).message,
);
}
}
if (!apiModels) return [];
const coerced: IChatModel[] = [];
for (const model of apiModels) {
if (model.metadata?.deprecated) continue;
const mapped = mapNeuralwattApiModel(model);
if (mapped) coerced.push(mapped);
}
return coerced;
}
/**
* Cached account accounting method (`energy` | `token`) from
* `GET /v1/quota`. Used only to annotate returned usage — billing
* always prefers `cost.request_cost_usd` on the completion.
*/
async getAccountingMethod(): Promise<
NeuralwattAccountingMethod | undefined
> {
let method = kv.get(KV_QUOTA_KEY) as
| NeuralwattAccountingMethod
| undefined;
if (method === 'energy' || method === 'token') return method;
try {
const resp = await axios.request({
method: 'GET',
url: `${this.#apiBaseUrl}/quota`,
headers: {
Authorization: `Bearer ${this.#apiKey}`,
},
});
const raw = resp.data?.balance?.accounting_method;
if (raw === 'energy' || raw === 'token') {
method = raw;
kv.set(KV_QUOTA_KEY, method, { EX: CACHE_TTL_SEC });
return method;
}
} catch (e) {
console.error(
'Failed to fetch Neuralwatt quota:',
(e as Error).message,
);
}
return undefined;
}
async complete({
messages,
stream,
model,
tools,
max_tokens,
temperature,
reasoning_effort,
reasoning,
}: ICompleteArguments): Promise<IChatCompleteResult> {
// Catalog carries per-model vision / reasoning_effort flags from
// Neuralwatt `GET /models` — resolve against it before shaping the
// upstream request so image-bearing prompts land on a vision model.
const availableModels = await this.models();
const hasImages = messagesHaveImageContent(messages ?? []);
const modelLower = (model ?? '').toLowerCase();
let modelUsed =
availableModels.find((m) =>
[m.id, ...(m.aliases || [])].some(
(id) => id.toLowerCase() === modelLower,
),
) ?? undefined;
if (!modelUsed) {
if (hasImages) {
modelUsed = availableModels.find((m) =>
modelSupportsVision(m),
);
}
modelUsed =
modelUsed ||
availableModels.find((m) => m.id === this.getDefaultModel()) ||
availableModels[0];
}
if (!modelUsed) {
throw new Error('No Neuralwatt models available');
}
if (hasImages && !modelSupportsVision(modelUsed)) {
throw new HttpError(
400,
`Model ${modelUsed.id} does not support image input`,
{ legacyCode: 'bad_request' },
);
}
const modelIdForParams = stripNeuralwattPrefix(modelUsed.id);
const actor = Context.get('actor');
const accountingMethod = await this.getAccountingMethod();
// Vision models: Neuralwatt (like Moonshot) expects inline data URLs
// rather than remote http(s) fetches for image_url parts.
if (modelSupportsVision(modelUsed)) {
await inlineHttpImageUrls(messages);
}
messages = await OpenAIUtil.process_input_messages(messages);
const requestedReasoningEffort =
reasoning_effort ?? reasoning?.effort;
const supportsReasoningEffort = modelUsed.reasoning_effort === true;
const completionParams = {
messages,
model: modelIdForParams,
...(tools ? { tools } : {}),
...(max_tokens !== undefined ? { max_tokens } : {}),
...(temperature !== undefined ? { temperature } : {}),
...(supportsReasoningEffort && requestedReasoningEffort
? { reasoning_effort: requestedReasoningEffort }
: {}),
stream,
...(stream
? {
stream_options: { include_usage: true },
}
: {}),
} as ChatCompletionCreateParams;
const completion =
await this.#openai.chat.completions.create(completionParams);
const usage_calculator = ({
usage,
cost,
energy,
}: {
usage: NeuralwattUsage;
cost?: NeuralwattCost;
energy?: NeuralwattEnergy;
}) => {
// Non-streaming spreads the full completion into this call;
// streaming merges top-level cost/energy onto `usage` via the
// index_usage_from_stream_chunk deviation below.
const requestCostUsd =
typeof cost?.request_cost_usd === 'number'
? cost.request_cost_usd
: typeof usage.request_cost_usd === 'number'
? usage.request_cost_usd
: undefined;
const energyBlock = energy ?? {
energy_kwh: usage.energy_kwh,
energy_joules: usage.energy_joules,
measurement_available: usage.measurement_available,
};
const trackedTokens = OpenAIUtil.extractMeteredUsage(usage);
const energyUnits: Record<string, number> = {};
if (
energyBlock.measurement_available !== false &&
typeof energyBlock.energy_kwh === 'number' &&
Number.isFinite(energyBlock.energy_kwh) &&
energyBlock.energy_kwh > 0
) {
energyUnits.energy_kwh = energyBlock.energy_kwh;
}
if (
energyBlock.measurement_available !== false &&
typeof energyBlock.energy_joules === 'number' &&
Number.isFinite(energyBlock.energy_joules) &&
energyBlock.energy_joules > 0
) {
energyUnits.energy_joules = energyBlock.energy_joules;
}
const annotate = (tracked: Record<string, number>) => {
const out: Record<string, number | string> = { ...tracked };
if (accountingMethod) {
out.accounting_method = accountingMethod;
}
return out;
};
if (
typeof requestCostUsd === 'number' &&
Number.isFinite(requestCostUsd)
) {
const billedTrackedUsage = {
...trackedTokens,
...energyUnits,
billedUsage: 1,
};
const costOverwrites = Object.fromEntries(
Object.keys(billedTrackedUsage).map((k) => [k, 0]),
);
costOverwrites.billedUsage = requestCostUsd * 100_000_000;
this.#meteringService.utilRecordUsageObject(
billedTrackedUsage,
actor!,
modelUsed.id,
costOverwrites,
);
const result = annotate(billedTrackedUsage);
result.usd_cents = requestCostUsd * 100;
return result;
}
// Fallback: catalog token rates (preflight / when Neuralwatt
// omits request_cost_usd).
const trackedUsage = { ...trackedTokens, ...energyUnits };
const costOverwrites = Object.fromEntries(
Object.entries(trackedUsage).map(([k, v]) => {
if (k === 'energy_kwh' || k === 'energy_joules') {
return [k, 0];
}
return [k, (modelUsed.costs[k] ?? 0) * v];
}),
);
this.#meteringService.utilRecordUsageObject(
trackedUsage,
actor!,
modelUsed.id,
costOverwrites,
);
return annotate(trackedUsage);
};
return OpenAIUtil.handle_completion_output({
deviations: {
index_usage_from_stream_chunk: (chunk: {
usage?: NeuralwattUsage;
cost?: NeuralwattCost;
energy?: NeuralwattEnergy;
}) => {
if (!chunk.usage) return chunk.usage;
return {
...chunk.usage,
...(typeof chunk.cost?.request_cost_usd === 'number'
? {
request_cost_usd:
chunk.cost.request_cost_usd,
}
: {}),
...(chunk.energy
? {
energy_kwh: chunk.energy.energy_kwh,
energy_joules: chunk.energy.energy_joules,
measurement_available:
chunk.energy.measurement_available,
}
: {}),
};
},
},
usage_calculator,
stream,
completion,
});
}
checkModeration(
_text: string,
): ReturnType<IChatProvider['checkModeration']> {
throw new Error('Method not implemented.');
}
}
export { NEURALWATT_ID_PREFIX, NEURALWATT_DEFAULT_MODEL };
@@ -0,0 +1,186 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { IChatModel } from '../../types.js';
import { usdPerMToken } from '../../utils/pricing.js';
export const NEURALWATT_ID_PREFIX = 'neuralwatt:';
export const NEURALWATT_DEFAULT_MODEL = 'neuralwatt:deepseek-v4-flash';
/**
* Shape of one entry in Neuralwatt's `GET /v1/models` catalog.
* Pricing is USD per million tokens under `metadata.pricing` and is used
* only for Puter preflight estimates / token-cost fallback. Live billing
* uses each completion's `cost.request_cost_usd`.
*/
export type NeuralwattApiModel = {
id: string;
object?: string;
created?: number;
owned_by?: string;
max_model_len?: number;
metadata?: {
display_name?: string;
description?: string | null;
provider?: string;
huggingface_id?: string | null;
pricing?: {
input_per_million?: number;
output_per_million?: number;
cached_input_per_million?: number | null;
cached_output_per_million?: number | null;
currency?: string;
pricing_tbd?: boolean;
};
capabilities?: {
tools?: boolean;
json_mode?: boolean;
vision?: boolean;
reasoning?: boolean;
reasoning_effort?: boolean;
streaming?: boolean;
system_role?: boolean;
developer_role?: boolean;
};
limits?: {
max_context_length?: number | null;
max_output_tokens?: number | null;
max_images?: number | null;
};
deprecated?: boolean;
deprecated_message?: string | null;
};
};
export type NeuralwattAccountingMethod = 'energy' | 'token';
export type NeuralwattEnergy = {
energy_joules?: number;
energy_kwh?: number;
measurement_available?: boolean;
avg_power_watts?: number;
duration_seconds?: number;
attribution_method?: string;
attribution_ratio?: number;
};
export type NeuralwattCost = {
request_cost_usd?: number;
cache_savings_usd?: number;
allowance_remaining_usd?: number;
};
/** Strip the Puter-facing `neuralwatt:` prefix for upstream API calls. */
export const stripNeuralwattPrefix = (modelId: string): string =>
modelId.startsWith(NEURALWATT_ID_PREFIX)
? modelId.slice(NEURALWATT_ID_PREFIX.length)
: modelId;
/**
* Map a Neuralwatt catalog entry to Puter's `IChatModel`. Returns `null`
* when pricing is TBD (placeholders) so the model is not offered for
* preflight credit checks until Neuralwatt publishes real rates.
*/
export const mapNeuralwattApiModel = (
model: NeuralwattApiModel,
): IChatModel | null => {
const pricing = model.metadata?.pricing;
if (pricing?.pricing_tbd) return null;
const inputUsd = Number(pricing?.input_per_million ?? 0);
const outputUsd = Number(pricing?.output_per_million ?? 0);
const cachedUsd =
pricing?.cached_input_per_million == null
? 0
: Number(pricing.cached_input_per_million);
const caps = model.metadata?.capabilities;
const limits = model.metadata?.limits;
const context =
limits?.max_context_length ?? model.max_model_len ?? undefined;
const maxTokens = limits?.max_output_tokens ?? context ?? 0;
const inputModalities = ['text'];
if (caps?.vision) inputModalities.push('image');
const displayName = model.metadata?.display_name || model.id;
const pathTail = model.id.includes('/')
? model.id.split('/').slice(1).join('/')
: undefined;
return {
id: `${NEURALWATT_ID_PREFIX}${model.id}`,
name: `${displayName} (Neuralwatt)`,
aliases: [
model.id,
`neuralwatt/${model.id}`,
...(pathTail && pathTail !== model.id ? [pathTail] : []),
],
context,
max_tokens: maxTokens,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: usdPerMToken(inputUsd, outputUsd, cachedUsd),
modalities: { input: inputModalities, output: ['text'] },
tool_call: caps?.tools === true,
// Surfaced from Neuralwatt `/models` so `complete()` can gate
// reasoning_effort without re-fetching the catalog entry.
reasoning_effort: caps?.reasoning_effort === true,
...(typeof limits?.max_images === 'number'
? { max_images: limits.max_images }
: {}),
...(model.metadata?.description
? { description: model.metadata.description }
: {}),
...(model.created
? {
release_date: new Date(model.created * 1000)
.toISOString()
.slice(0, 10),
}
: {}),
};
};
/** True when the Puter model catalog entry advertises image input. */
export const modelSupportsVision = (model: IChatModel): boolean =>
Array.isArray(model.modalities?.input) &&
model.modalities.input.includes('image');
/**
* Detect image / puter_path parts so we can prefer a vision-capable model
* from the Neuralwatt catalog (or reject a text-only pick).
*/
export const messagesHaveImageContent = (
messages: Array<{ content?: unknown }>,
): boolean => {
for (const message of messages) {
if (!Array.isArray(message.content)) continue;
for (const part of message.content as Array<Record<string, unknown>>) {
if (!part || typeof part !== 'object') continue;
if (part.type === 'image_url' || part.image_url) return true;
if (typeof part.puter_path === 'string' && part.puter_path) {
return true;
}
}
}
return false;
};
@@ -728,19 +728,4 @@ export const OPEN_AI_MODELS: IChatModel[] = [
context: 1_047_576,
max_tokens: 32768,
},
{
puterId: 'openai:openai/gpt-4.5-preview',
id: 'gpt-4.5-preview',
aliases: ['openai/gpt-4.5-preview'],
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 7500,
completion_tokens: 15000,
},
context: 128_000,
max_tokens: 32768,
},
];
@@ -105,6 +105,15 @@ const SAMPLE_API_MODELS = [
pricing: { prompt: 0.000002, completion: 0.00001 },
top_provider: { max_completion_tokens: 8192 },
},
{
// OpenRouter reports no separate output cap for some models.
id: 'meta/muse-spark-1.2',
name: 'Muse Spark 1.2',
created: 1786032000,
context_length: 1048576,
pricing: { prompt: 0.00000125, completion: 0.00000425 },
top_provider: { max_completion_tokens: null },
},
{
// 'openrouter/auto' is filtered out — disallowed.
id: 'openrouter/auto',
@@ -228,6 +237,25 @@ describe('OpenRouterProvider model catalog', () => {
expect(axiosRequestMock).toHaveBeenCalledTimes(1);
});
it('falls back to the context window when no output cap is reported', async () => {
const { provider } = makeProvider();
const models = await provider.models();
// A null max_completion_tokens previously landed on the model as-is,
// which made the driver's cap arithmetic go negative and reject the
// request as insufficient funds.
const noCap = models.find(
(m) => m.id === 'openrouter:meta/muse-spark-1.2',
)!;
expect(noCap.max_tokens).toBe(1048576);
// A model that reports its own cap keeps it.
const capped = models.find(
(m) => m.id === 'openrouter:openai/gpt-5-nano',
)!;
expect(capped.max_tokens).toBe(16000);
});
it('maps OpenRouter created timestamps to release_date metadata', async () => {
const { provider } = makeProvider();
const models = await provider.models();
@@ -72,9 +72,11 @@ export class OpenRouterProvider implements IChatProvider {
}
/**
* Returns a list of available model names including their aliases
*
* Retrieves all available model IDs and their aliases, flattening them into
* a single array of strings that can be used for model selection
*
* @returns {Promise<string[]>} Array of model identifiers and their aliases
* @description Retrieves all available model IDs and their aliases,
* flattening them into a single array of strings that can be used for model selection
*/
async list() {
const models = await this.models();
@@ -85,10 +87,7 @@ export class OpenRouterProvider implements IChatProvider {
return model_names;
}
/**
* AI Chat completion method.
* See AIChatService for more details.
*/
/** AI Chat completion method. See AIChatService for more details. */
async complete({
messages,
stream,
@@ -279,7 +278,11 @@ export class OpenRouterProvider implements IChatProvider {
model.id.split('/').slice(1).join('/'),
],
context: model.context_length,
max_tokens: model.top_provider.max_completion_tokens,
// OpenRouter leaves max_completion_tokens null when a model
// declares no output cap separate from its context window.
max_tokens:
model.top_provider.max_completion_tokens ??
model.context_length,
costs_currency: 'usd-cents',
input_cost_key: 'prompt',
output_cost_key: 'completion',
@@ -63,5 +63,33 @@ describe.skipIf(skipUnlessEnv(ENV_VAR))(
?.content;
expect(typeof text === 'string' && text.length > 0).toBe(true);
});
it('recovers when max_tokens leaves no room for the prompt', { timeout: INTEGRATION_TEST_TIMEOUT_MS }, async () => {
const provider = new TogetherAIProvider(
{ apiKey: optionalEnv(ENV_VAR)! },
makeMeteringStub(),
);
// Asking for the model's whole context as output leaves no room
// for the prompt, which Together rejects outright — the provider
// should retry uncapped rather than surface a 400.
const model = (await provider.models()).find(
(m) => m.id === 'togetherai:Qwen/Qwen2.5-7B-Instruct-Turbo',
)!;
const result = await withTestActor(() =>
provider.complete({
model: model.id,
messages: [
{ role: 'user', content: 'Say hi in one word.' },
],
max_tokens: model.context!,
}),
);
const text = (result as { message?: { content?: string } }).message
?.content;
expect(typeof text === 'string' && text.length > 0).toBe(true);
});
},
);
@@ -209,6 +209,17 @@ describe('TogetherAIProvider model catalog', () => {
expect(ids).toContain('model-fallback-test-1');
});
it('reserves headroom under the context length for the output cap', async () => {
const { provider } = makeProvider();
const model = (await provider.models()).find(
(m) => m.id === 'togetherai:Qwen/Qwen2.5-7B-Instruct-Turbo',
)!;
// The advertised context window is unchanged; only the output cap
// leaves room for the driver's under-counting input estimator.
expect(model.context).toBe(32768);
expect(model.max_tokens).toBe(Math.floor(32768 * 0.95));
});
it('caches the coerced model list in kv after the first call', async () => {
const { provider } = makeProvider();
await provider.models();
@@ -357,6 +368,109 @@ describe('TogetherAIProvider.complete request shape', () => {
expect(createMock).not.toHaveBeenCalled();
expect(recordSpy).not.toHaveBeenCalled();
});
// Together's APIError carries the whole response body on `.error`, so the
// provider message sits at `.error.error.message` — one level deeper than
// the OpenAI SDK puts it.
const contextLengthError = {
status: 400,
message:
'400 {"id":"ovG6YRd-6z2FuN","error":{"message":"Failed to start generation: The input token count (11) plus the requested output count (1048573) exceeds the model\'s maximum context length (1048576)"}}',
error: {
id: 'ovG6YRd-6z2FuN',
error: {
message:
"Failed to start generation: The input token count (11) plus the requested output count (1048573) exceeds the model's maximum context length (1048576)",
type: 'invalid_request_error',
},
},
};
it('retries without max_tokens when Together rejects with a context-length error', async () => {
const { provider } = makeProvider();
createMock
.mockRejectedValueOnce(contextLengthError)
.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'togetherai:Qwen/Qwen2.5-7B-Instruct-Turbo',
messages: [{ role: 'user', content: 'hi' }],
max_tokens: 1048573,
}),
);
// Provider mutates a single completionParams object across both calls
// (`delete completionParams.max_tokens` after the first throw), so the
// first call's recorded args are retroactively altered — only the
// surviving shape is worth asserting on.
expect(createMock).toHaveBeenCalledTimes(2);
expect('max_tokens' in createMock.mock.calls[1]![0]).toBe(false);
// The retry is still metered exactly once.
expect(recordSpy).toHaveBeenCalledTimes(1);
});
it('retries a streaming request the same way', async () => {
const { provider } = makeProvider();
createMock
.mockRejectedValueOnce(contextLengthError)
.mockReturnValueOnce(asAsyncIterable([]));
await withTestActor(() =>
provider.complete({
model: 'togetherai:Qwen/Qwen2.5-7B-Instruct-Turbo',
messages: [{ role: 'user', content: 'hi' }],
max_tokens: 1048573,
stream: true,
}),
);
expect(createMock).toHaveBeenCalledTimes(2);
expect('max_tokens' in createMock.mock.calls[1]![0]).toBe(false);
expect(createMock.mock.calls[1]![0].stream).toBe(true);
});
it('rethrows non-context-length errors without retrying', async () => {
const { provider } = makeProvider();
const apiError = {
status: 401,
error: { error: { message: 'Invalid API key' } },
};
createMock.mockRejectedValueOnce(apiError);
await expect(
withTestActor(() =>
provider.complete({
model: 'togetherai:Qwen/Qwen2.5-7B-Instruct-Turbo',
messages: [{ role: 'user', content: 'boom' }],
max_tokens: 100,
}),
),
).rejects.toBe(apiError);
expect(createMock).toHaveBeenCalledTimes(1);
expect(recordSpy).not.toHaveBeenCalled();
});
it('rethrows a bodyless connection error instead of failing to read it', async () => {
const { provider } = makeProvider();
const connErr = new Error('Connection error.');
createMock.mockRejectedValueOnce(connErr);
await expect(
withTestActor(() =>
provider.complete({
model: 'togetherai:Qwen/Qwen2.5-7B-Instruct-Turbo',
messages: [{ role: 'user', content: 'hi' }],
max_tokens: 100,
}),
),
).rejects.toBe(connErr);
expect(createMock).toHaveBeenCalledTimes(1);
});
});
// ── Model resolution ────────────────────────────────────────────────
@@ -29,6 +29,24 @@ const TOGETHER_AI_CHAT_COST_MAP = {
completion_tokens: 'output',
};
/**
* Whether the SDK rejected a request because the prompt plus the requested
* output exceeds the model's context window. Unlike the OpenAI SDK, Together's
* `APIError.error` is the whole response body, so the provider message sits one
* level deeper; `message` is the stringified body and covers older shapes.
*/
const isContextLengthError = (e: unknown) => {
const err = e as {
error?: { error?: { message?: string } };
message?: string;
};
const message = err?.error?.error?.message ?? err?.message;
return (
typeof message === 'string' &&
message.includes('maximum context length')
);
};
export class TogetherAIProvider implements IChatProvider {
#together: Together;
@@ -83,7 +101,13 @@ export class TogetherAIProvider implements IChatProvider {
),
),
},
max_tokens: model.context_length ?? 8000,
// Together only reports a context length. The driver caps
// output at max_tokens minus an estimated input count, and
// that estimate runs low — reserve headroom so a short
// prompt doesn't ask for more than the context allows.
max_tokens: model.context_length
? Math.floor(model.context_length * 0.95)
: 8000,
});
}
}
@@ -145,7 +169,7 @@ export class TogetherAIProvider implements IChatProvider {
messages = await OpenAIUtil.process_input_messages(messages);
const completion = await this.#together.chat.completions.create({
const completionParams = {
model: modelIdForParams,
messages,
stream,
@@ -153,7 +177,21 @@ export class TogetherAIProvider implements IChatProvider {
...(max_tokens !== undefined ? { max_tokens } : {}),
...(temperature !== undefined ? { temperature } : {}),
...(stream ? { stream_options: { include_usage: true } } : {}),
} as Together.Chat.Completions.CompletionCreateParamsNonStreaming);
} as Together.Chat.Completions.CompletionCreateParamsNonStreaming;
let completion;
try {
completion =
await this.#together.chat.completions.create(completionParams);
} catch (e: unknown) {
// An overestimated max_tokens makes Together reject the request
// outright rather than truncating. The user can afford the query
// either way, so retry once without the cap.
if (!isContextLengthError(e)) throw e;
delete completionParams.max_tokens;
completion =
await this.#together.chat.completions.create(completionParams);
}
return OpenAIUtil.handle_completion_output({
usage_calculator: ({ usage }) => {
Loaded 100 of 360 files, more files were not shown because too many files have changed in this diff. Show more