Compare commits

..
77 Commits
Author SHA1 Message Date
Daniel Salazar 5918e3f4e2 fix: kv max number value (#3629) 2026-08-23 06:31:08 -07:00
Nariman Jelveh 2c9e818e5f Docs/remove discord reddit links (#3628)
* docs: remove Discord and Reddit links

Drops both links from the docs README (header row and Support list) and
from the footer generated on every built docs page.

* docs: remove Mastodon link
2026-08-22 22:14:56 -07:00
Nariman Jelveh 8826a9fc51 Remove Discord and Reddit links from README (#3627)
Removed links to Discord and Reddit from the README.
2026-08-22 13:07:33 -07:00
Daniel Salazar 8307e8c9f0 chore: bump up deps (#3626) 2026-08-21 21:03:54 -07:00
Daniel Salazar 732786a029 fix: clean up email validation (#3622) 2026-08-21 20:27:05 -07:00
f0cd251626 🛠️ PUT-1521: Cleanup puter js permissions api + backend routes (#3607)
* refactor(puter-js): collapse puter.perms request* to resource + access

Fifteen request methods differed only by a folder name or an access level, so
every new resource meant another method. Replace them with requestFolder,
requestApps, requestSubdomains and requestAppRootDir, each taking the access
level as an argument.

The old names stay as @deprecated aliases: puter.js ships unpinned from
js.puter.com/v2, so removing them would break live apps. They remain in the
generated declarations because stripInternal has no effect on declarations
emitted from JavaScript, and hand-omitting them would break TypeScript callers
the runtime still serves.

Also drops the user-to-user and user-to-group grant wrappers (groups.js and the
grantUser/grantGroup half of grants.js) plus the req_ shim, none of which were
documented or called. The app, origin and dev-app grants stay: the dashboard
uses puter.perms.revokeApp() to clear grants on app uninstall.

* docs(perms): document the collapsed puter.perms surface

Replace the twelve one-method-per-task pages with requestFolder, requestApps
and requestSubdomains, and rewrite the Perms overview around the seven public
methods. The deprecated aliases keep working but are no longer documented.

Boy Scout: drops the long-dead commented-out grantUser/revokeOrigin sidebar
block for pages that were never published.

* refactor(perms): drop unused user-to-user and user-to-group permission routes

Filesystem access is shared through /share, which records the grant so the
owner can see and revoke it. The older direct-grant paths were left behind with
no caller anywhere - not the GUI, not a doc, not an app: grant-user-user
(already a 501 stub), revoke-user-user, grant/revoke-user-group, and the five
/group/* CRUD routes.

Removing them orphans PermissionService.grant/revokeUserGroupPermission and its
group-members cache bump, the three PermissionStore group writers, and six
GroupStore methods, so those go too.

What stays, and why:
- grant/revokeUserUserPermission - ACLService and ShareService power fs.share
  through them.
- The group permission read path (#scanUserGroup, readUserGroupPerms) - a
  migration seeds the admin group unrestricted driver access, so it is
  load-bearing.
- GroupStore getByUid/addUsers/removeUsers - signup, save_account, OIDC and the
  self-hosted default user assign group membership.

No schema change: user_to_user_permissions, user_to_group_permissions and their
audit tables are untouched. Group rows now come only from migrations, so tests
that need one seed it with SQL the way a migration does.

* refactor(perms): reduce GroupStore to membership writes

With the /group/* routes gone, `getByUid` had no production caller left — the
routes were the only thing that read a group back. Removing it takes the row
decoder and the GroupRow type with it, since they exist only to shape its
result.

What remains is `addUsers`/`removeUsers`: signup, save_account, OIDC and the
self-hosted admin bootstrap all assign group membership. Permissions attached to
a group are read through PermissionStore, which joins the junction table itself
and never needed the store.

Tests that wanted a group id now select it, which is all `getByUid` was doing
for them.

* chore(perms): drop the three by-hand groups no code reads

freeai, experimental and dangerous exist in prod but in no migration — they
were added by hand when hardcoded permissions were keyed by group name. That
map is now a flat per-user floor (`default_user_permissions`), so a group
nothing looks up grants nothing.

Guarded rather than unconditional, because both tables the delete can reach
cascade: dropping a group that still carries permissions or members would
silently revoke them from every member. Only a group with neither goes. One that
survives has dependents and needs a deliberate decision — query
user_to_group_permissions by group_id to see what it holds.

system, admin, user and temp are untouched: config names two of them and code
names the others.

Matches on `extra.name`, not `metadata.name` — `metadata` carries the display
title and colour, and `critical: true` is set on all of these including freeai,
so it does not discriminate.

* feat(puter-js): collapse puter.perms onto request(resource, details) + check()

One method per task meant a new method, doc page and sidebar entry for every
resource. `request` now takes the resource and a payload whose accepted fields
depend on it, and `check` answers the same question without prompting.

    request('folder', { name: 'Documents', access: 'write' })  -> path
    request('apps', { access: 'read' })                        -> boolean
    request('email')                                           -> address
    check('folder', { name: 'Documents', access: 'write' })     -> boolean

Returns stay per-resource: a folder gives its path, email the address, the rest
a boolean, and anything denied is falsy so one `if` covers both.

An array asks for several at once. Everything already held is settled first, so
the prompt covers only what is missing and does not appear when the whole set is
held - the user answers once for the lot. `check` answers per entry, in order,
so a caller can tell which parts are missing rather than only that some are.

Each resource declares four things in one registry entry: how to ask for it
alone, whether it is held, the strings a batch pools into a prompt, and the
value once held. The strings themselves are defined once in
lib/permissionStrings.js, so a request and its check cannot name them
differently. `check` is built on /auth/check-permissions, already live and
already used by UI.js, and it throws rather than answering false when the check
cannot run: a caller that cannot tell "denied" from "never ran" would prompt
someone who had already granted it.

Backward compatibility: all 22 older methods stay callable and typed, marked
@deprecated with the call that replaces them. A lone string still routes to the
raw-permission path - no resource name contains a `:` and every permission
string does, so the two forms cannot collide. The grant/revoke app methods are
untouched; the consent dialog and the dashboard's uninstall path use them.

Also drops three copies of the access-level assertion onto one shared
validator, and gives `appRootDir` a non-prompting server probe, since
`app-root-dir:` only resolves while a grant is being written and a permission
check on it always answers false.

* docs(perms): document request() and check() as the perms surface

Five per-method pages became one `request()` page carrying the resource table,
the batch form and the raw-string escape hatch, plus a `check()` page. The
overview is rewritten around the two methods.

requestAppData's page is re-homed as /Perms/appData rather than deleted - its
scope table, private-entry guidance and lifetime notes are not signature
documentation and have nowhere else to live. Inbound links from KV/set.md and
Objects/app.md follow it.

Playground examples move to the new call form. They are not wired into
examples.js, but an example demonstrating a deprecated method is worse than one
nobody loads.

* fix(perms): keep /auth/revoke-user-user as a deprecated route

Dropping this route with the rest of the unused user-to-user plumbing went too
far. The grant side is retired and stays retired - puter.fs.share() is the only
way in - but access those grants left behind has to remain withdrawable, and a
caller reaching the endpoint over HTTP directly had no replacement. Revoking can
only ever narrow what someone can reach, so keeping it carries no risk.

revokeUserUserPermission never left the permission service; it is load-bearing
for puter.fs.share(). This only re-wires the handler to it, with the gates it
always had.

Nothing in this repo calls the route, which makes it exactly what a later
cleanup reads as dead, so a test pins the registration and its gate alongside
the restored 400 and grant/revoke round-trip cases.

The 501 stub at grant-user-user and the never-called /group/* routes stay
deleted, as does puter.perms.revokeUser - puter.fs.unshare() replaces it and
falls back to live grants when no share row exists.

* fix(perms): let a write grant satisfy a read check on apps and subdomains

`apps-of-user:<uuid>:write` covers managing the user's apps, which includes
reading them, but nothing said so to the permission system. Prefix implication
only widens the other way — an `apps-of-user:<uuid>` grant covers both modes —
so a scan for `:read` missed a `:write` grant, and `puter.perms.check('apps')`
reported an app holding write as holding nothing. A batched request would then
prompt again for access already granted.

Adds the read-from-write exploder for both namespaces, mirroring
`fs-access-levels`. The widening runs one way only, and does not cross into
another user's namespace; both are covered by tests.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* fix(perms): answer the app-root-dir check without provisioning it

`/auth/request-app-root-dir` conflates two questions: may the caller claim its
root directory, and where is it. The second provisions `AppData/<uid>` on first
ask, so a caller that only wanted the first — `puter.perms.check('appRootDir')`
— created a directory by asking about it.

Adds `check: true`, which runs the same actor guard and stops at the answer.
A caller that may not claim it still gets the 403, so the flag can't widen
anything. Existing callers are unaffected: without it the route behaves exactly
as before.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* fix(perms): put request() and check() on one path

`request` dispatched to the old per-task methods while `check` asked the
permission tables, so the two answered different questions about the same
access. Concretely, before this: `request('folder', { access: 'write' })`,
`'apps'`, `'subdomains'`, `'appData'` and `'permission'` prompted every time,
whether or not the access was held — which the docs said they wouldn't;
`check('folder')` reported false for a folder the app could read through an ACL
grant that no `fs:` string names, so a batch prompted for it needlessly; a
batch entry for `'appRootDir'` skipped the post-grant retry the single call
does, resolving `undefined` after a grant that had in fact succeeded; and an
N-entry batch made N permission reads plus 2N `whoami` calls.

Both now run the same pipeline — resolve the permission strings, read what is
held once, prompt for the remainder, resolve each entry — with per-resource
hooks for the parts only that resource can answer. So a batch costs one
permission read and one `whoami`, a check reports exactly what a request would
skip the prompt for, and `'folder'` uses the same stat-or-permission reading in
both.

Also:

- A resource is looked up as an own property, so `request('constructor')` is
  the permission string it always was rather than a TypeError.
- A permission read that fails no longer decides anything: `request` falls
  through to the prompt it would have raised anyway, `check` throws. Before,
  `check('appRootDir')` folded a failed check into "not granted", which is what
  the documentation says must not happen.
- Drops `requestFolder`, `requestApps`, `requestSubdomains` and
  `requestAppRootDir`. They were added in this branch and immediately deprecated
  — never shipped, and `request()` no longer needs to route through them. The 22
  methods that did ship keep their exact behaviour, prompting without consulting
  what is held, which the suite now asserts alongside the new behaviour.
- Documents `'appRootDir'`, which was a supported resource in every overload and
  in `PermsResource` but named in none of the docs.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* fix(perms): act on review of the three preceding commits

- `src/puter-js/test/perms.test.js` still called `requestApps`,
  `requestFolder`, `requestSubdomains` and `requestAppRootDir`, which the
  previous commit removed. Four cases in the interactive browser harness threw.
  Pointed at `request(...)` instead.

- `request('appRootDir', …)` made two round trips where the shipped method
  makes one: a read-only probe, then the call that names the directory. A
  request is going to claim it either way, so the claim is now the check, and
  the entry it returns carries through to the result. `check` keeps the
  read-only mode, which is the reason that mode exists. Matters because the
  route sits on the FS_SIGN bucket, shared with signed-URL minting.

- `requestPermission` is one of the shipped methods, and the previous commit's
  message was wrong to say all 22 keep their exact behaviour: it forwards to
  `request`, so it now settles a permission the caller already holds instead of
  prompting for it. The value can differ, not just the prompt count — a user who
  would have clicked Deny on a re-prompt used to get `false`. It is the more
  honest answer (the app does hold the access, and denying a re-prompt never
  took it away), but it is a change, and the suite assertion had been switched
  to an unheld permission, which hid it. Asserted both ways instead, in the unit
  tests and the API suite.

- An entry that names no permission no longer rides a grant given for the other
  entries in the same call. Unreachable today — every resource either names one
  or reports itself held — but nothing pinned it.

- Reverted three type-union reformats in `LegacyFSController.ts` that a
  formatter had folded into the app-root-dir commit. That file was not
  prettier-clean to begin with; reformatting it is somebody else's change.

- Docs and types: `Perms.md`'s `appRootDir` row now matches `request.md`'s,
  `check.md` says that a `true` is per entry and a batch still prompts if any
  one entry is missing, and `types.js` no longer names `requestFolder` /
  `requestAppData` in prose that ships in the generated declarations.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

---------

Co-authored-by: Daniel Salazar <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
2026-08-20 23:48:08 -07:00
ZHOU7821 943adae622 docs: use absolute link to router page in workers create docs (#3615) 2026-08-21 13:30:30 +07:00
Daniel Salazar 65572dcf23 fix: nicer sms card fallback (#3621) 2026-08-20 23:12:09 -07:00
ZHOU7821andZHOU7821 b3923f86da docs: fix typo in workers description (#3614)
Co-authored-by: ZHOU7821 <[email protected]>
2026-08-21 12:26:20 +07:00
Nariman Jelveh 729114bd09 Prioritize app-center in recommendations
Reordered the recommended app list so app-center appears earlier and dev-center is kept at the end. This keeps the default app suggestions aligned with the expected homepage/app-center placement without changing behavior beyond ordering.
2026-08-20 19:16:03 -07:00
Nariman Jelveh da1c85d3c5 Add Meetings to recommended app defaults
Include `meetings` in `RECOMMENDED_APP_NAMES` so it appears in the default recommended apps set alongside other core productivity tools.
2026-08-20 17:16:39 -07:00
Nariman Jelveh 66410d7f1f Revert "Let a browser send the device fingerprint header it is offered"
This reverts commit fef1337b2b.
2026-08-20 17:10:55 -07:00
Nariman Jelveh 76bd05f228 Dashboard: let the trial be a detail line, not an alarm
The Usage card gives each of its three columns a meter and two lines. The
plan column had a boxed, tinted, bordered paragraph instead — and at the
~230px a third of that card comes to, thirty words of it wrapped into a
five-line ribbon standing twice as tall as the storage and resources meters
beside it. The box is the shape we use for the dunning banner. A trial is a
date you should know about, not a fault, and it was reading as the loudest
thing on the card.

So it takes the slot its neighbours use for "n% of 100 GB": a quiet hint-
coloured line under the badge. All three columns now measure the same.

The copy stopped repeating itself while it was there. The badge said "ends
Sep 20" and the sentence beneath it said "ends on September 20, 2026" — two
elements, one fact, and neither of them answering the question someone
actually has, which is how much room is left to decide. The badge now
carries "31 days left" and the line under it carries the consequence:
"First charge Sep 20 unless you cancel." Billing keeps the full sentence and
the opt-out, and Manage is directly below.
2026-08-20 16:59:07 -07:00
Nariman Jelveh fef1337b2b Let a browser send the device fingerprint header it is offered
`fingerprint.ts` reads the device fingerprint from a request body or, for
authenticated requests with no body to carry it, from
`x-puter-device-fingerprint`. That header is documented there as "the
header the GUI may send the device fingerprint on for non-signup
requests", and the middleware has always honoured it.

A browser could never actually send it. The CORS allowlist in
`#installCors` does not include it, so the preflight comes back without
it in `Access-Control-Allow-Headers`, the browser abandons the request,
and `fetch` rejects before anything reaches the server. Nothing in tree
sends the header today, which is why this went unnoticed: the first
client to try it sees every call fail with a network error rather than a
readable status.

Adds the header to the allowlist, and a preflight test asserting the
browser is allowed to ask for it.
2026-08-20 12:40:48 -07:00
404oops 58794f4f76 feat: add MetaProvider for Muse Spark models and integrate with ChatCompletionDriver (#3616)
- Implemented MetaProvider to interface with Meta's Muse Spark models.
- Integrated MetaProvider into ChatCompletionDriver, allowing for dynamic model selection.
- Added unit and integration tests for MetaProvider to ensure functionality and reliability.
- Updated documentation to include Meta as a supported vendor for AI models.
2026-08-20 15:09:56 -04:00
Nariman Jelveh 9809e428b5 Update shared label in email template
Adjust the share-notification email copy to reference the current "Shared" files section instead of "Shared with me," in both HTML and plain-text variants. This keeps onboarding messaging aligned with the UI.
2026-08-19 20:05:56 -07:00
Nariman Jelveh eaddd1d44e redesign: share notification emails (#3612)
* redesign: share notification emails

The share digests were an unstyled fragment: a 520px div with a couple
of paragraphs, no preview line, no small-screen or dark-mode handling,
and an html-only body that left every text-only client reading a machine
down-conversion of the markup.

Both digests (`file_shared_with_you` and `file_shared_invite`) now render
through one layout: a 600px table column that goes full width under
600px, every color written inline so a client that drops `<style>` still
gets the design, with the stylesheet carrying only what inline CSS can't
express — the small-screen and dark-mode overrides. The shares are a
tinted panel of hairline-separated rows rather than loose text, the call
to action is a cell-padded button Outlook can draw, and the reason for
receiving plus the unsubscribe sit below the card where they belong. No
images at all, so nothing depends on an asset a client may refuse.

Templates may now carry a `text` part, and both digests do: it goes out
beside the HTML as multipart/alternative, which is what plain-text
clients, screen readers and spam scoring prefer.

Copy: a preview line that adds to the subject instead of repeating it, a
heading before the list, and an honest unsubscribe label — the link
silences notification mail account-wide, not just shares. The invite now
says what to do with the address and that an existing account can claim
the shares by confirming it.

Also drops `share_by_username` and `share_by_email`, unreferenced since
the digests replaced them.

* fix: widen the template map to the interface before compiling

`Object.entries` over the literal map yields a union with a distinct
type per template, and only the share digests carry a `text` part —
reading it off the union is an error the build tsconfig hides
(`noCheck: true`) and `npm run typecheck` catches.
2026-08-19 18:49:11 -07:00
Reynaldi Chernando 53b792c52c connect to worker's kv with cli (#3613) 2026-08-20 08:36:46 +07:00
Daniel Salazar 2dd7073034 fix: vscode webdav (#3610) 2026-08-19 14:04:56 -07:00
Daniel Salazar a552ce0b87 fix: better metrics for cache hit rates (#3609)
* fix: better metrics for cache hit rates

* chore: type + docs
2026-08-19 11:38:14 -07:00
Neal Shah c04827fd3f add dynamic worker create event (#3608) 2026-08-19 14:20:15 -04:00
Daniel Salazar 9debdffc5d fix: email send frequency + ai ordering (#3604) 2026-08-18 18:13:55 -07:00
Nariman JelvehandJuan Castro 57b4cbab66 feat(dashboard): rewrite Files-tab sharing as a from-scratch modal (#3598)
* feat(dashboard): rewrite Files-tab sharing as a from-scratch modal

The Files tab's Share… action opened UIWindowShare, a desktop UIWindow,
which doesn't fit the dashboard. Replace it with UIShareModal, built on
the same overlay pattern as UIItemPropertiesModal: a centered card on
desktop, a bottom sheet on mobile, styled with the dashboard's design
tokens (dark mode included).

Wiring follows the onShowProperties precedent: generate_file_context_menu
gains an optional onShare hook that TabFiles supplies; every other caller
falls back to the desktop share window unchanged.

Feature parity with UIWindowShare — grant by email/username with an
access level, list who has access (owner, direct, and inherited-from-
ancestor grants), change a grant's mode, revoke — plus:

- revoke confirmation inlined into the row instead of a stacked UIAlert
- real form semantics: Enter submits, button disabled until input,
  spinner while in flight, API errors keep the typed name for correction
- dialog a11y: focus moves in on open and back on close, Tab is trapped,
  Escape closes (first Escape cancels an open revoke confirmation),
  status messages announce via aria-live
- backdrop close keyed on where the press started, so a text-selection
  drag out of the input can't dismiss the dialog
- name-derived avatar colors and a loading state for the first fetch

Also fixes a double-encoding bug (shared by UIWindowShare) where mode
labels ran through html_encode twice, rendering "Can edit &amp; share".

* fix(dashboard): keep keyboard focus inside the share modal across re-renders

Changing a grant's mode, revoking, or Escape-canceling the inline
confirmation re-renders the access list (or disables the focused
control), which dropped focus onto <body> - outside the dialog's Tab
trap and invisible to screen readers. Focus is now restored explicitly
after each of those actions: to the same holder's control when it
survives, else to the dialog itself (tabindex=-1), and the Tab trap
steps back into the cycle when focus rests on the dialog container.
A failed grant likewise returns focus to the recipient input.

* fix(dashboard): give share modal controls 44px touch targets

On touch devices the close (32x32) and revoke (30x30) buttons and the
26px-tall per-row mode select were below the 44px minimum. On coarse
pointers the icon buttons now hit-test at 44x44 through a centered
pseudo-element (visuals unchanged, and the row spacing absorbs the
overhang without overlapping neighbors), and the row select gets
taller padding. Fine-pointer rendering is untouched.

* fix(dashboard): clear the iOS home indicator in the share bottom sheet

The app opts into viewport-fit=cover, so on notched phones the
bottom-docked sheet extends to the physical screen edge and its last
row sat under the home indicator. The sheet body's bottom padding now
adds env(safe-area-inset-bottom), matching how the codebase pads other
bottom-docked surfaces.

* fix(share): say 'Updated access' when changing a grant's mode

Changing someone's access level reported 'Shared with X', the same
message as a fresh grant, which reads as if a new share happened.
Both share dialogs now confirm mode changes with a dedicated
'Updated access for X' message. The UIWindowShare call also drops the
html_encode() around the recipient - i18n() already encodes
interpolations, so the wrapper double-encoded.

* fix(dashboard): align the share dialog's accessible name with the properties modal

The dialog's aria-label read 'name - Share' with an em dash while the
sibling properties modal uses plain 'name Properties'; screen readers
announce the dash as noise. Use the same name-then-noun pattern.

* fix(dashboard): ellipsize the share input's placeholder when it overflows

At phone widths the 'Add people by email or username' placeholder was
clipped mid-letter; text-overflow: ellipsis truncates it cleanly.

* fix(dashboard): raise the share selects' chevron contrast in light mode

The hardcoded slate-400 stroke measured ~2.6:1 against light surfaces,
below the 3:1 minimum for non-text indicators. Light mode now uses
slate-500 (~4.7:1); dark mode keeps slate-400, which already clears
5:1 there.

* fix(dashboard): stop Files-tab shortcuts from firing behind the share modal

The document-level keydown.tabfiles handler kept running while the
share (or item-properties) modal was open. With focus on any of the
modal's buttons or selects, Enter and Space were preventDefault-ed
before they could activate the control, arrows could not drive the
mode selects, and letter typeahead was hijacked into row typesearch —
while Enter opened the selected row behind the overlay, Delete moved
it to Trash, and Cmd+A/C/X/V acted on the hidden list. Yield the
keyboard to the modal for as long as one is up; its own handlers
already cover Escape and Tab.

* fix(dashboard): keep the on-screen keyboard down when the share sheet opens

Autofocusing the recipient input popped the keyboard over the bottom
sheet the moment it opened, hiding the access list before the user had
chosen what to do — the same reason the revoke flow already focuses
the dialog instead of the input. On touch-primary devices give the
dialog container initial focus (which also anchors the Tab trap);
desktop keeps the input autofocus.

* fix(dashboard): give the share sheet's confirm and submit buttons 44px touch height

The inline revoke confirmation's Cancel/Remove pair rendered ~29px
tall on touch — small targets 6px apart where one of the two is
destructive — and the submit button was fixed at 38px. Grow both to
44px under pointer: coarse, matching the standard the modal's icon
buttons already meet. The touch block moves below the confirm-button
base rules it now overrides, since a media query adds no specificity
and source order decides.

* fix(dashboard): name the person in the share rows' accessible labels

Every grant row's mode select announced as bare 'Access level' and
every revoke button as 'Remove access', so a screen reader user
tabbing the list could not tell whose grant a control changes. Carry
the holder in the aria-label ('Access level for alice' / 'Remove
access for alice'); the visible UI and the revoke tooltip stay as
they were. i18n() encodes the interpolated string as a whole, quotes
included, so the labels stay attribute-safe for any holder name.

* fix(dashboard): keep the share status region in the accessibility tree

* fix(dashboard): give the share modal's add-row controls 44px touch height

* refactor(gui): extract the share dialogs' pure logic into tested modules

* feat(dashboard): adopt the shared mode helpers and handle pending invitations

---------

Co-authored-by: Juan Castro <[email protected]>
2026-08-18 17:34:40 -07:00
Juan Fernando Castro 5e1645d40e 🔧 PUT-1524: Sanity check file sharing api (#3603)
* fix: sharing answers "who can reach this" from the grants, not the index

Two ways a share listing could name access that was no longer there.

A recipient's listing paired the flat permission read with the list of
permissions it asked for by array index. That read drops misses and
dedupes its keys, so the two are not positional: one entry's live grant
vouched for another entry that had none, and listShared() kept
publishing a withdrawn item's name, size and signed thumbnail URL to
someone who could no longer open it. Read the permission off the value
instead.

getShares() had the same gap from the owner's side, with no liveness
check at all — a grant withdrawn through /auth/revoke-user-user or an
ACL mode change left the index row behind, and the owner was told
someone could reach a file they could not. Checked against the grants
now, one batched read per distinct holder. Pending invites are not
subject to it: they have no grant yet, which is the point of them.

Also covers the access-token actor, which reaches the same reach bound
as an app through a different arm of the ACL check. No behavior change
there — it was correct and untested.

* test: pin that a revoke crosses regions

A revoke reaches a peer region as a replicated SQL delete plus three
invalidation events, one per cache the region owns: the u2u row cache,
the flat view and the scan generation. The delete alone changes nothing
there, and each cache has a different consequence if its event is lost
— 20s for the generation, 5 minutes for the row cache, and forever for
the flat view, whose grant-path entries carry no expiry.

Nothing covered the whole path end to end. The store tests prove each
event is emitted and applied; this proves the result, which is that the
recipient stops being able to read. It asserts the read still succeeds
after the SQL delete alone, so the test also records why replication is
not sufficient on its own.

* test: pin that a leaked uuid buys no access

A masked share path hides which folder an item sits in; it was never the
thing deciding who may open it. Nothing checked that at the route level,
so the guarantee rested on unit tests of the resolver alone.

Reads one shared file through its masked path, then tries the sibling
four ways: the shared uuid with the sibling's name, the sibling's own
uuid, a `..` back out of the root, and the owner's real path.

Worth knowing about this one: it's mutation-checked. Removing the head !== root.name guard in sharePathMask.ts fails it with reachable: /testuser/55dd54c0…/share-http-1df8933e.txt. I verified that specifically because two tests I wrote earlier in this chunk passed with their guards broken — both were vacuous, and I deleted them rather than commit false assurance.

* docs: say what listShared's total actually counts

`total` counts the shares recorded for you; items are filtered after the
page is read, so a withdrawn grant leaves the count higher than anything
paging will yield. The page description already explained the short-page
behaviour, but the field read as an exact count and the example printed
it as one.

The test pins the gap it describes: two shares, one withdrawn outside
the index, one item listed and a total of two.

* fix: keep an undelivered broadcast event instead of dropping it

The outbound queue was cleared before the send, so a peer that timed out
took its events with it. Most were survivable — a lost cache invalidation
heals when the entry's TTL lapses. A revoke's flat-perm invalidation is
not: grant-path entries carry no expiry, so a peer went on serving a
withdrawn grant until something else wrote that key.

Failed sends now go back on the queue, which the existing flush timer
retries. Anything queued since wins over the retry, and the queue is
bounded at 10,000 with the oldest dropped first, so a peer that stays
down cannot grow it without limit. Each retry is signed at send time, so
it is not rejected against the replay window.

* fix: stop plus-addressing from deciding who a share reaches
2026-08-18 15:51:23 -07:00
b795b219a8 ✨ PUT-1497: Share file link sharing and notifications (#3595)
* refactor(share): move share notifications into their own service

* feat(share): invite an address with no account, and email it

* feat(share): surface pending invites in the SDK and share dialog

* fix(share): unreachable revoke confirmation, and double-encoded labels

* feat(share): budget share announcements, group them, and let people block senders

Sharing had one defence against noise: a 15-minute quiet window per (sender,
recipient) pair, which dropped the second share rather than folding it in.
Twenty senders each under their own window could still bury someone, and there
was no way to make one of them stop.

Announcements are now budgeted on two axes through the existing sliding-window
limiter: 1 per 15 minutes and 20 per day from one sender, and 10 per hour /
50 per day to one recipient from anyone. Over budget the share still succeeds
and the recipient's notification is still brought up to date — only the
interruption is dropped. Invite email to an address with no account is budgeted
the same way, keyed on a hash of the canonical address.

Notifications now fold across senders: a new share rewrites the notification
the recipient hasn't dismissed, so "alice and bob shared 5 items with you"
replaces a stack of five. The record is written even when suppressed, so the
count is right whenever they next look.

Blocking is a new `user_block` table with enforcement in ShareService: a blocked
sender's share is refused with `recipient_not_accepting_shares`, spends no
quota, and writes no row, and their unclaimed invite is dropped when the address
is confirmed. Existing access is untouched — that is what revoke is for.
Managed from a Blocked people card in the dashboard's Security tab.

Also publishes the sharing limits, including the ones already on this branch
that were never documented.

* fix(share): name the item in share email, instead of 'an item'

* fix(share): make the invite lifecycle canonical, authorized, and race-safe

* refactor(email): drop EmailClient.isConfigured; callers read config.email

* feat(share): batch share email into a per-recipient digest, durably

* docs(share): document the share error codes; steady the disk migration tests

* fix(share): log why a digest wasn't sent, and recover orphaned ones

* feat(share): email recipients about shares by default, with a way to decline

Share email was off unless a deployment opted in, which meant an account
holder was told about a share in the app only. It is now on unless
`share_email_notifications` is set to false.

The reason it defaulted off was that nobody could decline. So this also
honors `user.unsubscribed` — the account-wide opt-out the /unsubscribe page
already writes and app feedback already respects, which share email ignored —
and the digest carries that link. Sharing and the in-app notification are
unaffected by it; only the mail stops.

The link is composed in the template around an interpolated uuid rather than
passed pre-built: Handlebars escapes interpolated values, so a whole URL came
out as `user_uuid&#x3D;…`, which browsers decode but link scanners and older
mail clients need not.

* fix(share): count every shared file in the digest, not just the first

* feat(share): let a recipient refuse shares from everyone

Blocking answered "not from this person" but had no answer to "not from
anyone", so the only way to stop a stream of unwanted shares was to name
each sender after they had already reached you.

Stored as a key in the user row's existing `metadata` blob rather than a
column: the share path already holds the recipient's row by the time it
asks, so reading it costs nothing, and a one-bit preference doesn't earn
a migration per dialect. `updateMetadata` merges and refreshes the cached
row, so the switch bites on the very next share.

Refusing everyone reports the same code as refusing one person — which of
the two it is is the recipient's business, not the sender's. Enforced at
both moments the per-sender block is: when the share is issued, and when
a pending invite is claimed. The per-sender list is untouched while the
blanket switch is on, so turning it off restores what it hid.

`GET /share/blocks` now carries `all`; `POST`/`DELETE` take `{ all: true }`
beside the existing `{ username }`. Managed from the same Blocked people
card in the dashboard's Security tab.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* fix(share): keep the digest sweep off a window that still has a timer

The sweep treated an entry as orphaned the moment its window closed, which
is also the moment the node that armed it fires. Claiming an entry is only
exclusive among flushers that can see each other's deletes, so the two
could each claim a share of the same digest and both send. It now waits
out a grace period first, which costs a genuinely stranded digest that
much delay and nothing else.

Both digest listings were capped at 200 with no word when they hit it — a
truncated flush sends a digest that undercounts and reads as complete.
The cap is named and logged.

Also: `#emailHolder` still described share email as off by default, which
it stopped being; the config doc said the batch window defaults to 60s
when it is 90; and the two tests that need several calls inside one window
were racing a 50ms window across four sequential round trips, so they
failed under full-suite load rather than on the behaviour they cover.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* fix(share): stop double-encoding the recipient in two dialog messages

`i18n()` encodes what it returns, replacements included, so encoding the
recipient first showed the entities to anyone whose address or username
contains one. Same pattern already fixed two lines above.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* fix(notification): widen the mysql shown/acknowledged columns

Both arrived from the v1 schema as `tinyint(1)`, where they were flags.
The backend rework changed the writes to a unix second; sqlite (`INTEGER`)
and postgres (`bigint`) took it and mysql did not, so on mysql every
`markShown` and `markAcknowledged` has failed with
ER_WARN_DATA_OUT_OF_RANGE and left the column NULL. Dismissing a
notification never stuck — the unacknowledged count never moved and one
already delivered came back on every reconnect.

No backfill: every reader tests `IS NULL` / `IS NOT NULL` only, so a
legacy `1` keeps meaning "yes" once widened. Guarded on the current type,
because changing a column type copies the table and this directory
replays on every boot.

Not reachable from the test suite — it runs against sqlite and postgres,
both of which already have the right type. Verified by hand against mysql:
`/notif/mark-read` and `/notif/mark-ack` now persist, and a dismissed
share notification is no longer the one a later share folds into.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

---------

Co-authored-by: Daniel Salazar <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
2026-08-18 09:36:33 -07:00
Reynaldi Chernando 401b6066b3 Use gpt 5.6 luna as default for docs (#3602) 2026-08-18 09:36:23 -07:00
Daniel Salazar ec3c33fea6 fix: misc hardening fixes (#3600) 2026-08-18 00:06:58 -07:00
Anany Singh 95d797b047 feat(workers): expose app_uid in puter.workers.get and puter.workers.list (#3596) 2026-08-18 00:05:17 -07:00
Daniel Salazar bd06e88185 feat: bring back referrals PUT-1463 (#3599) 2026-08-17 23:49:09 -07:00
Nariman Jelveh cabedc30b0 Require login on app landing pages
Prevent first-time visitors from being treated as temporary users when they open an app landing page such as `/app/<name>` or `/desktop/app/<name>`. These routes now always go through the real login/signup flow.
2026-08-17 20:11:29 -07:00
Daniel Salazar f81b76c1d1 fix: webdav connectivity fixes for macos (#3597) 2026-08-17 19:02:55 -07:00
404oops 9604cc3f23 feat(ai-chat): overhaul provider model catalogs to match current APIs (#3586) 2026-08-17 14:06:44 -04:00
2c852bf6b3 ✨ PUT-1412 File sharing backend api (#3553)
* refactor(permissions): drop hardcoded group permission map for a flat default

* test(drivers): assert credential-gate intent instead of a 403 proxy

* fix(permissions): report whether a revoke removed anything and persist the linked grant row before the flat view

* feat(permissions): replicate permission invalidations across regions

* feat(share): extend the share table into an index of active shares

* feat(share): query and maintain active shares in ShareStore

* feat(users): add a batched lookup by email

* fix(cache): apply cache updates broadcast from peer regions

* fix(permissions): scope a revoke to the issuer that granted it

* feat(share): add ShareService with a per-day share limit

A share is two writes that belong together: the permission grant, which
authorizes access, and a share row, which makes it listable and ties it to an
fsentry so it dies with the file. Nothing else grants fs:* to a user.

Authorization reuses canManagePermission — an owner satisfies it through the
is-owner implicator, a delegate through an explicit manage:fs:<uid> grant. An
owner may clear any issuer's share of their node; anyone else only the ones
they issued, or their own access. Self-revoke skips the manage gate but still
requires `see`, so it cannot be used to probe for files.

The per-day limit counts shares created rather than live rows, so revoking and
re-sharing cannot recycle a slot, and changing an existing share's mode is not
new reach and does not spend budget. Tunable via share_daily_limit.

* feat(share): expose sharing over HTTP

POST /share, POST /share/revoke, GET /share/shared-with-me, GET /share/shares.
The controller was registered but entirely commented out.

Recipients × items fan out concurrently — every pair is a distinct
(holder, entry) key, so none of them contend — bounded by
runWithConcurrencyLimitSettled, which returns results index-aligned with the
input for the per-pair outcome list. Responses carry usernames only, never
internal ids, and the 404-not-403 rule is preserved so a failed call cannot
confirm a file the caller could not otherwise see. Notifications are fired off
the response path; a share must not fail over its own notification.

Per-request caps on recipients and items bound one call's fan-out; the daily
limit bounds the total.

* feat(share): keep recipients consistent when a shared item changes

* fix(fs): stop listing issuer homes at the filesystem root

* fix(acl): serialize concurrent mode changes on one node and pin app containment on shared paths

* fix(fs): expire signed URLs over entries the signer doesn't own

signFile defaults to a ~317k-year TTL and verifySignature checks only uid,
expires and signature — never the ACL. A recipient who ever signed a shared
file therefore held a permanent, revocation-proof URL to its bytes: revoking
the share did nothing to it.

signEntry now takes the acting user and drops to NON_OWNER_SIGNATURE_TTL_SECONDS
(1 hour) when the signer is not the entry's owner. Owners keep the permanent
default, so no existing client changes behavior.

The signature-authenticated directory listing bounds its children
unconditionally: that route has no session actor, and a signature proves
possession rather than ownership, so a recipient holding a short-lived
directory signature could otherwise mint permanent URLs for every child.

A bounded window is not revocation — the durable fix is a per-entry signature
epoch folded into the HMAC and bumped on any permission change.

* refactor(permissions): drop the unused permission-issuer lookup

listUserPermissionIssuers and its store method listUserPermissionIssuerIds
existed to synthesize the filesystem root from the home directories of everyone
who had granted the caller a permission. That listing is gone — it advertised
folders readdir then refused to open — and the share index answers "who shared
with me" directly, so nothing wants them back.

One removed test only asserted that the call returned an array; the other
covered readLinkedUserUserPerms round-tripping and is kept, rewritten without
the issuer lookup.

* fix(share): return the created share, not just an acknowledgement

* feat(puter.js): add file sharing to puter.fs

share(), unshare(), listShared() and getShares() on puter.fs, following the
existing FS operation shape: positional and options-object forms through
defineOperation, JSDoc overloads as the published signature, relative paths
resolved against the app's root directory.

A bare recipient string is read as an email when it contains @ and as a
username otherwise. Sharing an item with someone who already has it replaces
their access rather than stacking a second grant, so raising read to write is
one more call.

Adds a sharing suite to the API runner, which passes unchanged on node,
browser and workerd. Documents all four methods with runnable examples, and
corrects the FS overview callout that told readers one user cannot read
another's files — true before this, not after.

* feat(gui): add a Shared folder for items others shared with you

A sidebar entry listing everything other users have shared with you, backed by
puter.fs.listShared().

The path is the sentinel `puter://shared` rather than /<user>/Shared: this is a
query, not a directory, and a path-shaped value could collide with a folder
someone actually creates. refresh_item_container and update_window_path both
branch on it to skip the stat there is no fsentry for, and the listing swaps
readdir for listShared.

Entries render at their real paths under their owners' directories — the item
container already preferred an explicit fsentry.path over joining onto the
container, so nothing else had to change. Each carries who shared it and at
what level, which the context menu reads next.

* feat(gui): share items from the context menu

A sharing dialog shaped like its neighbours — options object, HTML-string
template, jQuery wiring, delegating to UIWindow() — with a recipient field, a
read/edit/share dropdown, and the current access list with revoke buttons.

Reached from a new "Share…" context menu entry, which is hidden on items shared
*with* you: re-sharing needs manage, so the dialog would only surface an error.

Those items get "Remove from Shared" in place of Delete. Delete moves an item
to *your* trash, which for someone else's file means moving their data out of
their tree — FSService refuses it, and the user saw a bare 403. Removing your
own access is what the action was reaching for, so that is what it now does.

* fix(share): withdraw what a removed recipient re-shared

* feat(share): report access inherited from a parent folder

* fix(gui): load the puter.js bundle the server configured

* refactor(gui): extract the action icon set into a helper

* feat(gui): surface Shared in the file browser

* feat(gui): manage access from the share dialog

* test(share): cover access inherited from a parent folder

* fix(share): keep downstream access from surviving a delegate who leaves

* fix(gui): name the real owner in the share dialog

* fix(gui): page through every shared item instead of the first 50

* feat(share): return item metadata with a share

* fix(share): invalidate a holder's cache when the entry is deleted

* fix(gui): treat items inside a shared folder as someone else's

* feat(permissions): let manage inherit down the filesystem tree

Access already reached descendants through the ancestor chain while authority did not, so someone trusted to manage a shared folder could re-share the folder but nothing inside it, and could not see who had access to a file within it.

A manage-inherits-from-ancestor implicator resolves it in the permission layer, beside is-owner, so every caller agrees rather than just ShareService. It consults only the immediate parent — resolving that re-enters one level up, making a chain of depth d cost d checks rather than d².

That makes two cascade gaps reachable, both fixed here. A revoke now walks the subtree, since a grant on a descendant can rest on authority held at the folder. And it stops at a delegate whose authority survives another issuer, because what they granted was never theirs to lose.

Also pins that manage is not transitive: granting it needs manage:manage:fs:<uid>, which only the owner holds, so delegation is one level deep by construction.

* fix(gui): offer sharing inside a folder you manage

The menus encoded "manage does not inherit" and would now hide an action that works. The Shared listing records each root's mode; the menus resolve a child's by longest matching ancestor, loading on demand so a deep link or restored window works too.

* fix(share): make the daily share limit hold under concurrency

* test(share): cover concurrency, measure cost, and name cases for what they verify

* fix(gui): import the ownership helpers the item menu calls

The single-item context menu handler calls is_owned_by_me and
shared_mode_for, but the imports were only ever added to
generate_file_context_menu.js — so every right-click on an item threw a
ReferenceError before the menu could build, and the non-owner Delete
gating never ran.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(share): authorize before resolving the recipient

share() looked up the recipient in parallel with the entry, before the
manage check — and the two failures carried different error codes. Any
verified user with a real entry uid could probe arbitrary emails and
usernames for account existence, at no quota cost. Resolve the entry,
authorize, and only then resolve the recipient: an unauthorized caller
now sees the identical safe 404 whether or not the recipient exists.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(permissions): broadcast permission row-cache invalidations to peer regions

Every publishCacheKeys call for the u2u, u2a, and access-token row
caches omitted broadcast, so a revoke only cleared the mutating
region's Redis. A peer region applied the replicated generation bump,
re-scanned, read the deleted row from its own still-warm 5-minute row
cache, and re-warmed the flat view from it — revoked access outlived
the revoke by the row-cache TTL instead of the intended 60-second
bound. CacheReplicationService already consumes these events; the
emits were just never sent.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(fs): refuse to rename an entry owned by another user

remove and move both refuse to act on an entry the caller does not
own, even when the ACL allows the write — rename had no such guard, so
a write-mode share recipient could rename the owner's file, or the
shared folder itself, rewriting the owner's whole subtree's paths.
rename now takes the acting user and applies the same policy.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(permissions): decide a flat delete from the primary, not a lagging replica

revokeUserUserPermission deletes the SQL grant, then only drops the
flat KV entry once no issuer still grants the permission. That
remaining-check read through the row cache the delete had just
invalidated, straight to a replica — under any lag the deleted row
reappeared, the flat delete was skipped, and the stale rows were
re-cached for another five minutes. Grant-path flat entries carry no
TTL, so the holder kept working access with zero SQL rows behind it,
invisible to every listing. The check now reads the primary and
re-warms the cache with what it actually saw.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(permissions): keep a failed remote flat-invalidation from crashing the process

The outer.permission.flatInvalidated applier was fire-and-forget with
no catch, and it awaits a KV delete — one transient KV error while
applying a peer region's revoke became an unhandled rejection, which
is process-fatal under default Node. Its sibling appliers were already
guarded; this one now logs and moves on, leaving the entry to the next
invalidation or its TTL, same as a lost event.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(share): revoke every requested item and recipient, not just the first

revokeShare destructured only the first recipient and first item while
the parsers accept arrays up to the request caps — unshare({items:
[a, b, c]}) returned success having revoked only a, leaving access the
caller believes is gone. Revoke now fans out over every (recipient,
item) pair exactly like POST /share, reports per-pair outcomes, and
sums the revoked count; the response stays backward compatible.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(share): only a confirmed email designates a recipient

Recipient resolution by email accepted unconfirmed accounts, so
pre-registering someone else's address (unconfirmed) was enough to
receive shares meant for them once no confirmed account held it.
An email now only resolves to an account that has confirmed it;
username shares are unaffected.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(share): accept tilde-rooted paths like the FS routes do

The SDK resolves relative paths to ~/..., but the share routes never
expanded the tilde — a ~-prefixed string was read as a uid and every
relative-path call 404'd. Item parsing now treats ~ as path-shaped and
expands it to the actor's home with the same helper the legacy FS
routes use, on share, revoke, and the shares listing.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(share): walk a directory revoke by parent linkage, not path prefix

listByFsentrySubtree matched descendants with fsentry_id = ? OR path
LIKE ?, which has two problems: fsentries.path is lazily backfilled
and NULL on old rows, so those descendants' shares silently survived a
directory revoke, and the OR'd predicates forced a scan of every
active share. A recursive CTE over parent_id — the same shape the
lineage resolver already uses — covers every descendant and runs on
idx_parentId_name.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(gui): give each item its own share dialog

single_instance keyed the dialog on the app id alone, so opening
Share… on a second file focused the first file's dialog — typing a
recipient there granted access to the wrong file, with only the title
hinting at it. The dialog is now instanced per path: same item
refocuses, different item opens fresh. Also stops pre-encoding the
title, which UIWindow encodes again.

Co-Authored-By: Claude Fable 5 <[email protected]>

* feat(permissions): let manage answer a write check

A manage grant let its holder re-share a folder but not work in it: the
ACL mode family stops at write, and the fs exploder had no rule for the
narrowest mode, so `manage:fs:<uid>` never satisfied `fs:<uid>:write`.

Fold manage into the candidate list for every non-manage mode, in both
the access-token branch and the scan branch, and give `write` an (empty)
exploder rule so the manage arm is emitted for it too.

* fix(fs): authorize restructuring by write on the parent

* fix(fs): let a share recipient work inside a shared folder

rename, remove and move refused outright when the entry belonged to
someone else, so a recipient with write could neither delete nor rename
anything inside a folder shared with them. The GUI compounded it by
hiding Delete for any item it did not own.

Authorize the three by ACL write on the entry's parent. For an owner
that is the same answer; for a recipient it grants the inside of a
shared folder and withholds the folder itself, whose parent is the
owner's private tree.

Deleting sends the item to its owner's trash rather than the deleter's,
so it leaves the recipient's view without leaving the owner's account
and without changing hands. A move may not otherwise carry someone
else's entry out of their tree.

* fix(fs): give a new entry to the owner of the folder it lands in

A file a share recipient added to a shared folder was recorded as
theirs while living in the owner's tree, so a subtree could hold rows
belonging to several people — and the storage it consumed was checked
against the writer while being counted against the owner.

Take the owner from the parent row at every insert, charge the
allowance to that owner, and hand a moved entry over to the tree it
moves into. An entry now always belongs to whoever owns the directory
holding it.

* feat(fs): address shared entries as ~/share/<uid>

A recipient could read the owner's whole path off any shared entry —
where they keep the file and what sits beside it, neither of which the
share is about.

Give shares their own namespace. `~/share/<entry-uid>/rel/path` resolves
to the real path on the way in, and outgoing paths are rewritten to it
on the way out. Entries the actor owns pass through untouched, so no
existing client contract moves.

* revert(fs): mask only the directory bar, not the addressing

fe535d598 made `~/share/<uid>` the actual address for every shared
entry. That reached far past the intent: item names became uuids, the
Shared views rendered uuids instead of filenames, and navigation
addressed entries through a namespace nothing else understood.

Put real paths back everywhere — responses, the share listing, and
request handling — and do the masking where it was wanted, in the
window's directory bar. A recipient sees `Shared › Contents › sub`
while every crumb keeps the real path it navigates to.

The share listing now carries the entry's name, content type, owner and
a signed thumbnail. A share row has no fsentry behind it for a client
to stat, and the stored thumbnail is an `s3://bucket/key` URI that no
client can render and none should see.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* fix: path obfuscation, webdav + small ui stuff

* test(share): assert the masked share path by its exact shape

The substring check tripped on the scratch files' own names, which start
with `sharing-`; the exact-equality assertion on `/<owner>/<uid>/<name>`
already proves nothing above the share leaks.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(share): rename policy, shared-view guards, webdav share parent, quota/bucket invariants

- Rename: a directly-shared FILE renames with write on it; a shared
  folder root stays fixed (its name is the owner's tree structure).
  GUI can_rename mirrors the backend, guards all editor entry points.
- Up from a share root goes to the Shared view on both surfaces; the
  Shared view gets a single crumb, a disabled Up button, and refuses
  drops, New/Paste, uploads and ctrl+V everywhere (it is a query, not
  a directory).
- WebDAV: PROPFIND on /owner/uuid answers as a virtual collection
  holding the share root (ACL-gated; 404 for strangers).
- Storage allowance override no longer crosses user boundaries: a
  recipient's plan cannot raise the owner's cap.
- Overwrites stay in the bucket the entry already lives in instead of
  repointing to the handling server's bucket and stranding the old
  object.
- manage mode documented as implying write (matches enforcement);
  share dialog label now "Can edit & share". Documented that fs socket
  events are owner-only.
- Sidebar: saved orders gain the Shared entry once the user has shares.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(share): close review findings, and retire /auth/grant-user-user

Security
- Recipient writes no longer echo the owner's real path back. The pending-write
  event and the upload-progress meta both go to the *acting* user, so a write
  into a shared folder handed out the layout above the share root.
- `maskerFor` failed open: the first caller fixed the actor, and one that ran
  before the request knew who was acting pinned `undefined` — after which every
  path published unmasked. It now adopts the first real actor and rebuilds for
  a different one.
- `/auth/grant-user-user` returns 501. It wrote user-to-user grants straight to
  the permission tables with no share row, so nothing could list or cascade a
  revoke over them — and the new `manage` mode meant a delegate could reach it
  for the owner's files. Filesystem access goes through `/share`; nothing else
  is meant to pass between two users. Undocumented (its docs page was never
  written) and no callers. Revoking is untouched, so grants made before this
  can still be withdrawn.
- `#assertCanManage` asks about the permission actually being granted rather
  than a fixed `read`, so sharing at `manage` is refused at the share layer
  instead of by `grantUserUserPermission` two levels down.
- `listSharedWithMe` checks the grants, not just the index: withdrawing access
  any other way left the row publishing name, size and a signed thumbnail URL.
- Moving an item into another tree now retires its shares. Grants are keyed on
  uuid, so they followed it and left the new owner with recipients they never
  agreed to.

Correctness
- `acl.check` gets the real path again, not the masked one. ACL matches on the
  path string, and a mask hides the `AppData/<appUid>` shape it needs.
- "Leave this share" works for a grant that predates the index; the fallback
  scoped the delete to the caller as issuer, which can never match.
- `move`/`copy` reject a name with a slash or a `.`/`..` segment, as `rename`
  already did. It matters more here: a move into the owner's Trash skips the
  destination write check.
- Descendant walks scope by path, not by owner, so rows predating the
  one-owner-per-subtree invariant aren't orphaned by their parent's deletion.

Performance
- Index `user_to_user_permissions(permission)`. Retiring a node's grants is a
  prefix match, but the primary key is (issuer, holder, permission), so it was
  a full scan.
- Retirement is coalesced and chunked. `remove()` emits one event per
  descendant, so deleting a directory fired one unindexed lookup per file, all
  at once and unawaited.
- `listReaching` is a plain `fsentry_id IN (...)` on `idx_share_fsentry`. It
  runs behind every file write, and the join-plus-OR it replaced was
  unindexable.

---------

Co-authored-by: Claude Fable 5 <[email protected]>
Co-authored-by: Daniel Salazar <[email protected]>
2026-08-17 09:08:23 -04:00
Ayush tiwari fc861da89f feat: add Gemini 3.7 Flash (#3587) 2026-08-16 23:51:37 -07:00
Parman Mohammadalizadeh facb844747 fix: derive express subdomain offset from the configured root domain (#3575)
Express derives `req.subdomains` by dropping a fixed number of labels
from the right of the hostname, defaulting to 2. A deployment whose
`domain` has more than two labels therefore reads its own root domain as
an active subdomain, so the user-site redirect sends the root origin to
the static hosting domain, which sends it back. Self-hosting docs
recommend exactly that shape (`puter.example.com`).

Set the offset from the label count of `config.domain`. Two-label
domains keep the express default, so existing deployments are unchanged.

Fixes #3561
2026-08-16 23:48:20 -07:00
Yashwarhade8010 3b791bb334 docs: update web search examples to gpt-5.6-luna (#3589) 2026-08-16 23:45:02 -07:00
Daniel Salazar a64a444a17 feat: batch delete for flush + metering fixes (#3593) 2026-08-16 15:26:33 -07:00
Daniel Salazar d3c9828207 fix: metering gui maths (#3592) 2026-08-16 14:57:29 -07:00
Daniel Salazar 87180b1081 fix: metering (#3591) 2026-08-16 14:19:05 -07:00
Daniel Salazar d22b656df8 fix: metering issues with addons (#3588) 2026-08-15 22:06:51 -07:00
Nariman Jelveh 19cd5f4476 feat: add BytePlus ModelArk providers (chat, image, video) (#3498)
* feat: add BytePlus ModelArk chat provider

Adds BytePlus ModelArk as a provider for the puter-chat-completion
driver, following the MiniMax/ZAI providers as reference per
doc/contributing-apis.md.

- OpenAI-compatible endpoint at ark.ap-southeast.bytepluses.com/api/v3
  (apiBaseUrl config selects the region)
- Static catalog of 16 chat models (Seed 2.x/1.x incl. vision, GLM,
  DeepSeek, GPT-OSS) with limits and per-token pricing from the
  official docs
- Passes Ark's thinking/response_format/stop params through custom;
  normalizes reasoning_content to reasoning
- Bare deepseek-v4-* names stay with the first-party DeepSeek
  provider; BytePlus only claims prefixed aliases
- Offline unit tests (mocked SDK against a real test server) plus an
  env-gated integration test

* feat: add BytePlus image and video providers

Extends the BytePlus ModelArk integration to the puter-image-generation
and puter-video-generation drivers, reusing the same services.byteplus
API key and regional apiBaseUrl as the chat provider.

Image (Seedream/SeedEdit via OpenAI-compatible /images/generations):
- dola-seedream-5-0-pro (pixel-tier pricing + billed input images from
  the 2nd on), seedream-5-0-lite, 4-5, 4-0, and seededit-3-0-i2i
- quality tiers 1K/1.5K/2K; aspect ratios resolve to Ark's documented
  pixel sizes; explicit WxH passes through with Ark's bounds enforced

Video (Seedance via Ark's async /contents/generations/tasks + polling):
- Seedance 2.0 / 2.0 Fast / 2.0 Mini / 1.5 Pro / 1.0 Pro / 1.0 Pro Fast
  (2.5 is priced but its API isn't live yet, so it's excluded)
- per-video-token billing from usage.completion_tokens, with per-second
  estimates feeding the credit cap; audio vs silent rates for 1.5 Pro
- first/last frame and reference-image inputs; generate_audio param
  added to IGenerateVideoParams

Pricing and capabilities hardcoded from the official docs (ModelArk
pages 1544106, 1330310, 1520757, 1521309, 1541523). Offline unit tests
mock the SDK / global fetch; integration tests are env-gated on
PUTER_TEST_AI_BYTEPLUS_API_KEY.

* fix: correct BytePlus catalogs and validation against the live API

Verified the three BytePlus providers against ModelArk with a real key;
these are the mismatches that surfaced.

- Drop seededit-3-0-i2i-250628. Ark reports it as Shutdown and every
  request 404s. Its now-unreachable image-to-image branches in the
  provider go with it.
- seedream-4-5 and the 5.0 series enforce a 3,686,400 pixel minimum, so
  they only accept the 2K tier. Mark them 2k-only and snap an
  unsupported tier up to the nearest allowed one, which also keeps the
  aspect-ratio table from mapping to a sub-minimum size.
- glm-4-7 has a 204,800 token context, not 256K.
- Guard the actor in the image provider like the video provider does.
- Round a sub-minimum video duration up to the shortest supported clip
  instead of reporting it as insufficient funds.
- Gate video resolution on the model's own dimensions; the dims table is
  shared across a family and accepts more than any one model does.

* Tighten BytePlus AI provider handling

Extract shared reasoning-content normalization for OpenAI-style chat providers, and harden BytePlus image/video behavior. This updates image tier and size validation, normalizes aspect ratios and input image refs, prevents mismatched BytePlus key/base URL fallback config, makes video resolution matching case-insensitive, and rejects excess reference images instead of silently truncating them. Tests were expanded to cover the new BytePlus request and validation paths.
2026-08-15 21:08:17 -07:00
404oops fc56075115 Merge pull request #3584 from rupesh0001-tech/rupesh
feat(ai-chat): add grok-4.6 model to xai provider
2026-08-15 21:22:01 +02:00
404oops e1998e39ab Increase max_tokens limit to 500,000 2026-08-15 21:20:45 +02:00
Daniel Salazar fb7968a1c7 fix: metering hardening; handle burst of unfinished ai requests (#3585) 2026-08-15 10:52:29 -07:00
rupesh0001-tech c88eeb3868 feat(ai-chat): add grok-4.6 model to xai provider
- Add grok-4.6 model definition with pricing and context specs
- Add aliases x-ai/grok-4.6 and grok-4.6-latest
- Add unit tests for alias resolution and metering calculation

Closes #3562
2026-08-15 20:32:22 +05:30
Reynaldi ChernandoandCopilot Autofix powered by AI 9e25ce1401 [PUT-1478] Add KV commands to CLI (#3579)
* Add KV commands to CLI

* docs

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

---------

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
2026-08-15 11:43:30 +07:00
Daniel Salazar f15d835eeb fix: restrict openai and anthropic compatible endpoints to be subscription (#3583) 2026-08-14 18:55:15 -07:00
Daniel SalazarandClaude Fable 5 27c4506e02 feat: usage rendered as credits on a 50c base, free allowance doubled (#3569)
* feat: usage rendered as credits on a 50c base, free allowance doubled

Usage is now displayed in credits: a configurable creditsPerDollar rate
(default 2,000, making the free tier's 50c allowance an even
1,000-credit base) ships with the metering usage response, and the
dashboard's usage cards and per-API table show raw credit numbers
through one shared formatter. The dashboard budget math is also fixed —
capacity is spend plus server-netted remainder, so held purchased
credit can never render as negative usage. The registered-user free
allowance doubles to 50c, tier display names move to Basic/Plus/Pro,
and dead referral-promise strings leave the English dictionary.

Co-Authored-By: Claude Fable 5 <[email protected]>

* refactor: the credits display rate exists only in deployment config

No default in code and no rate on any wire the plan surfaces read: a
deployment that doesn't configure creditsPerDollar renders usage in
dollars, exactly as before credits existed. The usage endpoint sends
the rate only when configured, and nothing in code or comments states
what any deployment's rate is.

Co-Authored-By: Claude Fable 5 <[email protected]>

* refactor: metering reports pre-scaled credits, never raw amounts

The usage, per-app usage, and cost-catalogue endpoints multiply every
monetary field by the config multiplier (renamed creditMultiplier)
before responding, and flag the unit; the multiplier itself never
ships. Counts, units, and byte figures pass through untouched. With no
multiplier configured the endpoints report raw amounts and clients
render dollars, as before credits existed.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix: absent addon fields stay absent when scaling to credits

Co-Authored-By: Claude Fable 5 <[email protected]>

---------

Co-authored-by: Claude Fable 5 <[email protected]>
2026-08-14 16:29:01 -07:00
Neal Shah 6a7c09f54e blacklist dynamic workers (#3582) 2026-08-14 18:35:34 -04:00
Reynaldi Chernando 6254714484 deprecate gemini 2 flash and lite (#3581)
* deprecate gemini 2 flash and lite

* fix test
2026-08-14 17:27:48 -04:00
Neal Shah efa2cb4441 add more extensibility to workers driver (#3565) 2026-08-14 17:24:36 -04:00
Nariman Jelveh 4bc969b99e Update RecommendedAppsService.ts 2026-08-14 14:05:44 -07:00
Reynaldi Chernando 65f6d38432 update gpt 5.6 luna and terra pricing (#3580) 2026-08-14 16:38:13 -04:00
Daniel SalazarandClaude Fable 5 d641f6f7a1 tests: count only this test's dynamo reads in the block-window assertion (#3576)
The spy sits on the test server's shared dynamo client, so a raw call
count also picks up background work and the async tail of earlier tests
in the file — which made the block-window test flakily report a third
read. Each test runs in its own random namespace, so filtering the
spy's calls to that namespace removes the cross-talk without loosening
the assertion.

Co-authored-by: Claude Fable 5 <[email protected]>
2026-08-14 11:05:25 -07:00
Daniel SalazarandClaude Fable 5 00828b9998 fix: concurrency limiter heals legacy INCR keys instead of failing open (#3574)
The zset rework of the redis concurrent backend (#3529) kept the same
key names the old INCR counter used. Against a leftover string key every
zset command in the acquire MULTI fails WRONGTYPE while the trailing
EXPIRE still succeeds — so steady traffic refreshes the stale key
forever, and the failed zcard result turned into NaN, which admitted
every caller unbounded. Release then errored WRONGTYPE on each request
(the '[concurrent] release failed' log storm).

Surface per-command MULTI errors instead of coercing them to NaN, and on
WRONGTYPE delete the legacy key and re-acquire against a clean zset.

Co-authored-by: Claude Fable 5 <[email protected]>
2026-08-14 01:40:05 -07:00
Daniel SalazarandClaude Fable 5 126ec09556 docs: publish rate limits and quotas, and make limit changes move the docs (#3568)
Developer-facing reference for every rate limit, concurrency cap,
quota, and error shape — an advanced page, since puter.js already turns
the common failures into prompts. AGENTS.md and CONTRIBUTING.md now
carry the rule that a PR moving any of these numbers updates the page
in the same PR: an undisclosed limit is one developers discover as a
service failure.

Co-authored-by: Claude Fable 5 <[email protected]>
2026-08-14 00:41:01 -07:00
jelveh 25a5799088 Update RecommendedAppsService.ts 2026-08-14 00:38:05 -07:00
Daniel SalazarandClaude Fable 5 ea2f9967a6 fix: over-quota batch uploads surface the storage prompt instead of failing quietly (#3566)
* fix: over-quota batch uploads surface the storage prompt instead of failing quietly

Partial batch failures now carry each item's code/status, and when every
failed item failed the same way the shared code/status is hoisted onto
the rejection itself — so an upload that exceeds the storage quota
rejects with storage_limit_reached/413 and the SDK's upload handler
shows the free-up-space prompt. A partial failure is also no longer
misread as the signed-batch endpoint being unavailable.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix: out-of-storage copies prompt to upgrade the same way uploads do

Any fs operation the server refuses with 413 storage_limit_reached now
surfaces the upgrade prompt — the check that lived inline in upload's
error handler moves to a shared helper wired into the operation
scaffold's reject path, so copy/move/mkdir/rename get it too. The
desktop's copy/paste suppresses its generic alert for that code, since
the SDK dialog already explains the refusal and carries the fix.

Co-Authored-By: Claude Fable 5 <[email protected]>

---------

Co-authored-by: Claude Fable 5 <[email protected]>
2026-08-14 00:26:07 -07:00
Daniel Salazar ba9b885a5a Revert "feat: dashboard app search continues into the App Center (#3567)" (#3572)
This reverts commit 6cc241ac31.
2026-08-13 21:00:02 -07:00
Daniel SalazarandClaude Fable 5 6cc241ac31 feat: dashboard app search continues into the App Center (#3567)
A search that matches none of the user's own apps now puts the same
query to the App Center catalogue and renders the hits under the empty
state — debounced, filtered to apps the user doesn't already have, and
launchable in place.

Co-authored-by: Claude Fable 5 <[email protected]>
2026-08-13 20:48:03 -07:00
Nariman Jelveh 7f8e24f06d fix: keep "Add an app" in the grid while searching
Typing in the Apps tab's search dropped the tail tile — deliberately, on the
grounds that a query asks which apps you HAVE and a tile matching nothing has
no business in the answer. That reads the search too narrowly: looking for an
app you don't have is the same gesture as looking for one you do, and the
moment the search comes up short is exactly when the way to GET an app is
worth having on screen.

So the tile is pushed unconditionally now, and a query that matches nothing no
longer replaces the grid with a bare "No apps match your search" — the line
moves above a grid holding just that tile. Keeping it a real grid, rather than
parking a tile inside the empty state, is also what keeps it reachable: arrows
and Enter only walk tiles inside .myapps-page, and the tail tile carries
tabindex -1 for the pager's roving focus.

The no-apps-at-all state is untouched: with nothing installed and nothing
typed, the empty state still stands in for the whole grid.
2026-08-13 18:53:59 -07:00
Nariman Jelveh 9b6627b1ea fix: an app's background helper must stay private to it
`launchApp(name, args, { background: true })` hid the child's window and
nothing else, so the instance was hidden but not private: it kept its
taskbar item, lit the running dot on its dashboard tile, and — because
the dashboard enforces one instance per app — WAS what clicking that tile
opened. Minimize the parent, click the child's tile, and Puter handed the
user the window the parent was mid-conversation with; the app, which only
ever meant that instance for its caller, said so on screen. Showing it
also dropped the never-shown marker, so from then on the helper outlived
its launcher and owned the tile.

Hidden now means unreachable. `user_facing_windows` is the one rule: of
an app's windows, the ones that are the user's. Everything the user
drives filters through it — the tile's reopen and its running dot and
Quit item, the taskbar item's click, window list and show/hide/close-all,
the file row's switcher and open-file dot, and the dashboard's
Back/Forward — so a helper is never focused, restored, listed, or counted
as running. With none of the user's own left, the tile simply launches a
fresh instance and the helper keeps serving whoever launched it. Paths
that act on the APP rather than on a window it happens to have open —
uninstall, and a launcher's close taking its helpers down with it —
deliberately keep the unfiltered list.

The taskbar item is now what a hidden launch EARNS by becoming visible,
rather than something it advertises from the start; makeWindowVisible
adds it (and fires the dots event) before showing, since focusWindow
marks an item active and needs it to exist. Windows stamp data-in_taskbar
when they take their place in data-open-windows, so the close path only
decrements a count it actually joined — otherwise a helper closing beside
the user's own window would zero the count and strip the item off a
window still open. For the same reason a helper's close no longer pops
the dashboard URL: it never claimed an entry (the push only happens for a
window created visible), and consuming one would minimize the user's
window of the same app.

is_unseen_background_window gains the other half of its own definition:
launched hidden AND launched BY another app. An app that is always
windowless (`background` on the app record) starts hidden too, but when
the user opens it there is nobody it is serving — it is the instance
their tile has to find, or every click would start another one they
cannot see.
2026-08-13 18:38:13 -07:00
Nariman Jelveh 7576e0b03a fix: route dashboard file menus by input modality, not touch capability
A mouse right-click on a file row in the dashboard Files tab opened the
mobile context-menu sheet (centered over the row, nowhere near the
cursor) on any touch-capable laptop or desktop, because phone/tablet
detection used navigator.maxTouchPoints > 0. The same check applied the
.touch-device class, degrading mouse selection and drag on those
machines.

Route by how the menu was invoked (the row's tracked pointer type) plus
isTouchPrimaryDevice() — coarse pointer and no hover — which is false on
touch-capable laptops but true on iPads whose UA claims macOS. Also let
sidebar-folder and background menus accept taphold on such iPads, where
they were previously unreachable (taphold dismissed, and iOS never fires
native contextmenu).
2026-08-13 17:46:58 -07:00
Nariman Jelveh d980a9f590 fix: don't invert full-color icons in the dashboard context menu 2026-08-13 17:33:14 -07:00
Nariman Jelveh d18ea1adb5 fix: a background app must not outlive the app that launched it
An app launched with `background: true` gets a real window from the moment it
starts, just hidden. Nothing ever took it down: when the app that launched it
closed, the child kept running with no way to reach it and no reason to be
there. On the dashboard the only sign was a running dot on a tile the user had
never lit up, and clicking that tile did nothing at all.

Three things were wrong, and all three had to go:

A window nobody has seen now dies with its launcher. UIWindow's close path
closes hidden children the closing app launched, keyed on a marker stamped at
creation. makeWindowVisible drops that marker the first time the window becomes
visible — showing itself with `puter.ui.showWindow()`, or the user showing it —
because from then on the window is the user's, and keeps running.

The dashboard tile is a real handle again. focusExistingAppWindow only routed
MINIMIZED windows through showWindow(); a hidden one fell through to
focusWindow(), which leaves it invisible while handing it the keyboard. With no
taskbar in dashboard mode the tile is the only handle on a background app, so
that click had nowhere else to go. It now asks whether the window is on screen
at all. The Files tab's row-click had the same one-line defect.

And a background instance can't take a tile from the user's own session: an
on-screen window wins, then a window the user has seen, then a hidden one.

Removing the child then hit a crash of its own: ExecService's `remove` handler
dereferenced the launcher's iframe to say goodbye, and the launcher was already
gone. Throwing there aborts jQuery's remove() itself, so the window stayed in
the DOM — running dot and all. That one also hit anyone closing a background app
from the taskbar after its launcher had closed.

The predicates behind all of this live in one helper, window_visibility.js, with
showWindow() reading the same `hidden, not minimized` rule it spelled out inline
before.
2026-08-13 14:36:08 -07:00
Nariman Jelveh 08f075d774 Update RecommendedAppsService.ts 2026-08-13 11:45:28 -07:00
Nariman Jelveh 664de0016b fix: a hidden window must not hold the keyboard
Launching an app to serve another one took focus away from the app the user
was working in, and they had to click it to get it back. Typing a guest name
into Calendar's event editor — which launches Contacts to read the address
book — dropped the user out of the field mid-word.

Four things conspired, all of them variations on "a window nobody can see is
treated as the window in front":

`window-active` was hardcoded into every window's markup, including windows
created hidden. focusWindow() — the thing that normally grants that class,
and strips it from everyone else — is already skipped for a hidden window, so
two windows ended up claiming it. It now follows the same rule as the
window_stack push a few lines above: visible windows only.

Both "focus the window once its IPC attaches" sites read that class to decide,
so a hidden child window pulled focus a beat after launching. Neither focuses
a window with data-is_visible="0" any more.

makeWindowInvisible left `window-active` on the window it had just hidden.
Since focusWindow() disables pointer events on every other app's iframe, an
app calling puter.ui.hideWindow() on itself left whoever launched it both
unfocused AND unclickable until the next click. It now hands activation back
to the top of the window stack, the way closing a window does, and drops
itself out of the activation order until it is shown again.

showWindow only ever un-minimized: for a window hidden by hideWindow() (which
keeps its geometry and writes no data-orig-*) every branch left it hidden and
stamped NaN geometry on it from the missing attributes. It now routes such a
window to makeWindowVisible. That is what makes the taskbar item a real
handle on a background app — clicking the item calls showWindow on the group.
2026-08-13 09:59:37 -07:00
Nariman Jelveh e273431f14 feat: let an app launch another app in the background
`puter.ui.launchApp(name, args)` had no way to say "I need this app's API,
not its window". That matters because we create and show an app's window
before the app's own code runs, so an app launched purely to serve another
one cannot avoid appearing on screen: the best it can do is call
`puter.ui.hideWindow()` once it boots, which reads as a window flashing
open and shut. In dashboard mode it was worse than a flash — the child
maximized into the tab and minimized its parent behind it, so asking a
service app a question took the user's app away from them.

So `launchApp` now accepts `background: true`, and the window starts
hidden. The app is otherwise entirely normal: it keeps its taskbar item,
so a user can see that it is running, show it, or close it, and it can
show itself with `puter.ui.showWindow()` whenever it has something to say.
Only a literal `true` counts, since the flag arrives over IPC from another
app.

The decision now lives in one predicate, `starts_hidden(app_info, options)`,
which folds this together with the existing app-level `background` flag and
is used everywhere the old flag was read — including the dashboard's
minimize-the-parent branch. `show_in_taskbar` deliberately still keys on the
app-level flag alone: an app that is always windowless has nothing to put in
the taskbar, while a background *launch* should stay visible there.

Existing callers are unaffected: with `background` unset, both paths
evaluate exactly as they did.
2026-08-13 09:47:35 -07:00
Daniel Salazar 8ed8feed4b fix: don't let an aborted upload take down the process (#3557) 2026-08-13 01:02:17 -07:00
Daniel Salazar 22f5bf5429 fix: duplicate emails (#3556) 2026-08-12 22:00:48 -07:00
Nariman Jelveh 198184f986 fix: drop the feedback dialog's rate limit entirely
Reopening still failed intermittently: any call the backoff refused looks
exactly like the user closing the dialog, so an app's "Send feedback"
button just did nothing, with nothing to see anywhere.

Rate limiting this dialog is not worth that. The other modals an app can
open with no user gesture — requestPermission, alert, prompt — have no
rate limit either, so one dialog at a time is the only rule left, and a
refusal now says so in the console.
2026-08-12 17:46:05 -07:00
Nariman Jelveh 026ae46530 fix: let an app reopen the feedback dialog after the user closes it
The abuse guard counted every dismissal that sent nothing as a strike
against the app, so closing the dialog blocked the app's next
showFeedbackDialog() for 10s, then 60s, then for the rest of the page's
life. The key is the app uuid, so relaunching the app did not clear it,
and a blocked call replies sent:false — the app just sees nothing happen.

Key the guard off how fast the app comes back instead. A reopen within a
second of the app's last dialog activity is machine-paced and gets the
same escalating backoff; attempts made while backed off count as activity
too, so a loop cannot wait out a tier and start from zero strikes. Any
human-paced reopen clears the app's record, so the tiers are only reached
by an app that reopens at machine speed several times running.

One dialog at a time, and a successful send clearing the record, are
unchanged.
2026-08-12 17:35:54 -07:00
Nariman Jelveh 0615c3aa3c feat: add "Show hidden" to the dashboard's folder context menu
The dashboard's files tab filtered out dot-files unconditionally, so
there was no way to reach them the way the desktop Explorer allows.
Add the same "Show hidden" toggle next to Refresh, driven by the
shared show_hidden_files preference so the two stay in sync, and dim
revealed rows with item-revealed like the Explorer does.

The socket-driven item.added path applies the same rule, so a hidden
file created elsewhere can't appear in a view that filtered its
siblings out.
2026-08-12 17:26:04 -07:00
Nariman Jelveh 748461aabb fix: show the new folder's row in the dashboard before mkdir answers (#3555)
* fix: show the new folder's row in the dashboard before mkdir answers

Both New Folder paths in the dashboard Files tab awaited mkdir before
drawing anything, so on a slow connection the button appeared to do
nothing for seconds. The toolbar path then did a full renderDirectory on
top of that -- a second round-trip, and with the default eventual
consistency the new folder could miss the listing entirely, leaving the
rename editor unopened.

Both now go through one createFolderInstant(), which renders the row
first: a locally predicted name (mirroring the backend's " (N)" dedupe
convention), a temporary uid, and the name editor already open. mkdir is
still asked for the plain "New Folder" path with rename: true, so the
server keeps owning deduplication; the prediction is only what we draw in
the meantime, and the row corrects itself if the two disagree. On failure
the row is withdrawn and the error is shown rather than swallowed.

Renaming is the next thing the user does and it needs the real uid, so
rename() awaits the create promise parked on the row. Everything else
that acts on the item -- open, menus, drag -- sits out while the row is
pending. mkdir's item.added comes back to the originating client (the
event carries no original_client_socket_id) and cannot match a temporary
uid, so _creatingItem still suppresses it, now as a counter: as a flag,
one create finishing uncovered another still in flight.

Two things found on the way:

- directory_depth_limit_exceeded had no translation key, so that alert
  (including the pre-existing desktop create_folder path) rendered the
  raw slug.
- The empty-directory notice is now restorable without refetching, since
  a withdrawn create has to put it back. It carries its own class because
  the loading overlay shares the same container.

Verified end to end against a dev backend with mkdir slowed to 3s:
toolbar and context-menu paths, rename and Escape before the response,
two overlapping creates, and a rejected mkdir.

* fix: don't draw the instant new-folder row while a listing is rebuilding

renderDirectory() clears .files before its readdir resolves, so a row
drawn during a load survives the clear and is left behind in whatever
directory the load lands on. Clicking the new-folder button while a
folder is still opening produced a row for `<old dir>/New Folder` sitting
in the new directory's listing, with the rename editor open on it —
clicking it navigated elsewhere and renaming it renamed a folder the user
wasn't looking at.

Drop the click while a rebuild is in flight, the same way renderDirectory
already drops navigation clicks that arrive while it renders.

* fix: stop drops onto a not-yet-created folder row from mangling files

A row drawn ahead of its mkdir carries a predicted path, but it was still
registered as a live drop target. Dropping a file on it called
move_items() with a destination that does not exist yet, and move treats
a non-existent destination as a rename target — so the dragged file was
silently renamed to "New Folder" instead of moving into the folder.
Reproduced against a slow mkdir: victim.txt became a 5-byte file named
"New Folder".

Sit the row out of the jQuery UI droppable (drop and the spring-load
hover) until the real fsentry lands, matching the guards already on
opening it, its menus and dragging it. The native-file dragster drop gets
the same guard: it uploads into the same predicted path.
2026-08-12 17:16:44 -07:00
Daniel Salazar 78cfe0c0f6 metrics: add metering buffer store coutns (#3554) 2026-08-12 15:55:03 -07:00
Nariman Jelveh 81f9f00fc0 fix: stop app feedback dialog clicks from focusing the iframe beneath
Clicking inside the feedback overlay (e.g. the textarea) reached
initgui's global mousedown -> focusWindow path: mouseover_window is
computed geometrically and is blind to the overlay, so focusWindow
focused the app window's iframe underneath — stealing keyboard focus
from the textarea and forwarding the click into the app.

Suppress it the established way (same fix as the dashboard app-group
panel): set window.mouseover_window = undefined in the overlay's own
mousedown handler, which runs before the document-level one; undefined
is the only value initgui's guard skips.
2026-08-12 15:00:32 -07:00
Nariman Jelveh 556380cf16 fix: keep app feedback send button blue on hover
The generic .app-feedback-btn:hover:not(:disabled) rule out-ranked the
.app-feedback-btn-primary:hover override (the :disabled argument inside
:not() counts toward specificity), so hovering Send repainted it with
the light --afb-hover background under white text. Strengthen the
primary hover selector so the blue gradient wins.
2026-08-12 13:42:11 -07:00
396 changed files with 42632 additions and 10360 deletions

No files matched your search

+1
View File
@@ -52,6 +52,7 @@ When in doubt, return less. Auth-, permission-, or data-export-related changes d
- **Test new behavior.** Every new function, endpoint, driver method, or logic branch gets a test; every bug fix gets a regression test that fails before the fix. If something is genuinely hard to test, skip it but say so in the PR.
- **Boy Scout Rule, proportional to the change.** Fix the obvious typo or dead import in files you're already touching; don't ride a refactor along with a bug fix.
- **Understand what you commit.** AI assistance is fine; shipping code you couldn't defend in review is not.
- **A limit change is not done until the docs change with it.** Every rate limit, concurrency cap, quota, and allowance in the code is published in [src/docs/src/rate-limits-and-quotas.md](src/docs/src/rate-limits-and-quotas.md) — an undisclosed limit is one developers discover as a service failure. If a PR moves any of these numbers, the same PR updates that page. The numbers live in `src/backend/controllers/fs/limits.ts` (filesystem), `src/backend/drivers/util/aiLimits.ts` (all AI drivers), per-driver `rateLimit`/`concurrent` configs, and `src/backend/data/subPolicies/` (free-tier allowances).
---
+2
View File
@@ -32,6 +32,8 @@ Don't commit code you couldn't have written, debugged, or defended yourself. Rea
If you add or change a public API (an endpoint, driver method, or puter-js method), follow [doc/contributing-apis.md](doc/contributing-apis.md) — backward compatibility, [developer docs](src/docs/), types, and tests all move in the same PR.
The same rule applies to limits: rate limits, concurrency caps, quotas, and allowances are published in [src/docs/src/rate-limits-and-quotas.md](src/docs/src/rate-limits-and-quotas.md), and a PR that changes one of those numbers updates that page in the same PR. A limit nobody published is a limit developers only discover as a service failure.
## 6. Boy Scout Rule — leave it 1% better
![Boy Scout Rule](https://imgs.search.brave.com/DMmIWl5-NuZVtrR9kXBb06AKF8kturkgSW9UMb2-6m4/rs:fit:860:0:0:0/g:ce/aHR0cHM6Ly9sYXdz/b2Zzb2Z0d2FyZWVu/Z2luZWVyaW5nLmNv/bS9pbWFnZXMvbGF3/cy9ib3ktc2NvdXQt/cnVsZS5wbmc)
-7
View File
@@ -12,10 +12,6 @@
·
<a href="https://developer.puter.com" target="_blank">Developers</a>
·
<a href="https://discord.com/invite/PQcx7Teh8u">Discord</a>
·
<a href="https://reddit.com/r/puter">Reddit</a>
·
<a href="https://twitter.com/HeyPuter">X</a>
</p>
@@ -82,10 +78,7 @@ Puter is available as a hosted service at [**puter.com**](https://puter.com).
Connect with the maintainers and community through these channels:
- Bug report or feature request? Please [open an issue](https://github.com/HeyPuter/puter/issues/new/choose).
- Discord: [discord.com/invite/PQcx7Teh8u](https://discord.com/invite/PQcx7Teh8u)
- X (Twitter): [x.com/HeyPuter](https://x.com/HeyPuter)
- Reddit: [reddit.com/r/puter/](https://www.reddit.com/r/puter/)
- Mastodon: [mastodon.social/@puter](https://mastodon.social/@puter)
- Security issues or abuse reports? [[email protected]](mailto:[email protected])
- Email maintainers at [[email protected]](mailto:[email protected])
+41 -2
View File
@@ -251,6 +251,28 @@
}
},
// ── Sharing ─────────────────────────────────────────────────────────
// Email a recipient who already has an account about a new share. On
// unless set to false; they opt out via the unsubscribe link, or by
// blocking a sender. An invite to an address with no account always goes.
// "share_email_notifications": false,
//
// New shares one account may create per UTC day. Default 200.
// "share_daily_limit": 200,
//
// How often a share may interrupt its recipient — the notification pushed
// to their screen and the email with it. Over budget the share still
// succeeds and their notification is still brought up to date; only the
// interruption is dropped. The recipient bounds are what stop many senders
// burying one person between them. A non-positive value removes a bound.
// "share_notify_limits": {
// "pairWindowSeconds": 900,
// "pairDaily": 20,
// "recipientHourly": 10,
// "recipientDaily": 50,
// "emailBatchSeconds": 90
// },
// ── Alarms / alerting ───────────────────────────────────────────────
// Where system alarms go. Severity is the routing decision — each
// transport takes everything at or above its own `minSeverity`:
@@ -324,6 +346,10 @@
"apiURL": ""
},
"gemini": { "apiKey": "" },
// Meta Model API (Muse Spark). `muse-spark-1.2-contributor`
// trades a ~12x discount for Meta training on its prompts and
// completions, so it is only served to callers that name it.
"meta": { "apiKey": "" },
"groq": { "apiKey": "" },
"deepseek": { "apiKey": "" },
"mistral": { "apiKey": "" },
@@ -341,6 +367,16 @@
"apiKey": "",
"apiBaseUrl": "https://llm.onerouter.pro/v1"
},
// BytePlus ModelArk. One key powers chat (Seed/GLM/DeepSeek), image
// generation (Seedream) and video generation (Seedance); `apiBaseUrl`
// selects the region; see
// https://docs.byteplus.com/en/docs/ModelArk/1330310 for options.
"byteplus": {
"apiKey": "",
"apiBaseUrl": "https://ark.ap-southeast.bytepluses.com/api/v3"
},
"zai": { "apiKey": "" },
"alibaba": { "apiKey": "" },
"together-ai": { "apiKey": "" },
// Local Ollama. `enabled: false` skips the auto-probe at startup
// (otherwise Puter logs ECONNREFUSED on every boot when no Ollama
@@ -448,8 +484,11 @@
// operations that spend it — file transfers, KV calls. Recording is
// unaffected either way. `workers` extends the same refusal to
// worker-driven calls, which are exempt by default because a deployed
// worker has nowhere to surface a payment prompt.
// "meteringEnforcement": { "enabled": true, "workers": false },
// worker has nowhere to surface a payment prompt. `subscriptions` covers
// the separate question of surfaces reserved for paid plans (the
// vendor-compatible AI endpoints): turn it off on a deployment with no
// paid plans, or every account is refused them.
// "meteringEnforcement": { "enabled": true, "workers": false, "subscriptions": true },
// Fleet-wide spend rate, in micro-cents per minute, past which metering
// raises the `metering:excessiveGlobalUsageRate` alarm. Omit it (the
+9 -1
View File
@@ -44,7 +44,15 @@ pass one explicitly unless you really mean "page someone".
event; the 429 is the whole signal, and alarming on it only produces noise
proportional to traffic. An *upstream provider* rate-limiting us is the
opposite case and still alarms (`upstream_rate_limited`, `info`) — that one
is not something we chose.
is not something we chose. The exception is a free model: nothing is billed,
nothing is actionable, and the throttling is the price of the model, so the
AI chat driver marks those `noAlarm` and only paid models still record.
`noAlarm` is the code-level mute: an `HttpError` carrying it skips the
terminal gate entirely, no matter its status or code. It is for failures the
call site *already knows* are expected and traffic-proportional — reach for it
there, not as a way to quiet an alarm you haven't diagnosed
(`severityOverrides` below is the knob for that).
An extension whose signals are all one tier can default its own local
`raiseAlarm` helper to that tier instead of repeating it at every call site —
+3 -1
View File
@@ -53,7 +53,9 @@ Both are supported ways to define an API. **Prefer a controller when you need fi
#### Middleware and gates
- **Controller routes** (and extension routes — same options) take [`RouteOptions`](../src/backend/core/http/types.ts): auth gates (`requireAuth`, `requireUserActor`, `noUserSession`, `adminOnly`, `allowedAppIds`, and the access-token controls), `subdomain` routing, per-route `rateLimit`, body parsers, and arbitrary extra `middleware`. The auth flavors are subtle and default-deny — read the JSDoc on each field before picking.
- **Driver methods** get their policies from the `@Driver` options: per-method `rateLimit` (limit/window/backend), `concurrent` in-flight caps (optionally `bySubscription`), and `noUserSession`. The `/drivers/call` surface enforces them.
- **Driver methods** get their policies from the `@Driver` options: per-method `rateLimit` (limit/window/backend), `concurrent` in-flight caps (optionally `bySubscription`), `requireSubscription`, and `noUserSession`. The `/drivers/call` surface enforces them.
- **A surface only paying accounts should reach** declares `requireSubscription` — the route option on a controller endpoint, the per-method `@Driver({ requireSubscription })` block on a driver (`/drivers/call` is one shared route, so a driver's requirement can't live in route options). `true` accepts any plan that isn't free, so a plan registered by an extension counts without core naming it; an array of policy ids (`['business', 'pro']`) accepts only those; `false` is "no requirement", the same as leaving it out. An empty array is a boot error rather than a silent no-op — it reads as subscribers-only while admitting everyone. Off unless asked for, and answered from the metering service's cached per-actor subscription, so it costs a map lookup. Distinct from `requireCredits`: this asks which plan the account is on, not whether it has budget left. Deployments with no paid plans (self-hosted installs) turn the whole thing off with `meteringEnforcement.subscriptions: false`.
- **An account must have verified a specific factor** — `requireVerified` (email, and only under `strict_email_verification_required`), `requirePhoneVerified`, `requireCardVerified`. These are opt-in and require the factor to have actually been verified, unlike the default-on gate that turns away accounts still carrying a pending verification the abuse harness asked for.
- **An endpoint that spends metered resources** on the caller's behalf — moving file content, making object-store requests, anything else the account is billed for — also declares `requireCredits: true`, which turns an account with nothing left of its budget away with a 402 before the handler runs. Endpoints that only describe or delete things deliberately don't: an account that has run out still has to be able to see what it has, clear it, and reach its billing pages. Drivers have no route options to declare this on, so they call `assertActorHasCredits` themselves ([src/backend/services/metering/enforcement.ts](../src/backend/services/metering/enforcement.ts)) — see `KVStoreDriver`, which does it once for every method.
### 3. puter.js
+17
View File
@@ -122,6 +122,10 @@ cat > puter/config/config.json <<EOF
"ollama": { "enabled": false }
},
"meteringEnforcement": {
"subscriptions": false
},
"trust_proxy": 1
}
EOF
@@ -136,6 +140,7 @@ Why these knobs:
- `database.migrationPaths` — Puter applies the bundled MySQL/MariaDB schema on boot. The migration files are idempotent, so it is safe to leave this configured across restarts.
- `dynamo.bootstrapTables: true` — Puter creates its KV table on boot. **Only set against a local emulator**, never real AWS.
- `dynamo.aws` keys are dummies; DynamoDB-local doesn't validate them but the AWS SDK requires _something_. **Note:** DynamoDB uses `access_key` / `secret_key` (snake_case); S3 below uses `accessKeyId` / `secretAccessKey` (camelCase). Not interchangeable.
- `meteringEnforcement.subscriptions: false` — a few surfaces (the OpenAI/Anthropic-compatible AI endpoints) are declared paid-plan-only. A self-hosted install has no paid plans, so every account sits on the free policy and those routes would answer `402 subscription_required` for everyone. Turning the plan gates off opens them; nothing else about metering changes.
- `providers.ollama.enabled: false` — Puter auto-probes a local Ollama at `127.0.0.1:11434` by default; without one running you'd see `ECONNREFUSED` on every boot. To run a bundled Ollama, see [Optional: local LLM (Ollama)](#optional-local-llm-ollama) below.
- `s3.s3Config.forcePathStyle: true` — RustFS / MinIO / fauxqs need path-style URLs (`<endpoint>/<bucket>`). Real AWS S3 wants virtual-hosted (`<bucket>.<endpoint>`) — drop this flag (or set `false`) when you swap to real S3.
- `s3.s3Config.publicEndpoint` — `endpoint` (`http://s3:9000`) only resolves inside the docker network; presigned upload/download URLs handed to the browser need a host-reachable URL. Caddy routes the `s3.<domain>` subdomain to RustFS internally and preserves the Host header end-to-end (required for S3 signature validation), so the browser hits the same port/protocol as the rest of the app — no separate published port, no mixed-content surprises when you turn on TLS. Switch to `https://s3.<your-domain>` once you enable TLS in Step 3. Real AWS S3 doesn't need this — its endpoint is already public; drop the field entirely.
@@ -295,6 +300,18 @@ Used for password resets, email confirmation, and notifications. Without it thos
To require email confirmation before login, also set `"strict_email_verification_required": true`.
To read the mail yourself instead of sending it, point the transport at a local SMTP sink — [MailHog](https://github.com/mailhog/MailHog) catches everything and serves it at `http://localhost:8025`. Publish both ports; a container that only exposes them is unreachable from the host:
```sh
docker run -d -p 1025:1025 -p 8025:8025 mailhog/mailhog
```
```json
"email": { "from": "\"Puter\" <[email protected]>", "host": "127.0.0.1", "port": 1025, "secure": false, "ignoreTLS": true }
```
Share email is on once a transport exists: a recipient who already has an account is emailed as well as notified in-app, and they opt out with the unsubscribe link the mail carries. Set `"share_email_notifications": false` to keep those in the app only — an address with no account is still emailed, since there is nothing else to reach them with.
### Sign in with Google (or another OIDC provider)
```json
+107 -3
View File
@@ -6,6 +6,10 @@ import {
servicesContainers,
} from '@heyputer/backend/src/exports';
import { extension } from '@heyputer/backend/src/extensions';
import {
creditMultiplierFrom,
toCredits,
} from '@heyputer/backend/src/services/metering/utils';
import type { Request, Response } from 'express';
const services = extension.import('service');
@@ -50,6 +54,42 @@ function collectAllCosts(): Record<string, unknown>[] {
return all;
}
// -- Credits scaling ---------------------------------------------------
//
// When the deployment configures a credit multiplier, every monetary amount
// is scaled to display credits BEFORE it leaves the server — raw metered
// amounts are never reported. Counts, units, and byte figures pass through
// untouched. A response that was scaled says so (`unit: 'credits'`); without
// a configured multiplier amounts pass through raw and clients render them
// as dollars.
/** The per-usage-type records of a UsageByType map, costs scaled. */
const scaleUsageByType = (
usage: Record<string, unknown>,
multiplier: number,
): Record<string, unknown> => {
const out: Record<string, unknown> = {};
for (const [key, value] of Object.entries(usage)) {
// `allowanceUsed` is the month's allowance-charged spend — monetary,
// scalar, scaled like the total.
if (key === 'total' || key === 'allowanceUsed') {
out[key] = toCredits(Number(value) || 0, multiplier);
} else if (
value &&
typeof value === 'object' &&
typeof (value as { cost?: unknown }).cost === 'number'
) {
out[key] = {
...(value as Record<string, unknown>),
cost: toCredits((value as { cost: number }).cost, multiplier),
};
} else {
out[key] = value;
}
}
return out;
};
export const handleMeteringUsage = async (
_req: Request,
res: Response,
@@ -61,7 +101,44 @@ export const handleMeteringUsage = async (
services.metering.getActorCurrentMonthUsageDetails(actor),
services.metering.getAllowedUsage(actor),
]);
res.json({ ...actorUsage, allowanceInfo });
const multiplier = creditMultiplierFrom(extension.config);
if (!multiplier) {
res.json({ ...actorUsage, allowanceInfo });
return;
}
res.json({
usage: scaleUsageByType(
actorUsage.usage as unknown as Record<string, unknown>,
multiplier,
),
appTotals: Object.fromEntries(
Object.entries(actorUsage.appTotals).map(([appId, totals]) => [
appId,
{ ...totals, total: toCredits(totals.total, multiplier) },
]),
),
allowanceInfo: {
...allowanceInfo,
remaining: toCredits(allowanceInfo.remaining, multiplier),
monthUsageAllowance: toCredits(
allowanceInfo.monthUsageAllowance,
multiplier,
),
// Monetary addon fields scale; absent ones stay absent rather
// than materializing as NaN/null. purchasedStorage is bytes.
addons: Object.fromEntries(
Object.entries(allowanceInfo.addons ?? {}).map(([k, v]) => [
k,
(k === 'purchasedCredits' ||
k === 'consumedPurchaseCredits') &&
typeof v === 'number'
? toCredits(v, multiplier)
: v,
]),
),
unit: 'credits',
},
});
};
export const handleMeteringUsageForApp = async (
@@ -92,7 +169,18 @@ export const handleMeteringUsageForApp = async (
actor,
appId,
);
res.json(appUsage);
const multiplier = creditMultiplierFrom(extension.config);
if (!multiplier) {
res.json(appUsage);
return;
}
res.json({
...scaleUsageByType(
appUsage as unknown as Record<string, unknown>,
multiplier,
),
unit: 'credits',
});
};
export const handleMeteringGlobalUsage = async (
@@ -109,7 +197,23 @@ export const handleMeteringAllCosts = async (
res: Response,
): Promise<void> => {
if (!cachedAllCosts) {
cachedAllCosts = collectAllCosts();
const raw = collectAllCosts();
const multiplier = creditMultiplierFrom(extension.config);
// Price the catalogue in the same unit the usage endpoints report:
// per-unit credits when a multiplier is configured, raw otherwise.
cachedAllCosts = multiplier
? raw.map(({ ucentsPerUnit, costs_currency: _cc, ...rest }) =>
typeof ucentsPerUnit === 'number'
? {
...rest,
creditsPerUnit: toCredits(
ucentsPerUnit,
multiplier,
),
}
: rest,
)
: raw;
}
res.json({ costs: cachedAllCosts });
};
+6
View File
@@ -160,6 +160,10 @@ export const handleWhoami = async (
desktop_bg_fit: user.desktop_bg_fit,
is_temp: user.password === null && user.email === null,
is_user_token: true,
// Present only once the account has actually asked for a code (see the
// referral extension) — null until then, and never minted from here:
// this endpoint is polled, and a mint is a write.
referral_code: user.referral_code,
oidc_only: oidcOnly,
taskbar_items: isUser
? await getTaskbarItems(
@@ -240,6 +244,8 @@ export const handleWhoami = async (
delete details.created_ts;
delete details.is_user_token;
delete details.metadata;
// An app has no business reading the code its user earns credit with.
delete details.referral_code;
}
if (actor.app) {
+9
View File
@@ -131,6 +131,11 @@ S3_SECRET_KEY=$S3_SECRET_KEY
S3_BUCKET=puter-local
EOF
# `meteringEnforcement.subscriptions` is off here because a self-hosted
# install has no paid plans to be on: every account resolves to the free
# policy, so leaving the plan gates armed would 402 the surfaces that
# declare one (the OpenAI/Anthropic-compatible AI endpoints) for everybody.
# The code default is on — this is a property of the install, not of Puter.
log "writing puter/config/config.json"
cat > puter/config/config.json <<EOF
{
@@ -191,6 +196,10 @@ EOF
"ollama": { "enabled": false }
},
"meteringEnforcement": {
"subscriptions": false
},
"trust_proxy": $PUTER_TRUST_PROXY
}
EOF
+2102 -3710
View File
File diff suppressed because it is too large. Load diff
+14 -25
View File
@@ -13,14 +13,9 @@
"@babel/core": "^7.29.7",
"@eslint/js": "^9.35.0",
"@playwright/test": "^1.56.1",
"@stylistic/eslint-plugin": "^5.3.1",
"@types/better-sqlite3": "^7.6.13",
"@types/express": "^5.0.0",
"@types/mime-types": "^3.0.1",
"@typescript-eslint/eslint-plugin": "^8.46.1",
"@typescript-eslint/parser": "^8.46.1",
"@vitest/coverage-v8": "^4.0.14",
"@vitest/ui": "^4.0.14",
"babel-loader": "^10.1.1",
"babel-plugin-istanbul": "^8.0.0",
"chalk": "^4.1.0",
@@ -30,25 +25,19 @@
"eslint": "^9.35.0",
"eslint-config-prettier": "^10.1.8",
"eslint-plugin-prettier": "^5.5.5",
"eslint-rule-composer": "^0.3.0",
"globals": "^15.15.0",
"html-webpack-plugin": "^5.6.0",
"husky": "^9.1.7",
"istanbul-lib-coverage": "^3.2.2",
"istanbul-lib-report": "^3.0.1",
"istanbul-reports": "^3.2.0",
"license-check-and-add": "^4.0.5",
"nodemon": "^3.1.0",
"prettier": "^3.8.3",
"prettier-plugin-jsdoc": "^1.8.1",
"simple-git": "^3.32.3",
"typescript": "^5.4.5",
"uglify-js": "^3.17.4",
"vite-plugin-static-copy": "^3.3.0",
"vitest": "^4.1.5",
"webpack": "^5.88.2",
"webpack-cli": "^5.1.1",
"yaml": "^2.8.1"
"webpack-cli": "^5.1.1"
},
"scripts": {
"test:backend": "npm run setupExtensions && npm run build:workerLib && vitest run --config src/backend/vitest.config.ts ",
@@ -73,9 +62,7 @@
"setupExtensions": "node ./tools/extensionSetup.mjs"
},
"workspaces": [
"src/*",
"tools/*",
"experiments/js-parse-and-output"
"src/*"
],
"nodemonConfig": {
"ext": "js, json, mjs, jsx, svg, css",
@@ -85,11 +72,8 @@
]
},
"dependencies": {
"@ai-sdk/openai": "^3.0.25",
"@aws-sdk/client-s3": "^3.1020.0",
"@aws-sdk/s3-request-presigner": "^3.1028.0",
"@heyputer/putility": "^1.0.2",
"ai": "^6.0.73",
"dedent": "^1.5.3",
"javascript-time-ago": "^2.5.11",
"libphonenumber-js": "1.13.6",
@@ -101,12 +85,17 @@
},
"allowScripts": {
"@google/genai": false,
"bcrypt@5.1.1": true,
"better-sqlite3@12.11.1": true,
"browser-tabs-lock@1.3.0": true,
"classic-level@3.0.0": true,
"[email protected]": true,
"[email protected]": true,
"[email protected]": true
"bcrypt": true,
"better-sqlite3": true,
"browser-tabs-lock": true,
"classic-level": true,
"esbuild": true,
"fsevents": true,
"protobufjs": true,
"sharp": true,
"workerd": true
},
"overrides": {
"undici": "^7.29.0"
}
}
@@ -18,7 +18,7 @@
*/
import Database from 'better-sqlite3';
import { existsSync, mkdtempSync, rmSync } from 'node:fs';
import { existsSync, mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
@@ -27,7 +27,15 @@ import { DatabaseClientFactory } from './index.js';
import { SqliteDatabaseClient } from './SqliteDatabaseClient.js';
/** Highest schema version the migration table can reach. */
const CURRENT_SCHEMA_VERSION = 61;
const CURRENT_SCHEMA_VERSION = 66;
/**
* These suites migrate real files on disk. Idle they finish in well under a
* second, but the whole migration chain runs against the filesystem — enough
* that a loaded runner blows the 5s default and reports a timeout where there
* is no fault.
*/
const DISK_MIGRATION_TIMEOUT_MS = 30_000;
const SYSTEM_USER_UUID = '5d4adce0-a381-4982-9c02-6e2540026238';
const sqliteConfig = (
@@ -52,7 +60,7 @@ const userVersionOf = async (client: SqliteDatabaseClient): Promise<number> => {
return row.user_version as number;
};
describe('SqliteDatabaseClient — boot and migrations', () => {
describe('SqliteDatabaseClient — boot and migrations', { timeout: DISK_MIGRATION_TIMEOUT_MS }, () => {
let client: SqliteDatabaseClient;
beforeEach(async () => {
@@ -83,6 +91,105 @@ describe('SqliteDatabaseClient — boot and migrations', () => {
expect(rows).toEqual([{ username: 'system' }]);
});
// 0070 sweeps three by-hand groups no code reads. Both tables it can reach
// cascade on delete, so the guard is the point: a group that still carries
// permissions or members has to survive, or the sweep silently revokes
// them. Re-running the statement is safe — it is a plain conditional DELETE.
describe('0070 drop-orphaned-default-groups', () => {
const MIGRATION = readFileSync(
new URL(
'./migrations/sqlite/0070_drop-orphaned-default-groups.sql',
import.meta.url,
),
'utf8',
);
const seedGroup = async (name: string): Promise<number> => {
const [system] = await client.read(
'SELECT `id` FROM `user` WHERE `uuid` = ?',
[SYSTEM_USER_UUID],
);
const uid = `grp-${name}-${Math.random().toString(36).slice(2, 10)}`;
await client.write(
'INSERT INTO `group` (`uid`, `owner_user_id`, `extra`, `metadata`) ' +
'VALUES (?, ?, ?, ?)',
[uid, system.id, JSON.stringify({ name }), '{}'],
);
const [row] = await client.read(
'SELECT `id` FROM `group` WHERE `uid` = ?',
[uid],
);
return Number(row.id);
};
const stillThere = async (id: number): Promise<boolean> => {
const rows = await client.read(
'SELECT `id` FROM `group` WHERE `id` = ?',
[id],
);
return rows.length > 0;
};
it('drops an orphaned freeai/experimental/dangerous group', async () => {
const ids = await Promise.all(
['freeai', 'experimental', 'dangerous'].map(seedGroup),
);
await client.write(MIGRATION);
for (const id of ids) expect(await stillThere(id)).toBe(false);
});
it('spares one that still carries a permission row', async () => {
const id = await seedGroup('freeai');
const [system] = await client.read(
'SELECT `id` FROM `user` WHERE `uuid` = ?',
[SYSTEM_USER_UUID],
);
await client.write(
'INSERT INTO `user_to_group_permissions` ' +
'(`user_id`, `group_id`, `permission`, `extra`) VALUES (?, ?, ?, ?)',
[system.id, id, 'service:some-paid-thing', '{}'],
);
await client.write(MIGRATION);
expect(await stillThere(id)).toBe(true);
// And the grant it carries is still intact, not cascaded away.
const perms = await client.read(
'SELECT `permission` FROM `user_to_group_permissions` WHERE `group_id` = ?',
[id],
);
expect(perms).toEqual([{ permission: 'service:some-paid-thing' }]);
});
it('spares one that still has members', async () => {
const id = await seedGroup('experimental');
const [system] = await client.read(
'SELECT `id` FROM `user` WHERE `uuid` = ?',
[SYSTEM_USER_UUID],
);
await client.write(
'INSERT INTO `jct_user_group` (`user_id`, `group_id`) VALUES (?, ?)',
[system.id, id],
);
await client.write(MIGRATION);
expect(await stillThere(id)).toBe(true);
});
it('leaves the groups the platform actually depends on alone', async () => {
// system, admin, user and temp are named by config or code.
await client.write(MIGRATION);
const rows = await client.read(
"SELECT json_extract(`extra`, '$.name') AS name FROM `group`",
);
const names = rows.map((r) => String(r.name));
for (const name of ['system', 'admin', 'user']) {
expect(names).toContain(name);
}
});
});
it('leaves an already-migrated database untouched on a second boot', async () => {
const dir = mkdtempSync(join(tmpdir(), 'puter-sqlite-'));
const path = join(dir, 'nested', 'puter.sqlite');
@@ -137,7 +244,10 @@ describe('SqliteDatabaseClient — boot and migrations', () => {
});
});
describe('SqliteDatabaseClient — legacy version inference', () => {
describe(
'SqliteDatabaseClient — legacy version inference',
{ timeout: DISK_MIGRATION_TIMEOUT_MS },
() => {
let dir: string;
beforeEach(() => {
@@ -95,6 +95,11 @@ const AVAILABLE_MIGRATIONS: [number, string[]][] = [
[58, ['0063_add_suspended_reason.sql']],
[59, ['0064_abuse-moderation-events.sql']],
[60, ['0065_app-feedback.sql']],
[61, ['0066_owned-email-unique.sql']],
[62, ['0067_share_entries.sql']],
[63, ['0068_referral-code-unique.sql']],
[64, ['0069_user-block.sql']],
[65, ['0070_drop-orphaned-default-groups.sql']],
];
export class SqliteDatabaseClient extends AbstractDatabaseClient {
@@ -0,0 +1,128 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Enforce "at most one account owns an email address". Mirrors SQLite
-- migration 0066.
--
-- `user.email` is deliberately not UNIQUE: several rows may legitimately hold
-- the same address while unconfirmed (admin-provisioned placeholders, signups
-- that were never confirmed, temp accounts on their way to becoming real). What
-- must never happen is two rows both *owning* an address — owning meaning the
-- row is confirmed, or holds a password and so can drive password recovery for
-- that inbox.
--
-- Signup, save-account, change-email, OIDC and admin provisioning each check for
-- an owner before writing, but a check and an insert are not one operation: two
-- requests can both read "free" and both write. This index is what actually
-- holds the invariant; the application checks just produce a nicer error most of
-- the time.
--
-- SQLite expresses that with a partial index. MySQL has none, so the predicate
-- lives in a generated column that evaluates to NULL for every row that does not
-- own its address — and NULLs do not collide in an InnoDB unique index, which is
-- exactly the "unlimited unconfirmed placeholders" behaviour we need.
--
-- The column is VIRTUAL, not STORED, on purpose: adding a stored generated
-- column rebuilds the table, while a virtual one is a metadata-only change and
-- the index that follows builds INPLACE. On a `user` table of any size with
-- read replicas attached, that is the difference between a routine change and an
-- outage. The ALGORITHM/LOCK clauses are spelled out so a server that cannot
-- honour them refuses the statement instead of quietly copying the table.
--
-- Matching is on the canonical address so provider aliases
-- (`[email protected]` vs `[email protected]`) collide. `clean_email` is
-- written on every modern write path; the COALESCE covers rows old enough to
-- predate the column. Run the `clean_email` backfill before this migration or
-- alias collisions among those rows go unnoticed.
--
-- Idempotent: both steps are guarded on INFORMATION_SCHEMA so the directory
-- replays safely.
--
-- If the index creation fails with ER_DUP_ENTRY, the DB already contains
-- duplicate owners. Collapse them first (admin → One-off Jobs → Collapse
-- Duplicate Emails) — there is no safe automatic merge of two accounts.
DROP PROCEDURE IF EXISTS _puter_add_owned_email;
DELIMITER //
CREATE PROCEDURE _puter_add_owned_email()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user'
AND COLUMN_NAME = 'owned_email'
) THEN
ALTER TABLE `user`
ADD COLUMN `owned_email` VARCHAR(256)
CHARACTER SET latin1 COLLATE latin1_swedish_ci
GENERATED ALWAYS AS (
CASE
WHEN `email` IS NOT NULL
AND (`email_confirmed` = 1 OR `password` IS NOT NULL)
THEN COALESCE(`clean_email`, LOWER(`email`))
ELSE NULL
END
) VIRTUAL,
ALGORITHM=INSTANT;
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user'
AND INDEX_NAME = 'idx_user_owned_email'
) THEN
ALTER TABLE `user`
ADD UNIQUE KEY `idx_user_owned_email` (`owned_email`),
ALGORITHM=INPLACE, LOCK=NONE;
END IF;
END//
DELIMITER ;
CALL _puter_add_owned_email();
DROP PROCEDURE IF EXISTS _puter_add_owned_email;
-- One Puter account per external identity. OIDCStore.link already assumes this
-- constraint exists — it catches the unique violation to tell "re-linking the
-- same account" apart from "this sub belongs to someone else" — but the table
-- never actually had it, so two concurrent first-time logins could each create
-- an account and each link the same sub. Subsequent logins then resolved to
-- whichever row came back first.
--
-- Dedupe `user_oidc_providers` before applying this: keep the lowest `id` per
-- (provider, provider_sub) and point it at the account the collapse job kept.
DROP PROCEDURE IF EXISTS _puter_add_oidc_sub_unique;
DELIMITER //
CREATE PROCEDURE _puter_add_oidc_sub_unique()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user_oidc_providers'
AND INDEX_NAME = 'idx_user_oidc_provider_sub'
) THEN
ALTER TABLE `user_oidc_providers`
ADD UNIQUE KEY `idx_user_oidc_provider_sub` (`provider`, `provider_sub`);
END IF;
END//
DELIMITER ;
CALL _puter_add_oidc_sub_unique();
DROP PROCEDURE IF EXISTS _puter_add_oidc_sub_unique;
@@ -0,0 +1,106 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Grow `share` from a pending-email-invite table into the index of active
-- shares. See sqlite/0067_share_entries.sql for the column rationale.
--
-- Idempotent: columns go through _puter_add_col (from mig_1), indexes and
-- foreign keys through the guarded procedure below. There is no per-file
-- applied-state tracking, so every statement must tolerate a re-run.
CALL _puter_add_col('share', 'holder_user_id', '`holder_user_id` int unsigned DEFAULT NULL');
CALL _puter_add_col('share', 'fsentry_id', '`fsentry_id` int unsigned DEFAULT NULL');
CALL _puter_add_col('share', 'mode', '`mode` varchar(20) DEFAULT NULL');
CALL _puter_add_col('share', 'applied_at', '`applied_at` timestamp NULL DEFAULT NULL');
DROP PROCEDURE IF EXISTS _puter_add_share_index_constraints;
DELIMITER //
CREATE PROCEDURE _puter_add_share_index_constraints()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_holder'
) THEN
ALTER TABLE `share` ADD INDEX `idx_share_holder` (`holder_user_id`, `id`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_fsentry'
) THEN
ALTER TABLE `share` ADD INDEX `idx_share_fsentry` (`fsentry_id`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_holder_entry_issuer'
) THEN
ALTER TABLE `share` ADD UNIQUE INDEX `idx_share_holder_entry_issuer`
(`holder_user_id`, `fsentry_id`, `issuer_user_id`);
END IF;
-- Retiring a deleted node's grants looks them up by permission text
-- (`fs:<uuid>` and `fs:<uuid>:%`); the subject lives in that text rather
-- than a column, so no foreign key can cascade it. The primary key is
-- (issuer, holder, permission), which never puts `permission` first, so
-- without this both the equality and the left-anchored LIKE degrade to a
-- full scan on the delete path. The column is ascii varchar(255), well
-- inside InnoDB's key limit, so it is indexed whole.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user_to_user_permissions'
AND INDEX_NAME = 'idx_user_to_user_permissions_permission'
) THEN
ALTER TABLE `user_to_user_permissions`
ADD INDEX `idx_user_to_user_permissions_permission` (`permission`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND CONSTRAINT_NAME = 'share_holder_user_fk'
) THEN
ALTER TABLE `share` ADD CONSTRAINT `share_holder_user_fk`
FOREIGN KEY (`holder_user_id`) REFERENCES `user` (`id`)
ON DELETE CASCADE ON UPDATE CASCADE;
END IF;
-- The cascade that retires a share with its file.
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND CONSTRAINT_NAME = 'share_fsentry_fk'
) THEN
ALTER TABLE `share` ADD CONSTRAINT `share_fsentry_fk`
FOREIGN KEY (`fsentry_id`) REFERENCES `fsentries` (`id`)
ON DELETE CASCADE ON UPDATE CASCADE;
END IF;
END//
DELIMITER ;
CALL _puter_add_share_index_constraints();
DROP PROCEDURE IF EXISTS _puter_add_share_index_constraints;
@@ -0,0 +1,35 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- One user refusing contact from another. See sqlite/0068_user-block.sql for
-- the rationale. `created_at` is unix seconds.
--
-- Idempotent: `CREATE TABLE IF NOT EXISTS` with the indexes declared inline,
-- so the directory can replay safely.
CREATE TABLE IF NOT EXISTS `user_block` (
`id` INT NOT NULL AUTO_INCREMENT,
`blocker_user_id` INT UNSIGNED NOT NULL,
`blocked_user_id` INT UNSIGNED NOT NULL,
`created_at` BIGINT NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_user_block_pair` (`blocker_user_id`, `blocked_user_id`),
CONSTRAINT `fk_user_block_blocker` FOREIGN KEY (`blocker_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_user_block_blocked` FOREIGN KEY (`blocked_user_id`)
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
@@ -0,0 +1,61 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Widen `notification.shown` and `notification.acknowledged` to hold the unix
-- second the store has been writing to them since the backend rework.
--
-- Both arrived from the v1 schema as `tinyint(1)`, where they were flags. The
-- rework changed the writes to timestamps and sqlite (`INTEGER`) and postgres
-- (`bigint`) took them; only mysql was left too narrow, so every `markShown`
-- and `markAcknowledged` there fails with ER_WARN_DATA_OUT_OF_RANGE and the
-- column stays NULL. Dismissing a notification therefore never sticks, and one
-- already delivered is re-sent on every reconnect.
--
-- No backfill: every reader tests `IS NULL` / `IS NOT NULL` only, so a legacy
-- `1` keeps meaning "yes" once widened.
--
-- Guarded on the current type rather than run unconditionally. Changing a
-- column type copies the table, and this one grows without bound — a migration
-- directory that replays on every boot must not rebuild it every time.
DROP PROCEDURE IF EXISTS _puter_widen_notification_stamps;
DELIMITER //
CREATE PROCEDURE _puter_widen_notification_stamps()
BEGIN
IF EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'notification'
AND COLUMN_NAME = 'shown'
AND DATA_TYPE = 'tinyint'
) THEN
ALTER TABLE `notification` MODIFY `shown` BIGINT DEFAULT NULL;
END IF;
IF EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'notification'
AND COLUMN_NAME = 'acknowledged'
AND DATA_TYPE = 'tinyint'
) THEN
ALTER TABLE `notification` MODIFY `acknowledged` BIGINT DEFAULT NULL;
END IF;
END //
DELIMITER ;
CALL _puter_widen_notification_stamps();
DROP PROCEDURE IF EXISTS _puter_widen_notification_stamps;
@@ -0,0 +1,22 @@
-- Drops three groups that no code has ever read: `freeai`, `experimental` and
-- `dangerous`. No migration creates them — they were added by hand back when
-- hardcoded permissions were keyed by group name. That map is now a flat
-- per-user floor (`default_user_permissions` in data/hardcoded-permissions.js),
-- so a group nothing looks up grants nothing, and the rows are dead weight.
--
-- Guarded, and deliberately so: `user_to_group_permissions.group_id` and
-- `jct_user_group.group_id` both cascade on delete, so removing a group that
-- still carries permissions or members would silently revoke them from every
-- member. Only a group with neither is dropped. A group that survives this
-- migration has dependents and needs a deliberate decision, not a sweep --
-- query `user_to_group_permissions` by `group_id` to see what it holds.
--
-- The joins rather than `NOT IN` subqueries: MySQL will not read from the table
-- it is deleting from inside a subquery on the same statement.
DELETE g FROM `group` g
LEFT JOIN `user_to_group_permissions` p ON p.`group_id` = g.`id`
LEFT JOIN `jct_user_group` j ON j.`group_id` = g.`id`
WHERE JSON_UNQUOTE(JSON_EXTRACT(g.`extra`, '$.name'))
IN ('freeai', 'experimental', 'dangerous')
AND p.`group_id` IS NULL
AND j.`group_id` IS NULL;
@@ -0,0 +1,40 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Enforce "at most one account owns an email address". Mirrors SQLite
-- migration 0066 and MySQL migration 21; see those for the full rationale.
--
-- In short: `user.email` is deliberately not UNIQUE because several rows may
-- hold the same address while unconfirmed. What must not happen is two rows
-- both owning it — confirmed, or holding a password and so able to drive
-- password recovery for that inbox. The application checks for an owner before
-- every write, but a check and a write are not one operation.
--
-- If this fails, the DB already contains duplicate owners; collapse them first
-- (admin → One-off Jobs → Collapse Duplicate Emails).
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_owned_email
ON "user" (COALESCE(clean_email, LOWER(email)))
WHERE email IS NOT NULL
AND (email_confirmed = TRUE OR password IS NOT NULL);
-- One Puter account per external identity. OIDCStore.link already assumes this
-- constraint exists — it catches the unique violation to tell "re-linking the
-- same account" apart from "this sub belongs to someone else" — but the table
-- never actually had it.
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_oidc_provider_sub
ON user_oidc_providers (provider, provider_sub);
@@ -0,0 +1,42 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Grow share from a pending-email-invite table into the index of active
-- shares. See sqlite/0067_share_entries.sql for the column rationale.
ALTER TABLE share
ADD COLUMN IF NOT EXISTS holder_user_id integer
REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
ADD COLUMN IF NOT EXISTS fsentry_id integer
REFERENCES fsentries (id) ON DELETE CASCADE ON UPDATE CASCADE,
ADD COLUMN IF NOT EXISTS mode varchar(20),
ADD COLUMN IF NOT EXISTS applied_at timestamp;
CREATE INDEX IF NOT EXISTS idx_share_holder
ON share (holder_user_id, id);
CREATE INDEX IF NOT EXISTS idx_share_fsentry
ON share (fsentry_id);
CREATE UNIQUE INDEX IF NOT EXISTS idx_share_holder_entry_issuer
ON share (holder_user_id, fsentry_id, issuer_user_id);
-- Retiring a deleted node's grants looks them up by permission text, and the
-- primary key is (issuer, holder, permission) — `permission` is never the
-- leading column, so the delete path would otherwise scan the table. The
-- text_pattern_ops class is what makes the left-anchored LIKE a range scan
-- under a non-C collation; plain equality still uses it.
CREATE INDEX IF NOT EXISTS idx_user_to_user_permissions_permission
ON user_to_user_permissions (permission text_pattern_ops);
@@ -0,0 +1,33 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- One user refusing contact from another. See sqlite/0068_user-block.sql for
-- the rationale. `created_at` is unix seconds.
--
-- Idempotent via IF NOT EXISTS.
CREATE TABLE IF NOT EXISTS user_block (
id BIGSERIAL PRIMARY KEY,
blocker_user_id integer NOT NULL
REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
blocked_user_id integer NOT NULL
REFERENCES "user" (id) ON DELETE CASCADE ON UPDATE CASCADE,
created_at BIGINT NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_block_pair
ON user_block (blocker_user_id, blocked_user_id);
@@ -0,0 +1,20 @@
-- Drops three groups that no code has ever read: `freeai`, `experimental` and
-- `dangerous`. No migration creates them — they were added by hand back when
-- hardcoded permissions were keyed by group name. That map is now a flat
-- per-user floor (`default_user_permissions` in data/hardcoded-permissions.js),
-- so a group nothing looks up grants nothing, and the rows are dead weight.
--
-- Guarded, and deliberately so: user_to_group_permissions.group_id and
-- jct_user_group.group_id both cascade on delete, so removing a group that
-- still carries permissions or members would silently revoke them from every
-- member. Only a group with neither is dropped. A group that survives this
-- migration has dependents and needs a deliberate decision, not a sweep --
-- query user_to_group_permissions by group_id to see what it holds.
DELETE FROM "group" g
WHERE g.extra ->> 'name' IN ('freeai', 'experimental', 'dangerous')
AND NOT EXISTS (
SELECT 1 FROM user_to_group_permissions p WHERE p.group_id = g.id
)
AND NOT EXISTS (
SELECT 1 FROM jct_user_group j WHERE j.group_id = g.id
);
@@ -0,0 +1,53 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Enforce "at most one account owns an email address".
--
-- `user.email` is deliberately not UNIQUE: several rows may legitimately hold
-- the same address while unconfirmed (admin-provisioned placeholders, signups
-- that were never confirmed, temp accounts on their way to becoming real). What
-- must never happen is two rows both *owning* an address — owning meaning the
-- row is confirmed, or holds a password and so can drive password recovery for
-- that inbox.
--
-- Signup, save-account, change-email, OIDC and admin provisioning each check for
-- an owner before writing, but a check and an insert are not one operation: two
-- requests can both read "free" and both write. This index is what actually
-- holds the invariant; the application checks just produce a nicer error most of
-- the time.
--
-- Matching is on the canonical address so provider aliases
-- (`[email protected]` vs `[email protected]`) collide. `clean_email` is
-- written on every modern write path; the COALESCE covers rows old enough to
-- predate the column.
--
-- If this CREATE fails, the DB already contains duplicate owners. Collapse them
-- first (admin → One-off Jobs → Collapse Duplicate Emails) — there is no safe
-- automatic merge of two accounts.
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_owned_email
ON user(COALESCE(clean_email, lower(email)))
WHERE email IS NOT NULL
AND (email_confirmed = 1 OR password IS NOT NULL);
-- One Puter account per external identity. OIDCStore.link already assumes this
-- constraint exists — it catches the unique violation to tell "re-linking the
-- same account" apart from "this sub belongs to someone else" — but the table
-- never actually had it, so two concurrent first-time logins could each create
-- an account and each link the same sub. Subsequent logins then resolved to
-- whichever row came back first.
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_oidc_provider_sub
ON user_oidc_providers(provider, provider_sub);
@@ -0,0 +1,60 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Grow `share` from a pending-email-invite table into the index of active
-- shares. Permissions stay the source of truth for access; this is what makes
-- shares listable and gives them a lifecycle.
--
-- - `holder_user_id` : NULL while an invite awaits signup.
-- - `fsentry_id` : the shared node. ON DELETE CASCADE retires the share
-- with the file, which permissions alone don't do.
-- - `mode` : see|list|read|write|manage. Unconstrained on purpose —
-- ACLService owns the mode set and adding one shouldn't
-- need a three-dialect migration.
-- - `applied_at` : set when an invite is claimed. Claiming updates the
-- row rather than deleting it, so the share stays
-- queryable.
ALTER TABLE `share` ADD COLUMN `holder_user_id` INTEGER DEFAULT NULL
REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE;
ALTER TABLE `share` ADD COLUMN `fsentry_id` INTEGER DEFAULT NULL
REFERENCES `fsentries` (`id`) ON DELETE CASCADE ON UPDATE CASCADE;
ALTER TABLE `share` ADD COLUMN `mode` TEXT DEFAULT NULL;
ALTER TABLE `share` ADD COLUMN `applied_at` TIMESTAMP DEFAULT NULL;
-- "Shared with me", keyset-paginated: ORDER BY ends in `id` as the tiebreaker.
CREATE INDEX IF NOT EXISTS `idx_share_holder`
ON `share` (`holder_user_id`, `id`);
-- Who has access to one node — also how an owner sees a manage-delegate's
-- re-grants, which the issuer/holder permission tables can't answer.
CREATE INDEX IF NOT EXISTS `idx_share_fsentry` ON `share` (`fsentry_id`);
-- One row per (holder, node, issuer). Pending invites have a NULL
-- holder_user_id and so aren't covered here; dedup for those belongs with the
-- invite flow.
CREATE UNIQUE INDEX IF NOT EXISTS `idx_share_holder_entry_issuer`
ON `share` (`holder_user_id`, `fsentry_id`, `issuer_user_id`);
-- Retiring a deleted node's grants looks them up by permission text
-- (`fs:<uuid>` and `fs:<uuid>:%`), and the subject only lives in that text —
-- no column, so no foreign key can cascade it. The table's primary key is
-- (issuer, holder, permission), which never puts `permission` first, so
-- without this the equality and the left-anchored LIKE both degrade to a full
-- scan on the delete path. With it, each is a range scan.
CREATE INDEX IF NOT EXISTS `idx_user_to_user_permissions_permission`
ON `user_to_user_permissions` (`permission`);
@@ -0,0 +1,32 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- One referral code, one account. The mysql and postgres baselines have
-- declared `user.referral_code` UNIQUE since the column was introduced; the
-- sqlite baseline only declared the column, so the invariant held on hosted
-- deployments and not on self-hosted ones.
--
-- Codes are minted by picking at random and writing, so this index is not a
-- nicety: it is what makes "did anyone already take this code" a decision the
-- database makes once, instead of a read the next mint can race. A caller that
-- loses the race sees a unique violation and picks again.
--
-- NULL is distinct from NULL in a sqlite unique index, so accounts that have
-- never asked for a code (the majority) are unaffected.
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_referral_code
ON user(referral_code)
WHERE referral_code IS NOT NULL;
@@ -0,0 +1,39 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- One user refusing contact from another. Sharing is what reads it today —
-- a blocked sender's share is refused outright, and their pending invite is
-- dropped when the blocker confirms the address it was aimed at — but the
-- table is deliberately not share-specific: the same answer serves anything
-- else one person can push at another.
--
-- Rows are read on the share path by exact (blocker, blocked) pair, which the
-- unique index answers as a point lookup; the same index's leading column
-- serves "who have I blocked". `created_at` is unix seconds, matching
-- `app_feedback`.
CREATE TABLE IF NOT EXISTS `user_block` (
"id" INTEGER PRIMARY KEY AUTOINCREMENT,
"blocker_user_id" INTEGER NOT NULL
REFERENCES `user` ("id") ON DELETE CASCADE ON UPDATE CASCADE,
"blocked_user_id" INTEGER NOT NULL
REFERENCES `user` ("id") ON DELETE CASCADE ON UPDATE CASCADE,
"created_at" INTEGER NOT NULL -- unix seconds
);
CREATE UNIQUE INDEX IF NOT EXISTS `idx_user_block_pair`
ON `user_block` (`blocker_user_id`, `blocked_user_id`);
@@ -0,0 +1,33 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Drops three groups that no code has ever read: `freeai`, `experimental` and
-- `dangerous`. No migration creates them — they were added by hand back when
-- hardcoded permissions were keyed by group name. That map is now a flat
-- per-user floor (`default_user_permissions` in data/hardcoded-permissions.js),
-- so a group nothing looks up grants nothing, and the rows are dead weight.
--
-- Guarded, and deliberately so: `user_to_group_permissions.group_id` and
-- `jct_user_group.group_id` both cascade on delete, so removing a group that
-- still carries permissions or members would silently revoke them from every
-- member. Only a group with neither is dropped. A group that survives this
-- migration has dependents and needs a deliberate decision, not a sweep --
-- query `user_to_group_permissions` by `group_id` to see what it holds.
DELETE FROM `group`
WHERE json_extract(`extra`, '$.name') IN ('freeai', 'experimental', 'dangerous')
AND `id` NOT IN (SELECT `group_id` FROM `user_to_group_permissions`)
AND `id` NOT IN (SELECT `group_id` FROM `jct_user_group`);
+39 -9
View File
@@ -3,18 +3,19 @@
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
* Puter is free software: you can redistribute it and/or modify it under the
* terms of the GNU Affero General Public License as published by the Free
* Software Foundation, either version 3 of the License, or (at your option) any
* later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
* details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
* along with this program. If not, see
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
*/
import { createServer, type Server } from 'node:http';
@@ -302,6 +303,35 @@ describe('DDBClient — batch writes', () => {
const stored = await client.query(TABLE, { pk: 'bulk' });
expect(stored.Items).toHaveLength(60);
});
it('reports no consumed capacity for an empty delete batch', async () => {
await expect(client.batchDel([])).resolves.toEqual({
ConsumedCapacity: [],
});
});
it('batch-deletes across chunks, leaving unrelated items alone', async () => {
const items = Array.from({ length: 30 }, (_, index) => ({
table: TABLE,
item: { pk: 'bulk-del', sk: `item-${index}`, index },
}));
await client.batchPut(items);
// Delete all but the last item — 29 keys still exercises chunking
// paths shared with batchPut while proving deletion is targeted.
const result = await client.batchDel(
items.slice(0, -1).map(({ table, item }) => ({
table,
key: { pk: item.pk, sk: item.sk },
})),
);
expect(result.ConsumedCapacity).toHaveLength(1);
expect(result.ConsumedCapacity[0].TableName).toBe(TABLE);
const stored = await client.query(TABLE, { pk: 'bulk-del' });
expect(stored.Items?.map((item) => item.sk)).toEqual(['item-29']);
});
});
describe('DDBClient — expired item sweep', () => {
+137 -31
View File
@@ -45,6 +45,11 @@ import { HttpError } from '../../core/http';
import type { IConfig, IDynamoConfig } from '../../types';
import { Span } from '../../util/span.js';
import { PuterClient } from '../types';
import {
clampStoredNumber,
isRepairableMarshallError,
repairForMarshall,
} from './marshallRepair.js';
const LOCAL_DYNAMO_MEMORY_PREFIX = ':memory:';
const localDynaliteEndpointPromises = new Map<string, Promise<string>>();
@@ -114,6 +119,49 @@ const sleep = async (ms: number) => {
await new Promise((resolve) => setTimeout(resolve, ms));
};
// One caller writing an out-of-range number usually keeps doing it, so the
// warning is per-target and throttled rather than one line per write.
const REPAIR_WARNING_INTERVAL_MS = 60_000;
const lastRepairWarningAt = new Map<string, number>();
const warnRepaired = (target: string, message: string): void => {
const now = Date.now();
const lastWarnedAt = lastRepairWarningAt.get(target) ?? 0;
if (now - lastWarnedAt < REPAIR_WARNING_INTERVAL_MS) return;
lastRepairWarningAt.set(target, now);
console.warn(
`[ddb] clamped an out-of-range value in ${target}: ${message}`,
);
};
/**
* Send a write, and if the payload was rejected while being encoded for a value
* the store cannot represent, clamp it and send it once more.
*
* Nothing is inspected on the way in: the payload of a successful write is
* never walked, so this costs one `try` on the hot path. The repair only runs
* on the failure, and only retries when it actually changed something —
* otherwise the original error stands.
*/
const sendRepairingValues = async <TPayload, TResult>(
payload: TPayload,
send: (payload: TPayload) => Promise<TResult>,
describe: () => string,
): Promise<TResult> => {
try {
return await send(payload);
} catch (error) {
if (!isRepairableMarshallError(error)) throw error;
const repaired = repairForMarshall(payload);
if (!repaired.changed) throw error;
warnRepaired(describe(), (error as Error).message);
return send(repaired.value as TPayload);
}
};
export class DDBClient extends PuterClient {
#documentClient: DynamoDBDocumentClient | null = null;
#localInitPromise: Promise<void> | null = null;
@@ -171,14 +219,19 @@ export class DDBClient extends PuterClient {
@Span('ddb.put', (table: string) => ({ 'db.table': table }))
async put<T extends Record<string, unknown>>(table: string, item: T) {
const command = new PutCommand({
TableName: table,
Item: item,
ReturnConsumedCapacity: 'TOTAL',
});
const client = await this.#getDocumentClient();
return client.send(command);
return sendRepairingValues(
item,
(itemToWrite) =>
client.send(
new PutCommand({
TableName: table,
Item: itemToWrite,
ReturnConsumedCapacity: 'TOTAL',
}),
),
() => `a put to ${table}`,
);
}
@Span('ddb.batchGet', (params: unknown[]) => ({
@@ -222,6 +275,37 @@ export class DDBClient extends PuterClient {
'db.batch_size': params.length,
}))
async batchPut(params: { table: string; item: Record<string, unknown> }[]) {
return this.#batchWrite(
params.map(({ table, item }) => ({
table,
request: { PutRequest: { Item: item } },
})),
);
}
@Span('ddb.batchDel', (params: unknown[]) => ({
'db.batch_size': params.length,
}))
async batchDel(params: { table: string; key: Record<string, unknown> }[]) {
return this.#batchWrite(
params.map(({ table, key }) => ({
table,
request: { DeleteRequest: { Key: key } },
})),
);
}
// Shared BatchWriteItem plumbing for batchPut/batchDel: 25-item chunks,
// UnprocessedItems retried with capped exponential backoff, consumed
// capacity accumulated per table across every request.
async #batchWrite(
params: {
table: string;
request: NonNullable<
BatchWriteCommandInput['RequestItems']
>[string][number];
}[],
) {
const consumedCapacityByTable = new Map<string, number>();
if (params.length === 0) {
return { ConsumedCapacity: [] };
@@ -258,11 +342,7 @@ export class DDBClient extends PuterClient {
let requestItems = chunk.reduce(
(acc, curr) => {
const tableRequests = acc[curr.table] ?? [];
tableRequests.push({
PutRequest: {
Item: curr.item,
},
});
tableRequests.push(curr.request);
acc[curr.table] = tableRequests;
return acc;
},
@@ -278,11 +358,17 @@ export class DDBClient extends PuterClient {
break;
}
const response = await client.send(
new BatchWriteCommand({
RequestItems: requestItems,
ReturnConsumedCapacity: 'TOTAL',
}),
const response = await sendRepairingValues(
requestItems,
(itemsToWrite) =>
client.send(
new BatchWriteCommand({
RequestItems: itemsToWrite,
ReturnConsumedCapacity: 'TOTAL',
}),
),
() =>
`a batch write to ${Object.keys(requestItems).join(', ')}`,
);
accumulateConsumedCapacity(
response.ConsumedCapacity as
@@ -328,11 +414,18 @@ export class DDBClient extends PuterClient {
}
@Span('ddb.del', (table: string) => ({ 'db.table': table }))
async del<T extends Record<string, unknown>>(table: string, key: T) {
async del<T extends Record<string, unknown>>(
table: string,
key: T,
opts?: { returnOld?: boolean },
) {
const command = new DeleteCommand({
TableName: table,
Key: key,
ReturnConsumedCapacity: 'TOTAL',
// ALL_OLD makes the delete an atomic claim: exactly one caller
// gets the attributes back.
...(opts?.returnOld ? { ReturnValues: 'ALL_OLD' as const } : {}),
});
const client = await this.#getDocumentClient();
@@ -423,20 +516,27 @@ export class DDBClient extends PuterClient {
!!expressionValues && Object.keys(expressionValues).length > 0;
const hasNames =
!!expressionNames && Object.keys(expressionNames).length > 0;
const command = new UpdateCommand({
TableName: table,
Key: key,
UpdateExpression: expression,
...(hasValues
? { ExpressionAttributeValues: expressionValues }
: {}),
...(hasNames ? { ExpressionAttributeNames: expressionNames } : {}),
ReturnValues: 'ALL_NEW',
ReturnConsumedCapacity: 'TOTAL',
});
const client = await this.#getDocumentClient();
return client.send(command);
return sendRepairingValues(
expressionValues,
(valuesToWrite) =>
client.send(
new UpdateCommand({
TableName: table,
Key: key,
UpdateExpression: expression,
...(hasValues
? { ExpressionAttributeValues: valuesToWrite }
: {}),
...(hasNames
? { ExpressionAttributeNames: expressionNames }
: {}),
ReturnValues: 'ALL_NEW',
ReturnConsumedCapacity: 'TOTAL',
}),
),
() => `an update to ${table}`,
);
}
async createTableIfNotExists(
@@ -538,6 +638,9 @@ export class DDBClient extends PuterClient {
marshallOptions: {
removeUndefinedValues: true,
},
unmarshallOptions: {
wrapNumbers: clampStoredNumber,
},
});
}
@@ -565,6 +668,9 @@ export class DDBClient extends PuterClient {
marshallOptions: {
removeUndefinedValues: true,
},
unmarshallOptions: {
wrapNumbers: clampStoredNumber,
},
});
}
@@ -0,0 +1,126 @@
import { describe, expect, it } from 'vitest';
import {
clampStoredNumber,
isRepairableMarshallError,
MAX_STORED_NUMBER,
MIN_STORED_NUMBER,
repairForMarshall,
} from './marshallRepair.ts';
describe('isRepairableMarshallError', () => {
it('recognizes the out-of-range and special-value failures', () => {
for (const message of [
'Number 1.6515584833071455e+55 is greater than Number.MAX_SAFE_INTEGER. Use NumberValue from @aws-sdk/lib-dynamodb.',
'Number -1e+55 is lesser than Number.MIN_SAFE_INTEGER. Use NumberValue from @aws-sdk/lib-dynamodb.',
'Special numeric value NaN is not allowed',
'Special numeric value Infinity is not allowed',
]) {
expect(isRepairableMarshallError(new Error(message))).toBe(true);
}
});
it('leaves every other failure alone', () => {
for (const error of [
new Error('Unsupported type passed: [object Symbol].'),
new Error('ValidationException'),
new Error(''),
undefined,
'not an error',
]) {
expect(isRepairableMarshallError(error)).toBe(false);
}
});
});
describe('repairForMarshall', () => {
it('clamps a number past the safe range in both directions', () => {
expect(repairForMarshall(1.6515584833071455e55)).toEqual({
value: MAX_STORED_NUMBER,
changed: true,
});
expect(repairForMarshall(-1e55)).toEqual({
value: MIN_STORED_NUMBER,
changed: true,
});
});
it('clamps infinities and nulls NaN', () => {
expect(repairForMarshall(Infinity).value).toBe(MAX_STORED_NUMBER);
expect(repairForMarshall(-Infinity).value).toBe(MIN_STORED_NUMBER);
expect(repairForMarshall(NaN)).toEqual({ value: null, changed: true });
});
it('repairs numbers nested in objects and arrays', () => {
const repaired = repairForMarshall({
profile: { netWorth: 1e55, level: 29 },
scores: [1, [2, -1e55]],
});
expect(repaired.changed).toBe(true);
expect(repaired.value).toEqual({
profile: { netWorth: MAX_STORED_NUMBER, level: 29 },
scores: [1, [2, MIN_STORED_NUMBER]],
});
});
it('returns the original payload untouched when nothing is out of range', () => {
const payload = {
a: MAX_STORED_NUMBER,
b: [1, 2, { c: 'three' }],
d: null,
};
const repaired = repairForMarshall(payload);
expect(repaired.changed).toBe(false);
expect(repaired.value).toBe(payload);
});
it('does not mutate the payload it repairs', () => {
const nested = { netWorth: 1e55 };
const payload = { nested };
repairForMarshall(payload);
expect(nested.netWorth).toBe(1e55);
});
it('keeps a `__proto__` key as data', () => {
const payload = JSON.parse('{"__proto__":{"big":1e55}}');
const repaired = repairForMarshall(payload) as {
value: Record<string, unknown>;
changed: boolean;
};
expect(repaired.changed).toBe(true);
expect(Object.getPrototypeOf(repaired.value)).toBe(Object.prototype);
expect(
Object.getOwnPropertyDescriptor(repaired.value, '__proto__')?.value,
).toEqual({ big: MAX_STORED_NUMBER });
});
it('leaves values it does not own alone', () => {
const date = new Date(0);
expect(repairForMarshall(date).value).toBe(date);
expect(repairForMarshall('str')).toEqual({
value: 'str',
changed: false,
});
});
});
describe('clampStoredNumber', () => {
it('decodes a number inside the range unchanged', () => {
expect(clampStoredNumber('42')).toBe(42);
expect(clampStoredNumber('-0.5')).toBe(-0.5);
expect(clampStoredNumber(String(MAX_STORED_NUMBER))).toBe(
MAX_STORED_NUMBER,
);
});
it('clamps a stored number past the range instead of decoding a BigInt', () => {
expect(clampStoredNumber('18014398509481982')).toBe(MAX_STORED_NUMBER);
expect(clampStoredNumber('-18014398509481982')).toBe(MIN_STORED_NUMBER);
expect(clampStoredNumber('1e400')).toBe(MAX_STORED_NUMBER);
});
});
@@ -0,0 +1,129 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* A stored number keeps its exact value only within the IEEE-754 safe integer
* range; a magnitude past this cannot round-trip, and the document client
* refuses to marshall one rather than write a value that reads back wrong.
*/
export const MAX_STORED_NUMBER = Number.MAX_SAFE_INTEGER;
export const MIN_STORED_NUMBER = Number.MIN_SAFE_INTEGER;
/**
* Marshalling failures a retry can fix by clamping: an out-of-range number, or
* a special numeric value (`NaN`, `±Infinity`) that has no representation at
* all. Matched on the message because they are plain `Error`s thrown while
* encoding the payload — no error name or code separates them from anything
* else, and they never reach the wire, so there is no response to inspect.
*/
const REPAIRABLE_MARSHALL_ERRORS = [
/is greater than Number\.MAX_SAFE_INTEGER/,
/is lesser than Number\.MIN_SAFE_INTEGER/,
/^Special numeric value /,
];
export const isRepairableMarshallError = (error: unknown): boolean => {
const message = (error as Error | undefined)?.message;
if (typeof message !== 'string') return false;
return REPAIRABLE_MARSHALL_ERRORS.some((pattern) => pattern.test(message));
};
/**
* The same bound, applied to what comes back. A stored number past the safe
* range — left by an older write, or reached by a counter incremented up to it
* — is otherwise decoded as a `BigInt`, which no JSON response can carry. This
* hands back the same kind of value a write accepts, and does less work per
* number than the default decoding it replaces.
*/
export const clampStoredNumber = (stored: string): number => {
const num = Number(stored);
if (num > MAX_STORED_NUMBER) return MAX_STORED_NUMBER;
if (num < MIN_STORED_NUMBER) return MIN_STORED_NUMBER;
return num;
};
export interface MarshallRepair {
value: unknown;
/** False when nothing in the payload was out of range — nothing to retry. */
changed: boolean;
}
const unchanged = (value: unknown): MarshallRepair => ({
value,
changed: false,
});
// Matches the shapes the marshaller walks into as a map: object literals and
// null-prototype objects, but not class instances (Date, Buffer, Set, …),
// which it encodes by their own rules.
const isPlainObject = (value: unknown): value is Record<string, unknown> =>
typeof value === 'object' &&
value !== null &&
(value.constructor === Object || value.constructor === undefined);
const repairNumber = (num: number): MarshallRepair => {
// No representation for these; JSON already turns them into null, so a
// caller that got one this far did not send it as JSON.
if (Number.isNaN(num)) return { value: null, changed: true };
if (num > MAX_STORED_NUMBER)
return { value: MAX_STORED_NUMBER, changed: true };
if (num < MIN_STORED_NUMBER)
return { value: MIN_STORED_NUMBER, changed: true };
return unchanged(num);
};
/**
* Clamp every number a payload holds into the range the store accepts,
* returning the original object untouched when nothing needed it — callers hold
* on to the values they pass (caches, retries), so a repair copies rather than
* mutating in place.
*
* Only run after a marshalling failure. Walking a payload this way costs as
* much as marshalling it, which is exactly why writes are not checked up
* front.
*/
export const repairForMarshall = (value: unknown): MarshallRepair => {
if (typeof value === 'number') return repairNumber(value);
if (Array.isArray(value)) {
let changed = false;
const repairedEntries = value.map((entry) => {
const repaired = repairForMarshall(entry);
changed = changed || repaired.changed;
return repaired.value;
});
return changed ? { value: repairedEntries, changed } : unchanged(value);
}
if (isPlainObject(value)) {
let changed = false;
const repairedEntries = Object.entries(value).map(([key, entry]) => {
const repaired = repairForMarshall(entry);
changed = changed || repaired.changed;
return [key, repaired.value] as const;
});
// `fromEntries` defines each property rather than assigning it, so a
// `__proto__` key in the payload stays data here.
return changed
? { value: Object.fromEntries(repairedEntries), changed }
: unchanged(value);
}
return unchanged(value);
};
+145 -5
View File
@@ -54,14 +54,22 @@ const sentMessage = (info: unknown) =>
};
describe('EmailClient — transport lifecycle', () => {
it('reports as configured once a transport is wired up', () => {
it('stops sending once the transport is shut down', async () => {
const client = startClient();
expect(client.isConfigured).toBe(true);
expect(
await client.sendRaw({ to: '[email protected]', subject: 'up', text: 'x' }),
).not.toBeNull();
client.onServerShutdown();
expect(client.isConfigured).toBe(false);
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
expect(
await client.sendRaw({ to: '[email protected]', subject: 'down', text: 'x' }),
).toBeNull();
warn.mockRestore();
});
it('warns and stays unconfigured when no transport is set', () => {
it('warns at boot when no transport is set', () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
const client = new EmailClient({
port: 0,
@@ -69,7 +77,6 @@ describe('EmailClient — transport lifecycle', () => {
} as unknown as IConfig);
client.onServerStart();
expect(client.isConfigured).toBe(false);
expect(warn).toHaveBeenCalledWith(
expect.stringContaining('no email transport configured'),
);
@@ -238,6 +245,139 @@ describe('EmailClient — template rendering', () => {
});
});
describe('EmailClient — share notification templates', () => {
/** Captures both parts of a send without touching the transport. */
const renderShare = async (
template: 'file_shared_with_you' | 'file_shared_invite',
values: Record<string, unknown>,
) => {
const client = startClient();
const captured: { html: string; text: string } = { html: '', text: '' };
vi.spyOn(client, 'sendRaw').mockImplementation(async (options) => {
captured.html = options.html ?? '';
captured.text = options.text ?? '';
return null;
});
await client.send('[email protected]', template, values);
return captured;
};
const HOLDER = {
recipient: 'alice',
subject_line: 'bob shared notes.md with you',
shares: [
{ sender: 'bob', what: 'notes.md' },
{ sender: 'carol', what: '3 items — a.txt, b.txt, +1 more' },
],
link: 'https://puter.test',
unsubscribe_uuid: null,
};
it('sends a plain-text alternative beside the html', async () => {
const { html, text } = await renderShare(
'file_shared_with_you',
HOLDER,
);
expect(html).toContain('<!DOCTYPE html>');
// Every sender reaches both parts, so a text-only client loses nothing.
for (const part of [html, text]) {
expect(part).toContain('bob');
expect(part).toContain('notes.md');
expect(part).toContain('carol');
expect(part).toContain('+1 more');
}
expect(text).not.toContain('<');
});
it('escapes item names in the html and leaves them raw in the text', async () => {
const { html, text } = await renderShare('file_shared_with_you', {
...HOLDER,
shares: [{ sender: 'bob', what: 'r&d "notes".md' }],
});
expect(html).toContain('r&amp;d &quot;notes&quot;.md');
expect(text).toContain('r&d "notes".md');
});
it('renders as a responsive single column with no remote assets', async () => {
const { html } = await renderShare('file_shared_with_you', HOLDER);
expect(html).toContain(
'<meta name="viewport" content="width=device-width, initial-scale=1" />',
);
expect(html).toContain('@media only screen and (max-width: 600px)');
expect(html).toContain('@media (prefers-color-scheme: dark)');
expect(html).toContain('max-width: 600px');
// Images are the one thing a client can refuse to load, so the design
// does without them — the layout can't depend on a blocked asset.
expect(html).not.toContain('<img');
// The colors have to survive a client that drops the stylesheet.
expect(html).toContain('style="background-color: #ffffff;');
});
it('opens with a preview line that is hidden in the body', async () => {
const { html } = await renderShare('file_shared_with_you', HOLDER);
const preheader = html.indexOf('Waiting for you under Shared with me.');
expect(preheader).toBeGreaterThan(-1);
expect(html.slice(0, preheader)).toContain('mso-hide: all');
expect(preheader).toBeLessThan(html.indexOf('Hi alice,'));
});
it('points the call to action at the app and nowhere else', async () => {
const { html, text } = await renderShare(
'file_shared_with_you',
HOLDER,
);
expect(html).toContain('href="https://puter.test"');
expect(html).toContain('>Open Puter</a>');
expect(text).toContain('Open Puter: https://puter.test');
});
it('separates senders without a rule above the first', async () => {
const { html } = await renderShare('file_shared_with_you', HOLDER);
// One rule between two senders, none leading the list.
expect(html.split('border-top: 1px solid').length - 1).toBe(1);
});
it('offers the unsubscribe link only to a recipient who has an account', async () => {
const withAccount = await renderShare('file_shared_with_you', {
...HOLDER,
unsubscribe_uuid: 'a-uuid',
});
expect(withAccount.html).toContain(
'href="https://puter.test/unsubscribe?user_uuid=a-uuid"',
);
expect(withAccount.text).toContain(
'https://puter.test/unsubscribe?user_uuid=a-uuid',
);
const anonymous = await renderShare('file_shared_with_you', HOLDER);
expect(anonymous.html).not.toContain('/unsubscribe');
expect(anonymous.text).not.toContain('/unsubscribe');
});
it('tells an invited address what to do with it', async () => {
const { html, text } = await renderShare('file_shared_invite', {
email: '[email protected]',
subject_line: 'bob shared notes.md with you on Puter',
shares: [{ sender: 'bob', what: 'notes.md' }],
link: 'https://puter.test',
});
for (const part of [html, text]) {
expect(part).toContain('[email protected]');
expect(part).toContain('Create your free account');
// An invite has no account to unsubscribe from.
expect(part).not.toContain('/unsubscribe');
}
expect(html).toContain('href="https://puter.test"');
});
});
describe('EmailClient.clean', () => {
const clean = (email: string) => startClient().clean(email);
+23 -8
View File
@@ -22,7 +22,11 @@ import handlebars, { template } from 'handlebars';
import nodemailer from 'nodemailer';
import type { IConfig } from '../../types';
import { PuterClient } from '../types';
import { EMAIL_TEMPLATES, type EmailTemplateName } from './templates';
import {
EMAIL_TEMPLATES,
type EmailTemplate,
type EmailTemplateName,
} from './templates';
/** Attachment shape passed through to the underlying transport. */
export interface EmailAttachment {
@@ -64,6 +68,7 @@ export type EmailValidator = (email: string) => Promise<boolean> | boolean;
interface CompiledTemplate {
subject: ReturnType<typeof template>;
html: ReturnType<typeof template>;
text?: ReturnType<typeof template>;
}
// -- Clean-email rules ------------------------------------------------
@@ -173,6 +178,7 @@ export class EmailClient extends PuterClient {
to,
subject: compiled.subject(values),
html: compiled.html(values),
...(compiled.text ? { text: compiled.text(values) } : {}),
...(options.replyTo ? { replyTo: options.replyTo } : {}),
});
}
@@ -183,7 +189,8 @@ export class EmailClient extends PuterClient {
*
* Returns the transport's send result, or `null` when no transport is
* configured (the send is a no-op in that case — callers that must not
* silently drop mail should check `isConfigured` first).
* silently drop mail should check `config.email` before composing, or the
* `null` return after).
*/
async sendRaw(options: SendMailOptions) {
if (!this.transport) {
@@ -199,11 +206,6 @@ export class EmailClient extends PuterClient {
});
}
/** Whether an SMTP transport is configured (sends are no-ops otherwise). */
get isConfigured(): boolean {
return this.transport !== null;
}
// -- Public API: clean / validate ---------------------------------
/**
@@ -289,7 +291,11 @@ export class EmailClient extends PuterClient {
}
private compileTemplates(): void {
for (const [name, template] of Object.entries(EMAIL_TEMPLATES)) {
// Widened to the interface: the literal map keeps a distinct type per
// template, and only some of them carry a `text` part.
const templates: Record<EmailTemplateName, EmailTemplate> =
EMAIL_TEMPLATES;
for (const [name, template] of Object.entries(templates)) {
this.compiledTemplates[name as EmailTemplateName] = {
// Subjects are plain-text headers: HTML-escaping would put
// literal entities in front of the recipient (&amp; etc.).
@@ -300,6 +306,15 @@ export class EmailClient extends PuterClient {
noEscape: true,
}),
html: handlebars.compile(dedent(template.html)),
// Same reasoning as the subject: a text part is not HTML, so
// escaping would show entities to the reader.
...(template.text
? {
text: handlebars.compile(dedent(template.text), {
noEscape: true,
}),
}
: {}),
};
}
}
+253 -24
View File
@@ -29,8 +29,184 @@
export interface EmailTemplate {
subject: string;
html: string;
/**
* Optional plain-text alternative. Where a template has one it goes out
* beside the HTML as multipart/alternative, which is what plain-text
* clients, screen readers and spam scoring all prefer to a machine
* down-conversion of the markup.
*/
text?: string;
}
// -- Shared layout ----------------------------------------------------
/**
* Colors, in one place so the two share templates can't drift apart. Every one
* of them is also written inline further down: clients that drop `<style>`
* (Gmail reading a non-Gmail account, Outlook.com) would otherwise render an
* unstyled page, so the stylesheet only carries what inline CSS can't express —
* the small-screen and dark-mode overrides.
*
* `ACCENT` is the GUI's primary blue darkened until white text on it clears
* 4.5:1, since a button label is small text however bold it is.
*/
const INK = '#101828';
const INK_SOFT = '#5b6472';
const INK_FAINT = '#667085';
const PAGE_BG = '#f2f4f7';
const CARD_BG = '#ffffff';
const CARD_BORDER = '#e4e7ec';
const RULE = '#e6e9ee';
const PANEL_BG = '#f8f9fb';
const PANEL_BORDER = '#eceff3';
const ACCENT = '#0a76cc';
const FONT =
"-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,'Helvetica Neue',Arial,sans-serif";
/** Pads the preview line so body text can't leak into it. */
const PREHEADER_FILL = '&#847;&zwnj;&nbsp;'.repeat(60);
/**
* The document both share notifications render into: one 600px column that goes
* full width on a phone, laid out with tables and `bgcolor` because Outlook
* renders through Word and ignores anything more modern.
*
* `preheader` is the line the inbox shows beside the subject — it should add to
* the subject, not repeat it. `content` is the card's rows; `footer` sits below
* the card, where the reason-for-receiving and unsubscribe belong.
*/
const shareEmailLayout = (parts: {
preheader: string;
content: string;
footer: string;
}): string => `
<!DOCTYPE html>
<html lang="en" dir="ltr" xmlns:o="urn:schemas-microsoft-com:office:office">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="x-apple-disable-message-reformatting" />
<meta name="color-scheme" content="light dark" />
<meta name="supported-color-schemes" content="light dark" />
<title>{{subject_line}}</title>
<!--[if mso]>
<noscript><xml><o:OfficeDocumentSettings><o:PixelsPerInch>96</o:PixelsPerInch></o:OfficeDocumentSettings></xml></noscript>
<![endif]-->
<style>
body { margin: 0; padding: 0; width: 100% !important; -webkit-text-size-adjust: 100%; -ms-text-size-adjust: 100%; }
table { border-spacing: 0; mso-table-lspace: 0pt; mso-table-rspace: 0pt; }
td { mso-line-height-rule: exactly; }
a[x-apple-data-detectors] { color: inherit !important; text-decoration: none !important; font-size: inherit !important; font-family: inherit !important; font-weight: inherit !important; line-height: inherit !important; }
@media only screen and (max-width: 600px) {
.col { width: 100% !important; }
.pad { padding-left: 24px !important; padding-right: 24px !important; }
.pad-y { padding-top: 28px !important; padding-bottom: 28px !important; }
.h1 { font-size: 22px !important; line-height: 30px !important; }
.btn { width: 100% !important; }
}
@media (prefers-color-scheme: dark) {
.page { background-color: #0e1116 !important; }
.card { background-color: #171b22 !important; border-color: #272d38 !important; }
.ink { color: #e9ebef !important; }
.ink-soft { color: #9aa3b2 !important; }
.ink-soft a { color: #9aa3b2 !important; }
.rule { border-color: #272d38 !important; }
.panel { background-color: #1d222a !important; border-color: #2b323d !important; }
.mark { color: #4da3f2 !important; }
}
</style>
</head>
<body class="page" style="margin: 0; padding: 0; width: 100%; background-color: ${PAGE_BG};">
<div style="display: none; font-size: 1px; line-height: 1px; max-height: 0; max-width: 0; opacity: 0; overflow: hidden; mso-hide: all;">${parts.preheader}${PREHEADER_FILL}</div>
<div role="article" aria-roledescription="email" lang="en">
<table role="presentation" class="page" width="100%" cellpadding="0" cellspacing="0" border="0" style="width: 100%; background-color: ${PAGE_BG};">
<tr>
<td align="center" style="padding: 32px 12px;">
<!--[if mso]><table role="presentation" width="600" cellpadding="0" cellspacing="0" border="0"><tr><td><![endif]-->
<table role="presentation" class="col" width="600" cellpadding="0" cellspacing="0" border="0" style="width: 100%; max-width: 600px; text-align: left;">
<tr>
<td class="pad mark" style="padding: 0 4px 14px; font-family: ${FONT}; font-size: 15px; line-height: 20px; font-weight: 700; letter-spacing: 0.01em; color: ${ACCENT};">Puter</td>
</tr>
<tr>
<td class="card" style="background-color: ${CARD_BG}; border: 1px solid ${CARD_BORDER}; border-radius: 14px;">
<table role="presentation" width="100%" cellpadding="0" cellspacing="0" border="0" style="width: 100%;">
<tr>
<td class="pad pad-y" style="padding: 36px 36px 34px;">
<table role="presentation" width="100%" cellpadding="0" cellspacing="0" border="0" style="width: 100%;">${parts.content}
</table>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="pad ink-soft" style="padding: 20px 4px 0; font-family: ${FONT}; font-size: 13px; line-height: 20px; color: ${INK_FAINT};">${parts.footer}
</td>
</tr>
</table>
<!--[if mso]></td></tr></table><![endif]-->
</td>
</tr>
</table>
</div>
</body>
</html>
`;
/** The greeting line above the heading. */
const greetingRow = (text: string): string => `
<tr>
<td class="ink-soft" style="padding: 0 0 6px; font-family: ${FONT}; font-size: 15px; line-height: 22px; color: ${INK_SOFT};">${text}</td>
</tr>`;
const headingRow = (text: string): string => `
<tr>
<td class="ink h1" style="padding: 0 0 18px; font-family: ${FONT}; font-size: 24px; line-height: 32px; font-weight: 700; letter-spacing: -0.01em; color: ${INK};">${text}</td>
</tr>`;
/**
* One row per sender, hairline-separated. The wording comes pre-composed from
* the digest (`digestLines`), so the list stays a list however many senders and
* items fold into it.
*/
const SHARE_LIST_ROW = `
<tr>
<td class="panel" style="padding: 6px 18px; background-color: ${PANEL_BG}; border: 1px solid ${PANEL_BORDER}; border-radius: 10px;">
<table role="presentation" width="100%" cellpadding="0" cellspacing="0" border="0" style="width: 100%;">
{{#each shares}}
<tr>
<td class="ink rule" style="padding: 12px 0;{{#unless @first}} border-top: 1px solid ${RULE};{{/unless}} font-family: ${FONT}; font-size: 16px; line-height: 24px; color: ${INK};"><strong style="font-weight: 600;">{{this.sender}}</strong> shared {{this.what}}</td>
</tr>
{{/each}}
</table>
</td>
</tr>`;
/** A body paragraph. `padding` lets a caller tune the rhythm around it. */
const textRow = (text: string, padding = '18px 0 0'): string => `
<tr>
<td class="ink-soft" style="padding: ${padding}; font-family: ${FONT}; font-size: 15px; line-height: 23px; color: ${INK_SOFT};">${text}</td>
</tr>`;
/**
* The call to action. Padding sits on the cell and the color on `bgcolor` so
* Outlook still draws a real button (square-cornered, which is fine); the table
* goes full width under 600px so the tap target spans the card.
*/
const buttonRow = (label: string): string => `
<tr>
<td style="padding: 26px 0 0;">
<table role="presentation" class="btn" cellpadding="0" cellspacing="0" border="0" style="border-collapse: separate;">
<tr>
<td align="center" bgcolor="${ACCENT}" style="background-color: ${ACCENT}; border-radius: 10px; padding: 14px 26px;">
<a href="{{link}}" style="display: inline-block; font-family: ${FONT}; font-size: 16px; line-height: 20px; font-weight: 600; color: #ffffff; text-decoration: none;">${label}</a>
</td>
</tr>
</table>
</td>
</tr>`;
export const EMAIL_TEMPLATES = {
'approved-for-listing': {
subject: '🎉 Your app has been approved for listing!',
@@ -183,32 +359,85 @@ immediately</p>
<p>Puter</p>
`,
},
share_by_username: {
subject: 'Puter share from {{susername}}',
html: `
<p>Hi there {{rusername}},</p>
<p>You've received a share from {{susername}} on Puter.</p>
<p>Go to puter.com to check it out.</p>
{{#if message}}
<p>The following message was included:</p>
<blockquote>{{message}}</blockquote>
{{/if}}
<p>Sincerely,</p>
<p>Puter</p>
/**
* A digest: shares to one recipient are held briefly and merged, so
* `shares` may carry several senders. The subject is composed by the
* service (see `digestSubject`), which owns the grouped wording.
*/
file_shared_with_you: {
subject: '{{subject_line}}',
html: shareEmailLayout({
preheader: 'Waiting for you under Shared with me.',
content:
greetingRow('Hi{{#if recipient}} {{recipient}}{{/if}},') +
headingRow('Shared with you') +
SHARE_LIST_ROW +
buttonRow('Open Puter') +
textRow(
'Shared items live under <span style="font-weight: 600;">Shared</span> in your files.',
'24px 0 0',
),
footer: `You're receiving this because someone shared with your Puter account.{{#if unsubscribe_uuid}}
<br /><a href="{{link}}/unsubscribe?user_uuid={{unsubscribe_uuid}}" style="color: inherit; text-decoration: underline;">Unsubscribe from notification emails</a>{{/if}}`,
}),
text: `
Hi{{#if recipient}} {{recipient}}{{/if}},
Shared with you on Puter:
{{#each shares}}
- {{this.sender}} shared {{this.what}}
{{/each}}
Open Puter: {{link}}
Shared items live under "Shared" in your files.
--
You're receiving this because someone shared with your Puter account.
{{#if unsubscribe_uuid}}Unsubscribe from notification emails:
{{link}}/unsubscribe?user_uuid={{unsubscribe_uuid}}{{/if}}
`,
},
share_by_email: {
subject: 'share by email',
html: `
<p>Hi there,</p>
<p>You've received a share from {{sender_name}} on Puter:</p>
<p><a href="{{link}}">{{link}}</a></p>
{{#if message}}
<p>The following message was included:</p>
<blockquote>{{message}}</blockquote>
{{/if}}
<p>Sincerely,</p>
<p>Puter</p>
// The only way to reach someone with no account. Same digest shape.
file_shared_invite: {
subject: '{{subject_line}}',
html: shareEmailLayout({
preheader: 'Claim it with a free Puter account.',
content:
greetingRow('Hi there,') +
headingRow('Shared with you on Puter') +
SHARE_LIST_ROW +
textRow(
'There\'s no Puter account for <span style="font-weight: 600;">{{email}}</span> yet. Create one with this address, confirm it, and everything above will be waiting for you. It\'s free and takes about a minute.',
) +
buttonRow('Create your free account') +
textRow(
'Already on Puter? Add {{email}} to your account and confirm it to get the same access.',
'24px 0 0',
),
footer: `You're receiving this because someone shared with {{email}}. Nothing is shared until the address is confirmed, so you can ignore this email and nothing happens.`,
}),
text: `
Hi there,
Shared with you on Puter:
{{#each shares}}
- {{this.sender}} shared {{this.what}}
{{/each}}
There's no Puter account for {{email}} yet. Create one with this
address, confirm it, and everything above will be waiting for you.
It's free and takes about a minute.
Create your free account: {{link}}
Already on Puter? Add {{email}} to your account and confirm it to get
the same access.
--
You're receiving this because someone shared with {{email}}. Nothing is
shared until the address is confirmed, so you can ignore this email
and nothing happens.
`,
},
} satisfies Record<string, EmailTemplate>;
+79 -8
View File
@@ -32,6 +32,24 @@ type GuiEvent<R = Record<string, unknown>> = {
response: R;
};
/**
* Extension-augmentable half of {@link EventMap}. Extensions that emit their own
* events declare the payload here by declaration merging, so both the emitter
* and every listener are typed against the same shape:
*
* declare module '@heyputer/backend/clients/event/types' {
* interface IExtensionEventMap {
* 'my.thing.happened': { thingId: string };
* }
* }
*
* Deliberately member-less and index-signature-free: an index signature here
* would widen `keyof EventMap` to `string` and silently disable key checking on
* every `emit` in the tree.
*/
// eslint-disable-next-line @typescript-eslint/no-empty-object-type
export interface IExtensionEventMap {}
export type EventMap = {
// ---- Server lifecycle ----
serverStart: Record<string, never>;
@@ -142,6 +160,19 @@ export type EventMap = {
'puter.signup.validate': {
allow: boolean;
email?: string;
/**
* The address in the same canonical form the `email.validate` hook was
* given (`cleanEmail`), so a handler can correlate the two hooks on one
* key. Aliases (`[email protected]`, `[email protected]`) differ from
* `email` here.
*/
clean_email?: string;
/**
* True for a temp (frictionless, no-password) signup. Those carry a
* synthetic `<username>@gmail.com` and never reach email validation, so
* a handler must not draw conclusions from `email`.
*/
is_temp?: boolean;
ip?: string | null;
source?: 'oidc';
req?: unknown;
@@ -290,7 +321,17 @@ export type EventMap = {
sourceObjectKey: string;
copyObjectKey: string;
};
'fs.move.node': { node: FSEntry; fromPath: string; toPath: string };
/**
* `fromUserId` is the owner the node had before the move. A move into
* someone else's tree re-owns the row, and `node` already carries the new
* owner — listeners that care about the hand-over need the old one.
*/
'fs.move.node': {
node: FSEntry;
fromPath: string;
toPath: string;
fromUserId?: number;
};
'fs.remove.node': { node: FSEntry; entry: FSEntry; target: FSEntry };
'fs.write.file': { node: FSEntry; entry: FSEntry; target: FSEntry };
'fs.storage.upload-progress': {
@@ -338,12 +379,39 @@ export type EventMap = {
// is pricing workers that came into existence, not deploys.
'worker.create': { actor: Actor; workerName: string };
// The same announcement for worker code that lives inside a hosted site
// and is deployed on demand rather than by an explicit call. Such a worker
// has no row of ours to key on, so what stands in for "genuinely new" is
// the absence of any prior record that it ran — which is why it is
// announced separately, and identified by where its source lives rather
// than by a worker name.
'worker.dynamic.create': {
actor: Actor;
subdomain: string;
worker: string;
};
// ---- Outer / GUI broadcast ----
'outer.cacheUpdate': {
cacheKey: string[];
data?: unknown;
ttlSeconds?: number;
};
/**
* Permission cache generations were bumped, so peer regions must bump their
* own — the counter is per-cluster, so a local bump says nothing to them.
* Carries the actors, not the values: the numbers only have to change.
*/
'outer.permission.generationBumped': { actorUids: string[] };
/**
* Flat permission entries were deleted. Grant-path flat entries carry no
* expiry, so without this a revoke never lands in a peer region whose KV
* table isn't replicated. Revoke-only: a grant that fails to replicate just
* denies there, which is the safe direction.
*/
'outer.permission.flatInvalidated': {
entries: Array<{ holderUserId: number; permission: string }>;
};
'outer.fs.write-hash': { hash: string; uuid: string };
/**
* Cache keys the KV read cache must stop serving, because the entries
@@ -367,8 +435,10 @@ export type EventMap = {
}>;
// ---- Subdomains ----
'subdomain.delete': { subdomain: string };
'subdomain.update': { subdomain: string };
// `uid` is the row uuid — for `delete` it is the only surviving handle a
// listener can use to find state keyed to the row after it is gone.
'subdomain.delete': { subdomain: string; uid?: string };
'subdomain.update': { subdomain: string; uid?: string };
'site.htmlServed': {
subdomain: string;
entry: unknown;
@@ -460,7 +530,7 @@ export type EventMap = {
* have all dropped that answer.
*/
'outer.pubsub.metering.credits-changed': { userUuid: string };
};
} & IExtensionEventMap;
/**
* Phase of a request/method lifecycle. `reject` is emitted when a `before`
@@ -540,10 +610,11 @@ export type EventKey = keyof EventMap & string;
// Generates a wildcard for every non-final dot-separated prefix of K.
export type WildcardPrefixes<K extends string> =
K extends `${infer Head}.${infer Tail}`
? | `${Head}.*`
| (Tail extends `${string}.${string}`
? `${Head}.${WildcardPrefixes<Tail>}`
: never)
?
| `${Head}.*`
| (Tail extends `${string}.${string}`
? `${Head}.${WildcardPrefixes<Tail>}`
: never)
: never;
export type ListenKey = EventKey | WildcardPrefixes<EventKey>;
@@ -30,7 +30,7 @@ describe('PreludeClient', () => {
expect(makeClient().isConfigured()).toBe(false);
});
describe('isCountrySupported (€0.07 cap)', () => {
describe('isCountrySupported (€0.20 cap)', () => {
const client = makeClient('sk_test');
it('allows revenue markets up to the cap (incl. the priciest)', () => {
@@ -40,6 +40,17 @@ describe('PreludeClient', () => {
expect(client.isCountrySupported('us')).toBe(true); // case-insensitive
});
it('allows the markets the old €0.07 cap excluded', () => {
// Raising the cap to €0.20 brought ~89 countries into the phone
// gate. These pin the new range so a future change to the default
// has to say so out loud.
expect(client.isCountrySupported('UA')).toBe(true); // €0.0940
expect(client.isCountrySupported('PH')).toBe(true); // €0.1237
expect(client.isCountrySupported('EG')).toBe(true); // €0.1561
expect(client.isCountrySupported('NG')).toBe(true); // €0.1980
expect(client.isCountrySupported('WS')).toBe(true); // €0.2000, at the cap
});
it('rejects countries above the cap, with no SMS, or unknown', () => {
expect(client.isCountrySupported('PK')).toBe(false); // €0.3548
expect(client.isCountrySupported('ID')).toBe(false); // €0.2430
@@ -63,9 +74,7 @@ describe('PreludeClient', () => {
});
it('createVerification POSTs the phone target + ip signal with bearer auth', async () => {
fetchMock.mockResolvedValue(
okJson({ id: 'vrf_1', status: 'success' }),
);
fetchMock.mockResolvedValue(okJson({ id: 'vrf_1', status: 'success' }));
const client = makeClient('sk_test');
const res = await client.createVerification('+14155550123', {
@@ -81,7 +90,11 @@ describe('PreludeClient', () => {
target: { type: 'phone_number', value: '+14155550123' },
// Defaults to RCS (cheaper); Prelude falls back to SMS. locale is
// hardcoded to en-US so the message text is always English.
options: { code_size: 6, preferred_channel: 'rcs', locale: 'en-US' },
options: {
code_size: 6,
preferred_channel: 'rcs',
locale: 'en-US',
},
signals: { ip: '203.0.113.7' },
});
});
@@ -196,9 +209,9 @@ describe('PreludeClient', () => {
it('throws (does not call fetch) when not configured', async () => {
const client = makeClient();
await expect(
client.createVerification('+14155550123'),
).rejects.toThrow(/not configured/i);
await expect(client.createVerification('+14155550123')).rejects.toThrow(
/not configured/i,
);
expect(fetchMock).not.toHaveBeenCalled();
});
+8 -5
View File
@@ -50,12 +50,15 @@ export type PreludeChannel =
export type PreludeDeliveryChannel = PreludeChannel | 'silent' | 'voice';
/**
* Default per-SMS cost ceiling (EUR). Countries whose Prelude SMS rate exceeds
* this — or that have no SMS channel — are not offered phone verification. The
* cap covers every realistic revenue market (priciest are Germany €0.0598 and
* Saudi Arabia €0.0638) while excluding the expensive, high-fraud long tail.
* Override per-deployment with `config.prelude.maxSmsCostEur`.
* this — or that have no SMS channel — are not offered phone verification.
* Raised from €0.07 to €0.20: the old cap sat only marginally above the
* priciest revenue markets (Germany €0.0598, Saudi Arabia €0.0638), so any rate
* drift silently withdrew the phone gate from a real market — and the abuse
* harness now routes far more legitimate signups into the SMS band, where an
* unavailable gate is a dead end rather than a mild inconvenience. Override
* per-deployment with `config.prelude.maxSmsCostEur`.
*/
const DEFAULT_MAX_SMS_COST_EUR = 0.07;
const DEFAULT_MAX_SMS_COST_EUR = 0.2;
/** Status returned by Prelude when creating/retrying a verification. */
export type PreludeCreateStatus =
@@ -38,7 +38,11 @@ import type { Actor } from '../../core/actor.js';
import { Context, runWithContext } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { requireUserActorGate } from '../../core/http/middleware/gates.js';
import type { TokenSource } from '../../core/http/types.js';
import {
ROUTES_METADATA_KEY,
type CollectedRoute,
type TokenSource,
} from '../../core/http/types.js';
import { PuterServer } from '../../server.js';
import { FULL_API_ACCESS } from '../../services/permission/consts.js';
import { setupTestServer } from '../../testUtil.js';
@@ -346,6 +350,188 @@ describe('puter.signup.validate event', () => {
// ── Signup flow ─────────────────────────────────────────────────────
// ── Concurrent claims on one address ────────────────────────────────
//
// The duplicate checks these flows run are not atomic with the writes that
// follow them — the validate hook and bcrypt sit in between, and both are slow
// enough for a second request to pass the same check. These tests fire the
// requests together and assert the address still ends up on exactly one row.
describe('concurrent claims on one email address', () => {
const uniq = () => Math.random().toString(36).slice(2, 10);
const countOwners = async (email: string): Promise<number> => {
// `email_confirmed` is TINYINT on MySQL and BOOLEAN on Postgres, so the
// literal has to come from the client rather than be hardcoded.
const isTrue = server.clients.db.booleanLiteral(true);
const rows = (await server.clients.db.read(
`SELECT COUNT(*) AS n FROM \`user\` WHERE \`email\` = ? AND (\`email_confirmed\` = ${isTrue} OR \`password\` IS NOT NULL)`,
[email],
)) as Array<{ n: number }>;
return Number(rows[0]?.n ?? 0);
};
it('lets only one of two simultaneous signups take the address', async () => {
const email = `race-${uniq()}@test.local`;
const signup = (username: string) =>
controller.handleSignup(
makeReq({ username, email, password: 'correct-horse-battery' }),
makeRes(),
);
// Distinct usernames on purpose: the UNIQUE on `username` would
// otherwise be what rejects the second request, and the email race
// would go untested.
const results = await Promise.allSettled([
signup(`r_a_${uniq()}`),
signup(`r_b_${uniq()}`),
]);
expect(results.filter((r) => r.status === 'fulfilled')).toHaveLength(1);
expect(await countOwners(email)).toBe(1);
const rejected = results.find((r) => r.status === 'rejected') as
| PromiseRejectedResult
| undefined;
expect(rejected?.reason).toMatchObject({ statusCode: 400 });
});
it('lets only one of many simultaneous signups take the address', async () => {
const email = `race-many-${uniq()}@test.local`;
const results = await Promise.allSettled(
Array.from({ length: 5 }, () =>
controller.handleSignup(
makeReq({
username: `r_m_${uniq()}`,
email,
password: 'correct-horse-battery',
}),
makeRes(),
),
),
);
expect(results.filter((r) => r.status === 'fulfilled')).toHaveLength(1);
expect(await countOwners(email)).toBe(1);
});
it('rejects a signup racing an admin-provisioned placeholder claim', async () => {
const email = `race-pseudo-${uniq()}@test.local`;
// The placeholder shape signup is allowed to convert: unconfirmed,
// no password. Two signups both see it as claimable.
await server.stores.user.create({
username: `r_p_${uniq()}`,
uuid: uuidv4(),
password: null,
email,
clean_email: email,
});
const results = await Promise.allSettled([
controller.handleSignup(
makeReq({
username: `r_p_a_${uniq()}`,
email,
password: 'correct-horse-battery',
}),
makeRes(),
),
controller.handleSignup(
makeReq({
username: `r_p_b_${uniq()}`,
email,
password: 'correct-horse-battery',
}),
makeRes(),
),
]);
expect(results.filter((r) => r.status === 'fulfilled')).toHaveLength(1);
expect(await countOwners(email)).toBe(1);
});
it('reports a duplicate address as a 400, not a constraint error', async () => {
const email = `dupe-${uniq()}@test.local`;
await controller.handleSignup(
makeReq({
username: `d_a_${uniq()}`,
email,
password: 'correct-horse-battery',
}),
makeRes(),
);
// The message has to be the one the pre-check produces — a user who
// loses the race should not be able to tell.
await expect(
controller.handleSignup(
makeReq({
username: `d_b_${uniq()}`,
email,
password: 'correct-horse-battery',
}),
makeRes(),
),
).rejects.toMatchObject({
statusCode: 400,
message:
'This email already exists in our database. Please use another one.',
});
});
it('refuses to give a placeholder row a password for a taken address', async () => {
const email = `recover-${uniq()}@test.local`;
await controller.handleSignup(
makeReq({
username: `rec_own_${uniq()}`,
email,
password: 'correct-horse-battery',
}),
makeRes(),
);
// An unconfirmed, password-less row is allowed to sit on the same
// address — but password recovery accepts a username, so it can be
// driven for this row rather than for the account that owns the
// address. Setting a password here would make it a second account able
// to recover that inbox.
const placeholderName = `rec_ph_${uniq()}`;
const placeholder = await server.stores.user.create({
username: placeholderName,
uuid: uuidv4(),
password: null,
email,
clean_email: email,
});
const token = uuidv4();
await server.stores.user.update(placeholder.id, {
pass_recovery_token: token,
});
const jwt = server.services.token.sign(
'otp',
{
token,
user_uid: placeholder.uuid,
email,
purpose: 'pass-recovery',
},
{ expiresIn: '1h' },
);
await expect(
controller.handleSetPassUsingToken(
makeReq({ token: jwt, password: 'another-strong-password' }),
makeRes(),
),
).rejects.toMatchObject({
statusCode: 400,
message:
'This email is already in use. Recover the account that uses it instead.',
});
});
});
describe('AuthController.handleSignup', () => {
const uniq = () => Math.random().toString(36).slice(2, 10);
@@ -1499,59 +1685,6 @@ describe('AuthController grant flows', () => {
} as Actor;
});
it('grant-user-user: rejects missing target_username/permission with 400', async () => {
await expect(
controller.handleGrantUserUser(
makeReq({ permission: 'fs:read' }, { actor: issuerActor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('grant-user-user: persists the permission and PermissionService.check sees it', async () => {
const permission = `service:test-grant-${uuidv4()}:ii:read`;
// The controller calls PermissionService.grantUserUserPermission,
// which gates on `manage:<permission>` for non-system actors. Pre-
// bootstrap the manage flag directly through the permission store
// (the system actor would skip this gate, but its in-memory shape
// has no user.id, so it can't issue grants). Then the controller
// call exercises persist + check end-to-end.
await server.stores.permission.setFlatUserPerm(
issuer.id,
`manage:${permission}`,
{
permission: `manage:${permission}`,
deleted: false,
issuer_user_id: issuer.id,
} as never,
);
const res = makeRes();
await inCtx(issuerActor, () =>
controller.handleGrantUserUser(
makeReq(
{
target_username: target.username,
permission,
extra: { reason: 'unit-test' },
},
{ actor: issuerActor },
),
res,
),
);
expect(res.body).toEqual({});
// The target now sees the permission via the user-to-user grant.
const targetActor = {
user: { ...target, email_confirmed: true },
} as Actor;
const granted = await server.services.permission
.check(targetActor, permission)
.catch(() => false);
expect(granted).toBeTruthy();
});
it('grant-user-app: persists a user→app permission grant', async () => {
// Create an app row owned by the issuer so the grant has somewhere
// to land.
@@ -2039,20 +2172,6 @@ describe('AuthController grant flows', () => {
expect(await storedPermissions()).not.toContain(NOTHING);
});
it('grant-user-group: 404 when the group does not exist', async () => {
await expect(
controller.handleGrantUserGroup(
makeReq(
{
group_uid: `does-not-exist-${uuidv4()}`,
permission: 'service:foo:ii:read',
},
{ actor: issuerActor },
),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 404 });
});
});
// ── Token grants: get-user-app-token / check-app ───────────────────
@@ -3235,8 +3354,10 @@ describe('AuthController.handleConfirmPhone', () => {
});
describe('AuthController.handleCardVerificationSetup', () => {
it('short-circuits to verified when the gate is not set', async () => {
const { actor } = await makeUserAndActor();
it('short-circuits to verified for an account with a card on file', async () => {
const { actor } = await makeUserAndActor({
card_fingerprint: 'fp_existing',
});
const res = makeRes();
await controller.handleCardVerificationSetup(
makeReq({}, { actor }),
@@ -3245,6 +3366,21 @@ describe('AuthController.handleCardVerificationSetup', () => {
expect(res.body).toMatchObject({ card_verified: true });
});
it('starts the flow for an account that was never asked and never verified', async () => {
// A route requiring a verified card (`requireCardVerified`) sends a
// user here with the gate clear and no card on file. Answering
// "already verified" on the flag alone would bounce them between a
// dialog reporting success and a route that keeps refusing; reaching
// the extension (absent here, hence 503) is the flow actually running.
const { actor } = await makeUserAndActor();
await expect(
controller.handleCardVerificationSetup(
makeReq({}, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 503 });
});
it('throws 403 when the account is suspended', async () => {
const { actor } = await makeUserAndActor({
requires_card_verification: 1,
@@ -3345,8 +3481,10 @@ describe('AuthController.handleCardVerificationConfirm', () => {
}
});
it('short-circuits to verified when the gate is not set', async () => {
const { actor } = await makeUserAndActor();
it('short-circuits to verified for an account with a card on file', async () => {
const { actor } = await makeUserAndActor({
card_fingerprint: 'fp_existing',
});
const res = makeRes();
await controller.handleCardVerificationConfirm(
makeReq({ setup_intent_id: 'seti_1' }, { actor }),
@@ -3355,6 +3493,18 @@ describe('AuthController.handleCardVerificationConfirm', () => {
expect(res.body).toMatchObject({ card_verified: true });
});
it('runs the confirm for an account that was never asked and never verified', async () => {
// Same reason as setup: the pending flag says nobody asked, not that
// a card was checked, so a voluntary verification has to go through.
const { actor } = await makeUserAndActor();
await expect(
controller.handleCardVerificationConfirm(
makeReq({ setup_intent_id: 'seti_1' }, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 503 });
});
it('throws 409 when phone verification must be completed first', async () => {
const { actor } = await makeUserAndActor({
requires_card_verification: 1,
@@ -3847,6 +3997,151 @@ describe('AuthController.handleConfirmEmail', () => {
});
expect(after!.email_confirmed).toBeFalsy();
});
it('strips a rival placeholder before confirming, not after', async () => {
const { user, actor } = await makeUserAndActor();
const email = user.email as string;
// A placeholder sitting on the same address. Confirming first and
// demoting second would momentarily leave two rows owning it, which
// the unique index rejects — turning a legitimate confirmation into a
// 500.
const rival = await server.stores.user.create({
username: `rival_${Math.random().toString(36).slice(2, 10)}`,
uuid: uuidv4(),
password: null,
email,
clean_email: email,
});
const refreshed = await server.stores.user.getById(user.id, {
force: true,
});
const res = makeRes();
await controller.handleConfirmEmail(
makeReq({ code: refreshed!.email_confirm_code! }, { actor }),
res,
);
expect(res.body).toMatchObject({ email_confirmed: true });
const confirmed = await server.stores.user.getById(user.id, {
force: true,
});
expect(confirmed!.email_confirmed).toBe(true);
const strippedRival = await server.stores.user.getById(rival.id, {
force: true,
});
expect(strippedRival!.email).toBeNull();
});
it('refuses when another account already confirmed the address', async () => {
const email = `owned-${uniq()}@test.local`;
// Confirmed with no password is the shape an identity provider
// creates, and it is what the old rival check (which also demanded a
// password) let through. Demoting it would take the address off an
// account that proved it owns the inbox.
const owner = await server.stores.user.create({
username: `owner_${uniq()}`,
uuid: uuidv4(),
password: null,
email,
clean_email: email,
email_confirmed: true,
});
// A second, unconfirmed row on the same address holding a confirm
// code — the legacy duplicate this path used to resolve in its favour.
const claimant = await server.stores.user.create({
username: `claim_${uniq()}`,
uuid: uuidv4(),
password: null,
email,
clean_email: email,
email_confirm_code: '123456',
});
const actor = {
user: {
id: claimant.id,
uuid: claimant.uuid,
username: claimant.username,
email,
email_confirmed: false,
},
} as Actor;
await expect(
controller.handleConfirmEmail(
makeReq({ code: '123456' }, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
const untouched = await server.stores.user.getById(owner.id, {
force: true,
});
expect(untouched!.email).toBe(email);
expect(untouched!.email_confirmed).toBe(true);
const stillUnconfirmed = await server.stores.user.getById(claimant.id, {
force: true,
});
expect(stillUnconfirmed!.email_confirmed).toBeFalsy();
});
});
describe('AuthController.handleSaveAccount address conflicts', () => {
it('refuses to promote a temp account onto a taken address', async () => {
const email = `save-${Math.random().toString(36).slice(2, 10)}@test.local`;
await controller.handleSignup(
makeReq({
username: `save_own_${Math.random().toString(36).slice(2, 10)}`,
email,
password: 'correct-horse-battery',
}),
makeRes(),
);
const tempRes = makeRes();
await controller.handleSignup(makeReq({ is_temp: true }), tempRes);
const tempUser = (
tempRes.body as { user: { username: string; uuid: string } }
).user;
const tempRow = await server.stores.user.getByUuid(tempUser.uuid);
const actor = {
user: {
id: tempRow!.id,
uuid: tempRow!.uuid,
username: tempRow!.username,
email: null,
email_confirmed: false,
},
} as Actor;
await expect(
controller.handleSaveAccount(
makeReq(
{
username: `save_new_${Math.random().toString(36).slice(2, 10)}`,
email,
password: 'another-strong-password',
},
{ actor },
),
makeRes(),
),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'email_already_in_use',
});
// The temp row must be left alone — a failed promotion that already
// wrote the username would strand the account half-converted.
const untouched = await server.stores.user.getById(tempRow!.id, {
force: true,
});
expect(untouched!.email).toBeNull();
expect(untouched!.password).toBeNull();
});
});
// ── Password recovery flow ──────────────────────────────────────────
@@ -4453,16 +4748,24 @@ describe('AuthController permission revokes', () => {
).rejects.toMatchObject({ statusCode: 400 });
});
it('revoke-user-group: 400 on missing group_uid/permission', async () => {
const { actor } = await makeUserAndActor();
await expect(
controller.handleRevokeUserGroup(
makeReq({ permission: 'fs:read' }, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
// No in-tree caller, so a later cleanup would read it as dead: pin the gate.
it('revoke-user-user: stays registered as a user-actor route', () => {
const proto = Object.getPrototypeOf(controller) as Record<
string,
CollectedRoute[] | undefined
>;
const route = (proto[ROUTES_METADATA_KEY] ?? []).find(
(r) => r.path === '/auth/revoke-user-user',
);
expect(route, '/auth/revoke-user-user is not registered').toBeDefined();
expect(route!.method).toBe('post');
expect(route!.options).toMatchObject({
subdomain: 'api',
requireUserActor: true,
});
});
// Why the deprecated route exists: older grants stay withdrawable.
it('revoke-user-user: round-trips a grant + revoke without throwing', async () => {
const { actor: issuerActor, user: issuer } = await makeUserAndActor();
const { user: target } = await makeUserAndActor();
@@ -4476,24 +4779,17 @@ describe('AuthController permission revokes', () => {
issuer_user_id: issuer.id,
} as never,
);
// Grant first.
// Through the service: the grant route is retired, the revoke is not.
await inCtx(issuerActor, () =>
controller.handleGrantUserUser(
makeReq(
{ target_username: target.username, permission },
{ actor: issuerActor },
),
makeRes(),
server.services.permission.grantUserUserPermission(
issuerActor,
target.username,
permission,
),
);
// Now revoke — must complete without throwing and return {}.
// We don't re-assert the post-revoke `check()` answer here: the
// Redis-mock scan cache is process-wide, and intervening grants
// from other tests have repeatedly been observed to leave the
// cached `true` answer in place even after a successful revoke.
// Verifying the controller path rather than the cache eviction
// semantics keeps this test focused.
// Asserts the controller path only: the process-wide Redis-mock scan
// cache makes a post-revoke `check()` unreliable across tests.
const res = makeRes();
await inCtx(issuerActor, () =>
controller.handleRevokeUserUser(
@@ -4569,9 +4865,10 @@ describe('AuthController.handleCheckPermissions + handleListPermissions', () =>
);
// A user→user grant must show up under `myself_to_user` for the
// issuer and `user_to_myself` for the holder. Grants gate on
// `manage:<permission>`, so bootstrap that flag first (mirrors the
// grant-user-user persistence test).
// issuer and `user_to_myself` for the holder. Seeded through the
// service: the grant route is retired, but the listing still has to
// report grants made before that, and the ones sharing writes.
// Grants gate on `manage:<permission>`, so bootstrap that flag first.
const { user: holder, actor: holderActor } = await makeUserAndActor();
const userPermission = 'service:tl-user:ii:read';
await server.stores.permission.setFlatUserPerm(
@@ -4584,16 +4881,11 @@ describe('AuthController.handleCheckPermissions + handleListPermissions', () =>
} as never,
);
await inCtx(actor, () =>
controller.handleGrantUserUser(
makeReq(
{
target_username: holder.username,
permission: userPermission,
extra: {},
},
{ actor },
),
makeRes(),
server.services.permission.grantUserUserPermission(
actor,
holder.username,
userPermission,
{},
),
);
@@ -5022,6 +5314,33 @@ describe('AuthController 2FA flows', () => {
).toHaveLength(10);
});
it('a recovery code minted by configure-2fa setup satisfies /login/recovery-code', async () => {
const { user, actor } = await makeUserAndActor({ email_confirmed: 1 });
const setupRes = makeRes();
await controller.handleConfigure2fa(
makeReq({}, { actor, params: { action: 'setup' } }),
setupRes,
);
const { codes } = setupRes.body as { codes: string[] };
await controller.handleConfigure2fa(
makeReq({}, { actor, params: { action: 'enable' } }),
makeRes(),
);
const otpJwt = server.services.token.sign(
'otp',
{ user_uid: user.uuid, purpose: 'otp-login' },
{ expiresIn: '5m' },
);
const res = makeRes();
await controller.handleLoginRecoveryCode(
makeReq({ token: otpJwt, code: codes[0] }),
res,
);
expect(isCompleteLoginResponse(res.body)).toBe(true);
});
it('configure-2fa setup: 409 when 2FA is already enabled', async () => {
const { actor } = await makeUserAndActor({ otp_enabled: 1 });
await expect(
@@ -5116,130 +5435,6 @@ describe('AuthController.handleGetDevProfile', () => {
});
});
// ── Group endpoints ────────────────────────────────────────────────
describe('AuthController group endpoints', () => {
it('group/create: rejects non-object extra/metadata with 400', async () => {
const { actor } = await makeUserAndActor();
await expect(
controller.handleGroupCreate(
makeReq({ extra: ['x'] }, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
await expect(
controller.handleGroupCreate(
makeReq({ metadata: ['x'] }, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('group/create + add-users + remove-users: full owner-driven lifecycle', async () => {
const { actor: owner } = await makeUserAndActor();
const { user: target } = await makeUserAndActor();
// Create.
const createRes = makeRes();
await controller.handleGroupCreate(
makeReq({ metadata: { name: 'g' } }, { actor: owner }),
createRes,
);
const { uid } = createRes.body as { uid: string };
expect(typeof uid).toBe('string');
// Add.
const addRes = makeRes();
await controller.handleGroupAddUsers(
makeReq({ uid, users: [target.username] }, { actor: owner }),
addRes,
);
expect(addRes.body).toEqual({});
// Remove.
const remRes = makeRes();
await controller.handleGroupRemoveUsers(
makeReq({ uid, users: [target.username] }, { actor: owner }),
remRes,
);
expect(remRes.body).toEqual({});
});
it('group/add-users: 400 on missing uid or non-array users', async () => {
const { actor } = await makeUserAndActor();
await expect(
controller.handleGroupAddUsers(
makeReq({ users: ['x'] }, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
await expect(
controller.handleGroupAddUsers(
makeReq({ uid: 'g-1' }, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('group/add-users: 404 on unknown uid; 403 when caller doesn’t own the group', async () => {
const { actor: a1 } = await makeUserAndActor();
const { actor: a2 } = await makeUserAndActor();
await expect(
controller.handleGroupAddUsers(
makeReq(
{ uid: `does-not-exist-${uuidv4()}`, users: [] },
{ actor: a1 },
),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 404 });
// Group owned by a1; a2 tries to add → 403.
const createRes = makeRes();
await controller.handleGroupCreate(
makeReq({}, { actor: a1 }),
createRes,
);
const { uid } = createRes.body as { uid: string };
await expect(
controller.handleGroupAddUsers(
makeReq({ uid, users: [] }, { actor: a2 }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 403 });
});
it('group/list: forwards to GroupStore listByOwner/listByMember (or surfaces the source-side method-name mismatch)', async () => {
const { actor } = await makeUserAndActor();
const res = makeRes();
try {
await controller.handleGroupList(makeReq({}, { actor }), res);
const body = res.body as {
owned_groups: unknown[];
in_groups: unknown[];
};
expect(Array.isArray(body.owned_groups)).toBe(true);
expect(Array.isArray(body.in_groups)).toBe(true);
} catch (e) {
// The handler calls `stores.group.listByOwner(...)`, but the
// GroupStore implementation may expose a differently-named
// method. Surface the mismatch so a future GroupStore rename
// re-enables the assertion above.
expect((e as Error).message).toMatch(
/listByOwner|listByMember|is not a function/,
);
}
});
it('group/public-groups: returns {user, temp} from config', async () => {
const res = makeRes();
await controller.handleGroupPublicGroups(makeReq({}), res);
const body = res.body as { user: string | null; temp: string | null };
expect(body).toHaveProperty('user');
expect(body).toHaveProperty('temp');
});
});
// ── GUI token + session sync cookie ────────────────────────────────
describe('AuthController.handleGetGuiToken + handleSessionSyncCookie', () => {
@@ -6125,16 +6320,6 @@ describe('AuthController grant/revoke additional branches', () => {
).rejects.toMatchObject({ statusCode: 400 });
});
it('grant-user-group: 400 on missing group_uid', async () => {
const { actor } = await makeUserAndActor();
await expect(
controller.handleGrantUserGroup(
makeReq({ permission: 'fs:read' }, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('revoke-user-app: 400 when permission is "*" but app_uid is missing', async () => {
const { actor } = await makeUserAndActor();
await expect(
@@ -6252,58 +6437,6 @@ describe('AuthController.handleGetDevProfile additional branches', () => {
});
});
describe('AuthController group endpoints: additional branches', () => {
it('group/remove-users: 400 on missing uid', async () => {
const { actor } = await makeUserAndActor();
await expect(
controller.handleGroupRemoveUsers(
makeReq({ users: ['x'] }, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('group/remove-users: 400 on non-array users', async () => {
const { actor } = await makeUserAndActor();
await expect(
controller.handleGroupRemoveUsers(
makeReq({ uid: 'g-1' }, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('group/remove-users: 404 on unknown uid', async () => {
const { actor } = await makeUserAndActor();
await expect(
controller.handleGroupRemoveUsers(
makeReq(
{ uid: `does-not-exist-${uuidv4()}`, users: [] },
{ actor },
),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 404 });
});
it('group/remove-users: 403 when caller does not own the group', async () => {
const { actor: a1 } = await makeUserAndActor();
const { actor: a2 } = await makeUserAndActor();
const createRes = makeRes();
await controller.handleGroupCreate(
makeReq({}, { actor: a1 }),
createRes,
);
const { uid } = createRes.body as { uid: string };
await expect(
controller.handleGroupRemoveUsers(
makeReq({ uid, users: [] }, { actor: a2 }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 403 });
});
});
describe('AuthController.handleGetGuiToken / handleSessionSyncCookie additional branches', () => {
it('get-gui-token: 404 when actor has a session but the user row is gone', async () => {
const { user, actor } = await makeUserAndActor();
+278 -383
View File
@@ -52,6 +52,8 @@ import {
createSecret as otpCreateSecret,
verify as verifyOtp,
} from '../../services/auth/OTPUtil.js';
import type { UserRow } from '../../stores/user/UserStore.js';
import { isOwnedEmailConflict } from '../../stores/user/UserStore.js';
import { sessionCookieFlags } from '../../util/cookieFlags.js';
import { cleanEmail, isBlockedEmail } from '../../util/email.js';
import { generate_identifier } from '../../util/identifier.js';
@@ -116,7 +118,7 @@ const TWO_FACTOR_LIMIT = {
key: 'user',
} as const;
/** Permission and membership writes. Never called in a loop by a client. */
/** Permission writes. Never called in a loop by a client. */
const GRANT_LIMIT = {
scope: 'auth-grant',
limit: 60,
@@ -156,7 +158,7 @@ const ANTI_CSRF_MINT_LIMIT = {
key: 'user',
} as const;
/** Settings-page reads — enumerating sessions, permissions, groups. */
/** Settings-page reads — enumerating sessions and permissions. */
const AUTH_LIST_LIMIT = {
scope: 'auth-list',
limit: 120,
@@ -177,6 +179,21 @@ const CARD_FALLBACK_OPEN_TTL_SECONDS = 24 * 60 * 60;
// How long a failed-SMS-send record stays readable by its error_id — long
// enough to cover the typical support round-trip.
const SMS_SEND_ERROR_TTL_SECONDS = 7 * 24 * 60 * 60;
/**
* Whether this account has already been through the card flow. Both halves
* matter: the pending flag being clear only says nobody asked, while the
* fingerprint is the artifact a completed check leaves behind. A route that
* requires a verified card (`requireCardVerified`) sends users here with the
* flag clear and no card on file, and answering "already verified" on the flag
* alone would bounce them between a dialog that reports success and a route
* that keeps refusing.
*/
const cardAlreadyVerified = (user: {
card_fingerprint?: string | null;
requires_card_verification?: boolean;
}): boolean =>
Boolean(user.card_fingerprint) && !user.requires_card_verification;
const RESERVED_USERNAMES = new Set([
'admin',
'administrator',
@@ -775,11 +792,9 @@ export class AuthController extends PuterController {
if (this.config.disable_user_signup) {
let claimable = false;
if (!is_temp) {
const existing =
(await this.stores.user.getByEmail(body.email)) ??
(await this.stores.user.getByCleanEmail(
cleanEmail(body.email),
));
const existing = await this.stores.user.findEmailOwner(
body.email,
);
claimable = Boolean(
existing &&
!existing.email_confirmed &&
@@ -817,28 +832,18 @@ export class AuthController extends PuterController {
// password to an OIDC account, the owner logs in via OIDC and
// uses the authenticated change-password flow.
//
// Match on both raw `email` and canonical `clean_email` so
// Matching runs against both raw `email` and canonical `clean_email` so
// gmail-style aliases (`[email protected]` vs
// `[email protected]`) collapse to the same account.
let pseudo_user = null;
if (!is_temp) {
const canonical = cleanEmail(body.email);
const existing =
(await this.stores.user.getByEmail(body.email)) ??
(await this.stores.user.getByCleanEmail(canonical));
if (existing) {
// Confirmed account (regardless of credential type) → reject.
if (existing.email_confirmed || existing.password !== null) {
throw new HttpError(
400,
'This email already exists in our database. Please use another one.',
{ legacyCode: 'bad_request' },
);
}
// Password-null AND unconfirmed → treat as pseudo.
pseudo_user = existing;
}
}
//
// This is the cheap early check: it keeps an obvious duplicate from
// paying for the validate hook and a bcrypt round. It is NOT the
// guarantee — everything between here and the insert widens the window,
// so the check runs again against the primary immediately before the
// write, and the unique index catches whatever still slips through.
let pseudo_user = is_temp
? null
: await this.#resolveSignupEmailClaim(body.email);
// Extension-level validation gate. Abuse-prevention extensions
// inspect the incoming signup and can:
@@ -857,6 +862,15 @@ export class AuthController extends PuterController {
req.socket?.remoteAddress ||
null) as string | null,
email: body.email,
// The same canonical form `email.validate` was given, so a check
// in the abuse harness can look up the verdict that hook cached
// for this address. Without it an alias (`[email protected]`,
// `[email protected]`) reaches the two hooks under two different keys.
clean_email: cleanEmail(body.email),
// Temp signups carry a synthetic `<username>@gmail.com` and skip
// #validateEmail entirely, so an email check must know not to
// reason about the address at all.
is_temp,
allow: true,
no_temp_user: false,
requires_email_confirmation: false,
@@ -945,24 +959,54 @@ export class AuthController extends PuterController {
.slice(0, 19)
.replace('T', ' ');
// Re-run the claim against the primary now that the slow work is done.
// The check above ran before the validate hook (network round-trips to
// the abuse listeners) and before bcrypt — hundreds of milliseconds in
// which a concurrent signup can take the address, or claim the very
// placeholder row we were about to convert.
if (!is_temp) {
pseudo_user = await this.#resolveSignupEmailClaim(body.email, {
force: true,
});
}
let user;
if (pseudo_user) {
// -- Pseudo-user claim (convert the placeholder row) --
await this.stores.user.update(pseudo_user.id, {
username: body.username,
password: password_hash,
uuid: user_uuid,
email_confirm_code,
email_confirm_token,
email_confirmed: 0,
requires_email_confirmation: 1,
last_activity_ts: signupSqlTs,
...(validateEvent.reputation != null
? { reputation: validateEvent.reputation }
: {}),
requires_phone_verification: force_phone_verification ? 1 : 0,
requires_card_verification: force_card_verification ? 1 : 0,
});
//
// Guarded, not a plain update: the address never changes hands here
// (the row already holds it), so the unique index has nothing to
// catch. Two signups that both read this row as claimable would
// otherwise both "succeed", the second overwriting the first's
// username and password on a row the first was already given a
// session for.
const claimed = await this.stores.user.claimPlaceholder(
pseudo_user.id,
{
username: body.username,
password: password_hash,
uuid: user_uuid,
email_confirm_code,
email_confirm_token,
email_confirmed: 0,
requires_email_confirmation: 1,
last_activity_ts: signupSqlTs,
...(validateEvent.reputation != null
? { reputation: validateEvent.reputation }
: {}),
requires_phone_verification: force_phone_verification
? 1
: 0,
requires_card_verification: force_card_verification ? 1 : 0,
},
);
if (!claimed) {
throw new HttpError(
400,
'This email already exists in our database. Please use another one.',
{ legacyCode: 'bad_request' },
);
}
// Move from temp group to regular user group
if (this.config.default_temp_group) {
@@ -994,37 +1038,51 @@ export class AuthController extends PuterController {
const clientIp = req.ip || req.socket?.remoteAddress || null;
const proxyIpChain = req.headers['x-forwarded-for'];
user = await this.stores.user.create({
username: body.username,
uuid: user_uuid,
password: password_hash,
email: is_temp ? null : body.email,
clean_email: is_temp ? null : cleanEmail(body.email),
free_storage: this.config.storage_capacity ?? null,
requires_email_confirmation:
!is_temp || force_email_confirmation,
email_confirm_code,
email_confirm_token,
audit_metadata: {
ip: clientIp,
ip_fwd: proxyIpChain,
user_agent: req.headers?.['user-agent'],
origin: req.headers?.origin,
fingerprint,
},
signup_ip: clientIp,
signup_ip_forwarded: proxyIpChain,
signup_user_agent: req.headers?.['user-agent'] ?? null,
signup_origin: (req.headers?.origin as string | null) ?? null,
signup_server: (this.config as { serverId?: string }).serverId,
referrer: req.body.referrer ?? null,
last_activity_ts: signupSqlTs,
reputation: validateEvent.reputation,
// Phone collected later in the verification dialog (null now).
phone: null,
requires_phone_verification: force_phone_verification,
requires_card_verification: force_card_verification,
} as never);
try {
user = await this.stores.user.create({
username: body.username,
uuid: user_uuid,
password: password_hash,
email: is_temp ? null : body.email,
clean_email: is_temp ? null : cleanEmail(body.email),
free_storage: this.config.storage_capacity ?? null,
requires_email_confirmation:
!is_temp || force_email_confirmation,
email_confirm_code,
email_confirm_token,
audit_metadata: {
ip: clientIp,
ip_fwd: proxyIpChain,
user_agent: req.headers?.['user-agent'],
origin: req.headers?.origin,
fingerprint,
},
signup_ip: clientIp,
signup_ip_forwarded: proxyIpChain,
signup_user_agent: req.headers?.['user-agent'] ?? null,
signup_origin:
(req.headers?.origin as string | null) ?? null,
signup_server: (this.config as { serverId?: string })
.serverId,
referrer: req.body.referrer ?? null,
last_activity_ts: signupSqlTs,
reputation: validateEvent.reputation,
// Phone collected later in the verification dialog (null now).
phone: null,
requires_phone_verification: force_phone_verification,
requires_card_verification: force_card_verification,
} as never);
} catch (e) {
// Lost the race to another signup between the re-check above and
// this insert. The index is the only thing that can see that, so
// translate it into the answer the pre-check would have given.
if (!isOwnedEmailConflict(e)) throw e;
throw new HttpError(
400,
'This email already exists in our database. Please use another one.',
{ legacyCode: 'bad_request' },
);
}
// Add to default group
const defaultGroup = is_temp
@@ -1274,6 +1332,34 @@ export class AuthController extends PuterController {
// after signup but before confirmation.
await this.#validateEmail(user.email!);
// An account that already confirmed this address proved access to the
// inbox, and revoking it below would hand the address to whoever
// confirmed second. Refuse instead — a duplicate this old is data to
// repair, not a race to resolve.
const canonical = cleanEmail(user.email!);
const confirmedRival = await this.stores.user.findConfirmedOtherByEmail(
user.id,
user.email!,
canonical,
);
if (confirmedRival) {
throw new HttpError(
400,
'This email was confirmed on a different account.',
{ legacyCode: 'email_already_in_use' as never },
);
}
// Revoke the address from every remaining (unconfirmed) account holding
// it, THEN confirm this one. Only one row may own an address, so
// confirming first would momentarily create a second owner — which the
// unique index rejects, turning a legitimate confirmation into a 500.
await this.stores.user.unconfirmOthersByEmail(
user.id,
user.email!,
canonical,
);
await this.stores.user.update(user.id, {
email_confirmed: 1,
requires_email_confirmation: 0,
@@ -1281,16 +1367,6 @@ export class AuthController extends PuterController {
email_confirm_token: null,
});
// Revoke confirmation from any other accounts sharing this
// email so only the account whose owner just proved inbox
// access retains verified status.
const canonical = cleanEmail(user.email!);
await this.stores.user.unconfirmOthersByEmail(
user.id,
user.email!,
canonical,
);
await promoteToVerifiedGroup(this.stores.group, this.config, user);
try {
@@ -1863,7 +1939,7 @@ export class AuthController extends PuterController {
// Phone normally comes first, but the fallback lets a phone-gated user
// in once they've exhausted SMS attempts.
const fallbackEligible = await this.isCardFallbackEligible(user);
if (!user.requires_card_verification && !fallbackEligible) {
if (cardAlreadyVerified(user) && !fallbackEligible) {
res.json({ card_verified: true });
return;
}
@@ -1983,7 +2059,7 @@ export class AuthController extends PuterController {
});
// Same fallback exception as setup: card may come before phone.
const fallbackEligible = await this.isCardFallbackEligible(user);
if (!user.requires_card_verification && !fallbackEligible) {
if (cardAlreadyVerified(user) && !fallbackEligible) {
res.json({ card_verified: true });
return;
}
@@ -2277,10 +2353,26 @@ export class AuthController extends PuterController {
// Atomic check: only update if the recovery token still matches
const password_hash = await bcrypt.hash(password, 8);
const result = await this.clients.db.write(
'UPDATE `user` SET `password` = ?, `pass_recovery_token` = NULL, `change_email_confirm_token` = NULL WHERE `id` = ? AND `pass_recovery_token` = ?',
[password_hash, user.id, decoded.token],
);
let result;
try {
result = await this.clients.db.write(
'UPDATE `user` SET `password` = ?, `pass_recovery_token` = NULL, `change_email_confirm_token` = NULL WHERE `id` = ? AND `pass_recovery_token` = ?',
[password_hash, user.id, decoded.token],
);
} catch (e) {
if (!isOwnedEmailConflict(e)) throw e;
// Recovery can be requested by username, so this row may be an
// unconfirmed placeholder that shares its address with a real
// account. Giving it a password would make it a second account able
// to drive recovery for that inbox, which is the thing the address
// constraint exists to stop. The inbox owner has an account
// already — they should be recovering that one.
throw new HttpError(
400,
'This email is already in use. Recover the account that uses it instead.',
{ legacyCode: 'email_already_in_use' as never },
);
}
const affected =
(result as { affectedRows?: number; changes?: number })
?.affectedRows ??
@@ -2441,10 +2533,7 @@ export class AuthController extends PuterController {
// aliases — which is also why the caller has to be excluded: an
// alias of your own current address resolves back to you, and
// "already in use" about yourself is nonsense.
const canonical = cleanEmail(new_email);
const existing =
(await this.stores.user.getByEmail(new_email)) ??
(await this.stores.user.getByCleanEmail(canonical));
const existing = await this.stores.user.findEmailOwner(new_email);
if (
existing &&
existing.id !== req.actor!.user.id &&
@@ -2545,7 +2634,7 @@ export class AuthController extends PuterController {
}
const rows = (await this.clients.db.read(
'SELECT * FROM `user` WHERE `change_email_confirm_token` = ? LIMIT 1',
'SELECT * FROM `user` WHERE `change_email_confirm_token` = ? ORDER BY `id` ASC LIMIT 1',
[decoded.token],
)) as Array<Record<string, unknown>>;
const user = rows[0] as
@@ -2566,11 +2655,12 @@ export class AuthController extends PuterController {
// Re-check nobody claimed the new email meanwhile. Match raw +
// canonical; block if any real account (confirmed OR
// password-holding) already owns it.
// password-holding) already owns it. Read the primary — the request
// that took the address may have landed moments ago.
const canonical = cleanEmail(newEmail);
const owner =
(await this.stores.user.getByEmail(newEmail)) ??
(await this.stores.user.getByCleanEmail(canonical));
const owner = await this.stores.user.findEmailOwner(newEmail, {
force: true,
});
if (
owner &&
owner.id !== user.id &&
@@ -2581,15 +2671,30 @@ export class AuthController extends PuterController {
});
}
await this.stores.user.update(user.id, {
email: newEmail,
clean_email: cleanEmail(newEmail),
unconfirmed_change_email: null,
change_email_confirm_token: null,
pass_recovery_token: null,
email_confirmed: 1,
requires_email_confirmation: 0,
});
// Strip the address off any unconfirmed placeholder still holding it
// before taking it, so this row is the only owner.
await this.stores.user.unconfirmOthersByEmail(
user.id,
newEmail,
canonical,
);
try {
await this.stores.user.update(user.id, {
email: newEmail,
clean_email: canonical,
unconfirmed_change_email: null,
change_email_confirm_token: null,
pass_recovery_token: null,
email_confirmed: 1,
requires_email_confirmation: 0,
});
} catch (e) {
if (!isOwnedEmailConflict(e)) throw e;
throw new HttpError(400, 'This email is already in use.', {
legacyCode: 'email_already_in_use' as never,
});
}
await this.stores.oidc.unlinkAllByUserId(user.id);
@@ -2692,9 +2797,7 @@ export class AuthController extends PuterController {
// reject on ANY confirmed account (OIDC accounts have
// password=null but are real) — not just password-holders.
const canonical = cleanEmail(email);
const existingEmail =
(await this.stores.user.getByEmail(email)) ??
(await this.stores.user.getByCleanEmail(canonical));
const existingEmail = await this.stores.user.findEmailOwner(email);
if (
existingEmail &&
existingEmail.id !== user.id &&
@@ -2710,16 +2813,38 @@ export class AuthController extends PuterController {
const email_confirm_code = String(crypto.randomInt(100000, 1000000));
const email_confirm_token = uuidv4();
await this.stores.user.update(user.id, {
username,
email,
clean_email: cleanEmail(email),
password: password_hash,
email_confirm_code,
email_confirm_token,
email_confirmed: 0,
requires_email_confirmation: 1,
// bcrypt above is slow enough for someone else to take the address in
// the meantime, so re-check against the primary before the write.
const raced = await this.stores.user.findEmailOwner(email, {
force: true,
});
if (
raced &&
raced.id !== user.id &&
(raced.email_confirmed || raced.password !== null)
) {
throw new HttpError(400, 'This email is already in use.', {
legacyCode: 'email_already_in_use' as never,
});
}
try {
await this.stores.user.update(user.id, {
username,
email,
clean_email: canonical,
password: password_hash,
email_confirm_code,
email_confirm_token,
email_confirmed: 0,
requires_email_confirmation: 1,
});
} catch (e) {
if (!isOwnedEmailConflict(e)) throw e;
throw new HttpError(400, 'This email is already in use.', {
legacyCode: 'email_already_in_use' as never,
});
}
// Rename the user's FS home so `/<temp>/Desktop` etc.
// become `/<new>/Desktop`. Without this cascade, any
@@ -2849,30 +2974,6 @@ export class AuthController extends PuterController {
// -- Permission grants -------------------------------------------
@Post('/auth/grant-user-user', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGrantUserUser(req: Request, res: Response): Promise<void> {
const { target_username, permission, extra, meta } = req.body;
if (!target_username || !permission) {
throw new HttpError(
400,
'Missing `target_username` or `permission`',
{ legacyCode: 'bad_request' },
);
}
await this.services.permission.grantUserUserPermission(
req.actor!,
target_username,
permission,
extra,
meta,
);
res.json({});
}
/**
* Shared input validation for the user-app grant/revoke handlers, which
* accept a caller-supplied `origin` as an alternative to `app_uid`. All
@@ -3093,35 +3194,13 @@ export class AuthController extends PuterController {
res.json({});
}
@Post('/auth/grant-user-group', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGrantUserGroup(req: Request, res: Response): Promise<void> {
const { group_uid, permission, extra, meta } = req.body;
if (!group_uid || !permission) {
throw new HttpError(400, 'Missing `group_uid` or `permission`', {
legacyCode: 'bad_request',
});
}
const group = await this.stores.group.getByUid(group_uid);
if (!group)
throw new HttpError(404, 'Group not found', {
legacyCode: 'not_found',
});
await this.services.permission.grantUserGroupPermission(
req.actor!,
group,
permission,
extra,
meta,
);
res.json({});
}
// -- Permission revokes ------------------------------------------
/**
* @deprecated Use `puter.fs.unshare()`, which withdraws the share row and
* the grant together. Kept for direct HTTP callers: the grant side is
* retired, but access it left behind has to stay withdrawable.
*/
@Post('/auth/revoke-user-user', {
subdomain: 'api',
requireUserActor: true,
@@ -3189,27 +3268,6 @@ export class AuthController extends PuterController {
res.json({});
}
@Post('/auth/revoke-user-group', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleRevokeUserGroup(req: Request, res: Response): Promise<void> {
const { group_uid, permission, meta } = req.body;
if (!group_uid || !permission) {
throw new HttpError(400, 'Missing `group_uid` or `permission`', {
legacyCode: 'bad_request',
});
}
await this.services.permission.revokeUserGroupPermission(
req.actor!,
{ uid: group_uid } as never,
permission,
meta,
);
res.json({});
}
// -- Permission checks -------------------------------------------
@Post('/auth/check-permissions', {
@@ -3884,147 +3942,6 @@ export class AuthController extends PuterController {
});
}
// -- Group management --------------------------------------------
@Post('/group/create', {
subdomain: 'api',
requireUserActor: true,
// Creates a persistent row per call with no quota behind it, so it
// sits on the hour-scale budget rather than the grant one.
rateLimit: { ...CREDENTIAL_MINT_LIMIT, scope: 'group-create' },
})
async handleGroupCreate(req: Request, res: Response): Promise<void> {
const extra = req.body.extra ?? {};
const metadata = req.body.metadata ?? {};
if (typeof extra !== 'object' || Array.isArray(extra))
throw new HttpError(400, '`extra` must be an object', {
legacyCode: 'bad_request',
});
if (typeof metadata !== 'object' || Array.isArray(metadata))
throw new HttpError(400, '`metadata` must be an object', {
legacyCode: 'bad_request',
});
const uid = await this.stores.group.create({
ownerUserId: req.actor!.user.id,
extra: {},
metadata,
} as never);
res.json({ uid });
}
@Post('/group/add-users', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGroupAddUsers(req: Request, res: Response): Promise<void> {
const { uid, users } = req.body ?? {};
if (!uid)
throw new HttpError(400, 'Missing `uid`', {
legacyCode: 'bad_request',
});
if (!Array.isArray(users))
throw new HttpError(400, '`users` must be an array', {
legacyCode: 'bad_request',
});
const group = await this.stores.group.getByUid(uid);
if (!group)
throw new HttpError(404, 'Group not found', {
legacyCode: 'not_found',
});
if (
(group as { owner_user_id?: number }).owner_user_id !==
req.actor!.user.id
)
throw new HttpError(403, 'Forbidden', {
legacyCode: 'forbidden',
});
await this.stores.group.addUsers(uid, users);
// New members inherit the group's permissions immediately, not
// after the permission-cache TTL.
await this.services.permission.bumpPermissionCacheForUsernames(users);
res.json({});
}
@Post('/group/remove-users', {
subdomain: 'api',
requireUserActor: true,
rateLimit: GRANT_LIMIT,
})
async handleGroupRemoveUsers(req: Request, res: Response): Promise<void> {
const { uid, users } = req.body ?? {};
if (!uid)
throw new HttpError(400, 'Missing `uid`', {
legacyCode: 'bad_request',
});
if (!Array.isArray(users))
throw new HttpError(400, '`users` must be an array', {
legacyCode: 'bad_request',
});
const group = await this.stores.group.getByUid(uid);
if (!group)
throw new HttpError(404, 'Group not found', {
legacyCode: 'not_found',
});
if (
(group as { owner_user_id?: number }).owner_user_id !==
req.actor!.user.id
)
throw new HttpError(403, 'Forbidden', {
legacyCode: 'forbidden',
});
await this.stores.group.removeUsers(uid, users);
// Removed members must lose the group's permissions immediately,
// not after the permission-cache TTL.
await this.services.permission.bumpPermissionCacheForUsernames(users);
res.json({});
}
@Get('/group/list', {
subdomain: 'api',
requireUserActor: true,
rateLimit: AUTH_LIST_LIMIT,
})
async handleGroupList(req: Request, res: Response): Promise<void> {
const userId = req.actor!.user.id!;
const [owned, member] = await Promise.all([
this.stores.group.listGroupsWithOwner(userId),
this.stores.group.listGroupsWithMember(userId),
]);
res.json({
owned_groups: owned,
in_groups: member,
});
}
@Get('/group/public-groups', {
subdomain: 'api',
// The only unauthenticated route in the group set, so IP is the
// only key available — and that makes the bucket an aggregate:
// one office, campus or carrier gateway is a single key for
// everybody behind it, and each of them reads this once while
// bootstrapping. Sized for that population of real people rather
// than one browser, and no wider: this sits next to the sign-in
// surface, so it stays a real bound on enumeration.
rateLimit: {
scope: 'public-groups',
limit: 1_200,
window: 60_000,
key: 'ip',
},
})
async handleGroupPublicGroups(_req: Request, res: Response): Promise<void> {
res.json({
user: this.config.default_user_group ?? null,
temp: this.config.default_temp_group ?? null,
});
}
// -- Session helpers ---------------------------------------------
@Get('/get-gui-token', {
@@ -4425,57 +4342,7 @@ export class AuthController extends PuterController {
// -- Private helpers ----------------------------------------------
async #cascadeDeleteUser(userId: number): Promise<void> {
// Capture the identifiers downstream teardown needs before the row is
// gone — the marketplace extension cancels the user's Stripe
// subscriptions off `user.delete`, keyed by uuid / customer id.
let userUuid: string | undefined;
let stripeCustomerId: string | null = null;
try {
const rows = (await this.clients.db.read(
'SELECT `uuid`, `stripe_customer_id` FROM `user` WHERE `id` = ?',
[userId],
)) as Array<{ uuid?: string; stripe_customer_id?: string | null }>;
userUuid = rows[0]?.uuid;
stripeCustomerId = rows[0]?.stripe_customer_id ?? null;
} catch (e) {
console.warn('[cascade-delete-user] identifier lookup failed:', e);
}
try {
await this.services.fs.removeAllForUser(userId);
} catch (e) {
// Proceed with user-row delete anyway — orphaned fsentries are
// better than a resurrected account.
console.warn('[cascade-delete-user] fs cleanup failed:', e);
}
// Sessions FK is SET NULL, so delete explicitly to avoid dangling rows.
await this.clients.db.write(
'DELETE FROM `sessions` WHERE `user_id` = ?',
[userId],
);
await this.clients.db.write('DELETE FROM `user` WHERE `id` = ?', [
userId,
]);
await this.stores.user.invalidateById(userId);
// Fire-and-forget: let listeners purge external state tied to the
// account (Stripe subscriptions are cancelled immediately, without
// proration). Emitted after the row delete — listeners key off the
// payload, not the DB row.
try {
this.clients.event?.emit(
'user.delete',
{
user_id: userId,
user_uuid: userUuid,
stripe_customer_id: stripeCustomerId,
},
{},
);
} catch {
// ignore — event emission shouldn't block deletion
}
await this.services.userAccount.cascadeDelete(userId);
}
async #generateRandomUsername(): Promise<string> {
@@ -4494,6 +4361,34 @@ export class AuthController extends PuterController {
return username;
}
/**
* Decide whether a signup may take `email`, and hand back the placeholder
* row it should convert instead of inserting a new one.
*
* Throws when a live account already owns the address. Returns the
* unconfirmed, password-less pseudo row when one exists (admin
* pre-provisioning — signup claims it), or null when the address is free.
*
* Called twice per signup: once early, to fail fast before the validate
* hook and bcrypt, and once against the primary immediately before the
* write.
*/
async #resolveSignupEmailClaim(
email: string,
opts: { force?: boolean } = {},
): Promise<UserRow | null> {
const existing = await this.stores.user.findEmailOwner(email, opts);
if (!existing) return null;
if (existing.email_confirmed || existing.password !== null) {
throw new HttpError(
400,
'This email already exists in our database. Please use another one.',
{ legacyCode: 'bad_request' },
);
}
return existing;
}
/**
* Config-blocklist + extension-driven email validation. Config blocklist
* (suffix match on cleaned email) blocks first; then the `email.validate`
@@ -0,0 +1,164 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/* eslint-disable @typescript-eslint/no-explicit-any */
import { EventEmitter } from 'node:events';
import type { Request, RequestHandler, Response } from 'express';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { runWithContext } from '../../core/context.js';
import { isHttpError } from '../../core/http/HttpError.js';
import { configureRateLimit } from '../../core/http/middleware/rateLimit.js';
import { DriverController } from './DriverController.js';
// The subscription requirement a driver declares per method, exercised
// through `#handleCall` the same way the concurrent test does: a synthetic
// driver carrying only the field under test, plus a stub metering service
// that answers with a fixed plan.
const captureCallHandler = (controller: DriverController): RequestHandler => {
let handler: RequestHandler | undefined;
const fakeRouter = {
post: (path: string, _opts: unknown, h: RequestHandler) => {
if (path === '/call') handler = h;
return fakeRouter;
},
get: () => fakeRouter,
use: () => fakeRouter,
};
controller.registerRoutes(
fakeRouter as unknown as Parameters<
typeof controller.registerRoutes
>[0],
);
if (!handler) throw new Error('failed to capture POST /call handler');
return handler;
};
class StubRes extends EventEmitter {
statusCode = 200;
body: unknown = undefined;
status(code: number) {
this.statusCode = code;
return this;
}
json(body: unknown) {
this.body = body;
return this;
}
setHeader() {
return this;
}
type() {
return this;
}
send() {
return this;
}
}
const makeReq = (method: string, tag: string): Request =>
({
body: { interface: 'test-iface', method },
actor: { user: { uuid: `u-${tag}`, username: 'u' } },
headers: { 'user-agent': tag },
query: {},
ip: '127.0.0.1',
socket: { remoteAddress: '127.0.0.1' },
}) as unknown as Request;
const syntheticDriver = {
driverInterface: 'test-iface',
driverName: 'test-driver',
isDefault: true,
requireSubscription: {
methods: { paid: true, premium: ['business', 'pro'] },
},
onServerStart() {},
onServerPrepareShutdown() {},
onServerShutdown() {},
free: async () => 'free-ok',
paid: async () => 'paid-ok',
premium: async () => 'premium-ok',
};
const buildController = (plan: string | null) => {
const clients = { alarm: { create: () => {} } };
const services = plan
? {
metering: {
getActorSubscription: vi.fn().mockResolvedValue({ id: plan }),
},
}
: {};
return new DriverController(
{} as any,
clients as any,
{} as any,
services as any,
{ syntheticDriver } as any,
);
};
const call = (plan: string | null, method: string, tag: string) => {
const handler = captureCallHandler(buildController(plan));
const res = new StubRes();
return runWithContext({}, () =>
handler(makeReq(method, tag), res as unknown as Response, () => {}),
).then(() => res);
};
const expect402 = (promise: Promise<unknown>) =>
expect(promise).rejects.toSatisfy(
(e) =>
isHttpError(e) &&
(e as { statusCode: number }).statusCode === 402 &&
(e as { legacyCode?: string }).legacyCode ===
'subscription_required',
);
describe('DriverController — per-method subscription requirement', () => {
beforeEach(() => {
configureRateLimit();
});
it('leaves a method that declares nothing open to a free account', async () => {
const res = await call('user_free', 'free', 'open');
expect(res.body).toMatchObject({ success: true, result: 'free-ok' });
});
it('rejects a free account on a method that requires a subscription', async () => {
await expect402(call('user_free', 'paid', 'paid-free'));
});
it('admits a subscriber on that same method', async () => {
const res = await call('basic', 'paid', 'paid-sub');
expect(res.body).toMatchObject({ success: true, result: 'paid-ok' });
});
it('honours a per-method plan allowlist', async () => {
await expect402(call('basic', 'premium', 'premium-basic'));
const res = await call('pro', 'premium', 'premium-pro');
expect(res.body).toMatchObject({ success: true, result: 'premium-ok' });
});
it('enforces nothing on a deployment with no metering service', async () => {
const res = await call(null, 'paid', 'no-metering');
expect(res.body).toMatchObject({ success: true, result: 'paid-ok' });
});
});
@@ -36,7 +36,9 @@ import {
resolveDriverMeta,
resolveDriverMethodConcurrent,
resolveDriverMethodRateLimit,
resolveDriverMethodRequireSubscription,
} from '../../drivers/meta.js';
import { assertActorHasSubscription } from '../../services/metering/enforcement.js';
import type { PermissionService } from '../../services/permission/PermissionService.js';
import { PermissionUtil } from '../../services/permission/permissionUtil.js';
import type { WithLifecycle } from '../../types';
@@ -325,6 +327,25 @@ export class DriverController extends PuterController {
}
}
// Subscriber-only methods. Declared per-driver
// (`@Driver({ requireSubscription })`) because `/drivers/call` is a
// single route and a route option would apply to every driver at once.
// Checked before the rate limit — the same order the route chain uses
// — so a caller whose plan never included the method is told that
// rather than spending a bucket on it.
const subscriptionRequirement = resolveDriverMethodRequireSubscription(
driverMeta?.requireSubscription,
method,
);
if (subscriptionRequirement !== undefined) {
await assertActorHasSubscription(
this.services.metering,
req.actor,
subscriptionRequirement,
this.config,
);
}
// Per-method rate-limit and concurrent specs both live on the
// driver's resolved meta (set by `@Driver({ rateLimit, concurrent })`
// or imperative fields). Rate-limit is single-shot; concurrent
@@ -55,6 +55,8 @@ afterAll(async () => {
afterEach(() => {
vi.restoreAllMocks();
// Back to the self-hosted no-SMTP baseline the server booted with.
delete liveConfig().email;
});
const makeUser = async (): Promise<{ actor: Actor; userId: number }> => {
@@ -101,11 +103,15 @@ const makeApp = async (
// Feedback is only offered when the deployment can deliver it (email
// transport configured); most tests want that baseline without asserting
// anything about the mail itself.
const mockEmailConfigured = () =>
vi.spyOn(server.clients.email, 'isConfigured', 'get').mockReturnValue(
true,
);
// anything about the mail itself. The service reads `config.email` — the
// same live object every layer holds — so the helper writes it there and
// the global afterEach clears it.
const liveConfig = () =>
(server.clients.email as unknown as { config: Record<string, unknown> })
.config;
const mockEmailConfigured = () => {
liveConfig().email = { jsonTransport: true };
};
const confirmOwnerEmail = async (userId: number) => {
await server.clients.db.write(
+84 -23
View File
@@ -25,7 +25,15 @@ import type { Actor } from '../../core/actor.js';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { Controller, Get, Post } from '../../core/http/decorators.js';
import { assertNormalized } from '../../services/fs/resolveNode.js';
import {
assertNormalized,
isOwnersTrash,
} from '../../services/fs/resolveNode.js';
import {
maskEntryPath,
maskPathForRequest,
resolveSharePath,
} from '../../services/fs/sharePathMask.js';
import type {
PreparedBatchWrite,
UploadedBatchWriteItem,
@@ -144,7 +152,7 @@ export class FSController extends PuterController {
const userId = this.#getActorUserId(req);
const storageAllowanceMax = this.#getStorageAllowanceMaxOverride(req);
const requestBody = this.#withGuiMetadata(req.body, req.body);
requestBody.fileMetadata = this.#normalizeFileMetadataPath(
requestBody.fileMetadata = await this.#normalizeFileMetadataPath(
req,
requestBody.fileMetadata,
requestBody,
@@ -215,7 +223,7 @@ export class FSController extends PuterController {
req.body,
);
normalizedRequestBody.fileMetadata =
this.#normalizeFileMetadataPath(
await this.#normalizeFileMetadataPath(
req,
normalizedRequestBody.fileMetadata,
normalizedRequestBody,
@@ -435,7 +443,7 @@ export class FSController extends PuterController {
const userId = this.#getActorUserId(req);
const storageAllowanceMax = this.#getStorageAllowanceMaxOverride(req);
const requestBody = this.#withGuiMetadata(req.body, req.body);
requestBody.fileMetadata = this.#normalizeFileMetadataPath(
requestBody.fileMetadata = await this.#normalizeFileMetadataPath(
req,
requestBody.fileMetadata,
requestBody,
@@ -449,7 +457,7 @@ export class FSController extends PuterController {
await this.#assertWriteAccess(req, requestBody.fileMetadata, {
pathAlreadyNormalized: true,
});
const normalizedPath = this.#normalizePath(
const normalizedPath = await this.#resolveClientPath(
requestBody.fileMetadata.path,
);
const uploadTracker = await this.#createUploadTracker(
@@ -555,7 +563,7 @@ export class FSController extends PuterController {
...parsedManifest,
items: parsedManifest.items.map((item) => ({
...item,
fileMetadata: this.#normalizeFileMetadataPath(
fileMetadata: this.#expandFileMetadataPath(
req,
item.fileMetadata,
item,
@@ -586,7 +594,12 @@ export class FSController extends PuterController {
...item,
fileMetadata:
await this.#resolveAssociatedAppMetadata(
item.fileMetadata,
{
...item.fileMetadata,
path: await this.#unmaskPath(
item.fileMetadata.path,
),
},
item,
appUidLookupCache,
userId,
@@ -831,7 +844,7 @@ export class FSController extends PuterController {
req.body,
);
normalizedRequestBody.fileMetadata =
this.#normalizeFileMetadataPath(
await this.#normalizeFileMetadataPath(
req,
normalizedRequestBody.fileMetadata,
normalizedRequestBody,
@@ -994,7 +1007,7 @@ export class FSController extends PuterController {
uuid: entry.uuid,
uid: entry.uid ?? entry.uuid,
parentUid: entry.parentUid ?? null,
path: entry.path,
path: maskEntryPath(entry),
name: entry.name,
isDir: entry.isDir,
isShortcut: entry.isShortcut,
@@ -1110,7 +1123,6 @@ export class FSController extends PuterController {
const rootChildren = await listRootEntries(
actor,
this.stores.fsEntry,
this.services.permission,
);
const rootSuggestions =
await this.services.suggestedApps.getSuggestedAppsForEntries(
@@ -1385,7 +1397,7 @@ export class FSController extends PuterController {
// would compute a wrong parent for `~/...` inputs (e.g. dirname of
// `/~/Documents/foo` is `/~/Documents`, not `/<username>/Documents`).
const username = this.#getActorUsername(req);
const path = this.#normalizePath(rawPath, username);
const path = await this.#resolveClientPath(rawPath, username);
if (path === '/')
throw new HttpError(400, 'Cannot mkdir at root', {
legacyCode: 'bad_request',
@@ -1426,7 +1438,7 @@ export class FSController extends PuterController {
});
const username = this.#getActorUsername(req);
const path = this.#normalizePath(rawPath, username);
const path = await this.#resolveClientPath(rawPath, username);
if (path === '/')
throw new HttpError(400, 'Cannot touch root', {
legacyCode: 'bad_request',
@@ -1467,7 +1479,8 @@ export class FSController extends PuterController {
const entry = await this.#resolveEntryForRequest(body);
await this.#assertAccess(actor, entry.path, 'write');
const renamed = await this.services.fs.rename(entry, newName);
const userId = this.#getActorUserId(req);
const renamed = await this.services.fs.rename(userId, entry, newName);
this.#emitGuiItemUpdated(renamed);
res.json(this.#toClientEntry(renamed));
}
@@ -1511,7 +1524,9 @@ export class FSController extends PuterController {
await this.#resolveEntryForRequest(destinationRef);
await this.#assertAccess(actor, source.path, 'write');
await this.#assertAccess(actor, destinationParent.path, 'write');
if (!isOwnersTrash(source, destinationParent)) {
await this.#assertAccess(actor, destinationParent.path, 'write');
}
const moved = await this.services.fs.move(userId, {
source,
@@ -1623,7 +1638,7 @@ export class FSController extends PuterController {
const ref = {
path:
rawPath !== undefined
? mod.expandTildePath(rawPath, username)
? await this.#resolveClientPath(rawPath, username)
: undefined,
uid:
typeof source.uid === 'string'
@@ -2178,7 +2193,8 @@ export class FSController extends PuterController {
// the ActorUser type. Access via the escape hatch until a proper
// storage-quota mechanism is in place.
const actorUser = req.actor?.user as
Record<string, unknown> | undefined;
| Record<string, unknown>
| undefined;
const candidates = [
this.#toStorageCapacityCandidate(actorUser?.free_storage),
@@ -2191,6 +2207,18 @@ export class FSController extends PuterController {
return Math.max(...candidates);
}
/**
* `#normalizePath`, plus turning a masked share path back into the owner's
* real one. Every client-authored path goes through here.
*/
async #resolveClientPath(path: string, username?: string): Promise<string> {
return resolveSharePath(
this.stores.fsEntry,
Context.get('actor'),
this.#normalizePath(path, username),
);
}
#normalizePath(path: string, username?: string): string {
const trimmedPath = path.trim();
if (trimmedPath.length === 0) {
@@ -2221,7 +2249,7 @@ export class FSController extends PuterController {
return normalizedPath;
}
#normalizeFileMetadataPath(
#expandFileMetadataPath(
req: Request,
fileMetadata: FSEntryWriteInput | undefined,
fallbackSource?: unknown,
@@ -2236,13 +2264,38 @@ export class FSController extends PuterController {
});
}
const username = this.#getActorUsername(req);
return {
...resolvedFileMetadata,
path: this.#normalizePath(resolvedFileMetadata.path, username),
path: this.#normalizePath(
resolvedFileMetadata.path,
this.#getActorUsername(req),
),
};
}
/** As above, plus turning a masked share path into the owner's real one. */
async #normalizeFileMetadataPath(
req: Request,
fileMetadata: FSEntryWriteInput | undefined,
fallbackSource?: unknown,
): Promise<FSEntryWriteInput> {
const expanded = this.#expandFileMetadataPath(
req,
fileMetadata,
fallbackSource,
);
return { ...expanded, path: await this.#unmaskPath(expanded.path) };
}
/** The un-masking half, for callers that already expanded the path. */
async #unmaskPath(path: string): Promise<string> {
return resolveSharePath(
this.stores.fsEntry,
Context.get('actor'),
path,
);
}
#extractGuiMetadata(
input: unknown,
fallback: WriteGuiMetadata | undefined,
@@ -2321,7 +2374,7 @@ export class FSController extends PuterController {
}
const normalizedFileMetadata = options?.pathAlreadyNormalized
? fileMetadata
: this.#normalizeFileMetadataPath(req, fileMetadata);
: await this.#normalizeFileMetadataPath(req, fileMetadata);
if (!normalizedFileMetadata) {
throw new HttpError(400, 'Missing path', {
legacyCode: 'bad_request',
@@ -2484,14 +2537,19 @@ export class FSController extends PuterController {
requestBody: SignedWriteRequest,
response: SignedWriteResponse,
): Promise<void> {
const normalizedPath = this.#normalizePath(
const normalizedPath = await this.#resolveClientPath(
requestBody.fileMetadata.path,
);
// This goes to the *acting* user, not the owner, so a recipient writing
// into a shared folder would otherwise be handed the owner's real path
// — the one thing the masking exists to withhold. There is no row yet,
// so the mask is built from the path the caller already named.
const publishedPath = maskPathForRequest(normalizedPath);
const pendingResponse = {
id: response.objectKey,
uid: response.objectKey,
uuid: response.objectKey,
path: normalizedPath,
path: publishedPath,
name: pathPosix.basename(normalizedPath),
is_dir: false,
content_type: response.contentType,
@@ -2774,7 +2832,10 @@ export class FSController extends PuterController {
user_id: userId,
userId: userId,
item_uid: itemUid,
item_path: itemPath,
// Progress is reported to the acting user, who is not the
// owner when the write lands in a shared folder — publish
// the path in the form they addressed it by.
item_path: maskPathForRequest(itemPath),
...this.#toEventGuiMetadata(guiMetadata),
},
},
@@ -1166,6 +1166,115 @@ describe('LegacyFSController.move', () => {
};
expect(removedPayload.response?.uid).toBe(replaced.uuid);
});
it('lets a share recipient trash an item into the owner’s trash', async () => {
const owner = await makeUser();
const holder = await makeUser();
const ownerName = owner.actor.user!.username!;
const sharedPath = `/${ownerName}/Documents/Contents`;
await withActor(owner.actor, () =>
controller.mkdir(
makeReq({ body: { path: sharedPath }, actor: owner.actor }),
makeRes().res,
),
);
await withActor(owner.actor, () =>
controller.touch(
makeReq({
body: { path: `${sharedPath}/note.txt` },
actor: owner.actor,
}),
makeRes().res,
),
);
const shared = (await server.stores.fsEntry.getEntryByPath(sharedPath))!;
await server.services.acl.setUserUser(
owner.actor,
holder.actor,
{
path: shared.path,
resolveAncestors: () =>
server.services.fs.getAncestorChain(shared.path),
},
'write',
);
const { res, captured } = makeRes();
await withActor(holder.actor, () =>
controller.move(
makeReq({
body: {
source: `${sharedPath}/note.txt`,
destination: `/${ownerName}/Trash`,
},
actor: holder.actor,
}),
res,
),
);
// The move landed in the owner's trash, but the recipient is told so
// in masked form — the owner's real layout stays theirs.
const body = captured.body as { moved: { uid: string; path: string } };
expect(body.moved.path).toBe(
`/${ownerName}/${body.moved.uid}/note.txt`,
);
const moved = await server.stores.fsEntry.getEntryByUuid(
body.moved.uid,
);
expect(moved!.path).toBe(`/${ownerName}/Trash/note.txt`);
});
it('refuses a share recipient moving an item into their own trash', async () => {
const owner = await makeUser();
const holder = await makeUser();
const ownerName = owner.actor.user!.username!;
const holderName = holder.actor.user!.username!;
const sharedPath = `/${ownerName}/Documents/Shared`;
await withActor(owner.actor, () =>
controller.mkdir(
makeReq({ body: { path: sharedPath }, actor: owner.actor }),
makeRes().res,
),
);
await withActor(owner.actor, () =>
controller.touch(
makeReq({
body: { path: `${sharedPath}/theirs.txt` },
actor: owner.actor,
}),
makeRes().res,
),
);
const shared = (await server.stores.fsEntry.getEntryByPath(sharedPath))!;
await server.services.acl.setUserUser(
owner.actor,
holder.actor,
{
path: shared.path,
resolveAncestors: () =>
server.services.fs.getAncestorChain(shared.path),
},
'write',
);
await expect(
withActor(holder.actor, () =>
controller.move(
makeReq({
body: {
source: `${sharedPath}/theirs.txt`,
destination: `/${holderName}/Trash`,
},
actor: holder.actor,
}),
makeRes().res,
),
),
).rejects.toMatchObject({ statusCode: 403 });
});
});
// ── search ──────────────────────────────────────────────────────────
@@ -1586,8 +1695,11 @@ describe('LegacyFSController.writeFile (write IDOR)', () => {
),
);
// The write landed on the signed file …
expect((captured.body as { path?: string }).path).toBe(target);
// The write landed on the signed file — reported masked, since the
// attacker doesn't own it …
expect((captured.body as { path?: string }).path).toBe(
`/${victim.actor.user!.username}/${entry!.uuid}/secret.txt`,
);
// … and never created the attacker-named sibling.
const siblingEntry =
await server.stores.fsEntry.getEntryByPath(sibling);
@@ -1982,6 +2094,50 @@ describe('LegacyFSController.requestAppRootDir', () => {
expect(body.path).toBe(`/${username}/AppData/${appUid}`);
expect(body.is_dir).toBe(true);
});
// `check: true` answers the question without acting on the answer, so that
// asking whether the access is held is not itself a filesystem write.
it('answers check:true without creating the directory', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const appUid = 'app-check-only';
const appActor = makeActor({ ...actor, app: { uid: appUid } });
const rootPath = `/${username}/AppData/${appUid}`;
const { res, captured } = makeRes();
await withActor(appActor, () =>
controller.requestAppRootDir(
makeReq({
body: { app_uid: appUid, access: 'read', check: true },
actor: appActor,
}),
res,
),
);
expect(captured.body).toEqual({ allowed: true });
expect(
await server.stores.fsEntry.getEntryByPath(rootPath),
).toBeFalsy();
});
// The guard runs first either way, so a caller that may not claim it is
// refused rather than told it is allowed.
it('still refuses check:true from a caller that is not the app', async () => {
const { actor } = await makeUser();
const { res } = makeRes();
await expect(
withActor(actor, () =>
controller.requestAppRootDir(
makeReq({
body: { app_uid: 'app-xyz', check: true },
actor,
}),
res,
),
),
).rejects.toMatchObject({ statusCode: 403 });
});
});
// ── checkAppAcl ─────────────────────────────────────────────────────
@@ -37,7 +37,11 @@ import type { PuterRouter } from '../../core/http/PuterRouter.js';
import type { ACLService } from '../../services/acl/ACLService.js';
import { assertActorHasCredits } from '../../services/metering/enforcement.js';
import type { SignedFile } from '../../util/fileSigning.js';
import { verifySignature } from '../../util/fileSigning.js';
import {
NON_OWNER_SIGNATURE_TTL_SECONDS,
verifySignature,
} from '../../util/fileSigning.js';
import { maskEntryPath } from '../../services/fs/sharePathMask.js';
import {
buildHostedBackingDenial,
hostedIndexUrlBackingIsUnavailable,
@@ -66,6 +70,7 @@ import {
asRecord,
assertAccess,
assertCanCreate,
assertCanMoveInto,
getBoolean,
getString,
loadLegacyAssociatedApps,
@@ -484,7 +489,6 @@ export class LegacyFSController extends PuterController {
const rootChildren = await listRootEntries(
actor,
this.stores.fsEntry,
this.services.permission,
);
const rootSuggestions =
await this.services.suggestedApps.getSuggestedAppsForEntries(
@@ -790,12 +794,12 @@ export class LegacyFSController extends PuterController {
source.path,
'write',
);
await assertAccess(
await assertCanMoveInto(
this.services.acl,
this.services.fs,
actor,
destinationParent.path,
'write',
source,
destinationParent,
);
// The v1 wire contract reports the entry an overwrite replaced
@@ -944,7 +948,8 @@ export class LegacyFSController extends PuterController {
'write',
);
const renamed = await this.services.fs.rename(entry, newName);
const userId = this.#getActorUserId(req);
const renamed = await this.services.fs.rename(userId, entry, newName);
await this.#emitGuiEvent('outer.gui.item.updated', renamed);
res.json(await toLegacyEntry(this.clients.event, renamed));
};
@@ -1331,6 +1336,11 @@ export class LegacyFSController extends PuterController {
);
let finalAction: 'read' | 'write' = 'read';
if (action === 'write') {
// The real path, not the masked one: ACL reads the path
// string itself for its short-circuits (own home, an app's
// AppData under another user), and a mask hides the
// `AppData/<appUid>` shape those match on. Nothing here
// publishes the descriptor, so there is nothing to hide.
const writeOk = await this.services.acl.check(
actor,
{
@@ -1354,12 +1364,20 @@ export class LegacyFSController extends PuterController {
);
}
const signed = signEntry(entry, signingCfg);
const signed = signEntry(entry, signingCfg, {
actorUserId: actor.user?.id,
});
if (finalAction !== 'write') {
const { write_url: _, ...rest } = signed;
result.signatures.push({ ...rest, path: entry.path });
result.signatures.push({
...rest,
path: maskEntryPath(entry),
});
} else {
result.signatures.push({ ...signed, path: entry.path });
result.signatures.push({
...signed,
path: maskEntryPath(entry),
});
}
} catch {
// Silently skip unresolvable items.
@@ -1465,8 +1483,10 @@ export class LegacyFSController extends PuterController {
'outer.gui.item.added',
uploadResult.fsEntry,
);
const signed = signEntry(uploadResult.fsEntry, signingCfg);
res.json({ ...signed, path: uploadResult.fsEntry.path });
const signed = signEntry(uploadResult.fsEntry, signingCfg, {
actorUserId: callerActor.user?.id,
});
res.json({ ...signed, path: maskEntryPath(uploadResult.fsEntry) });
return;
}
@@ -1494,7 +1514,12 @@ export class LegacyFSController extends PuterController {
dedupeName: true,
});
await this.#emitGuiEvent('outer.gui.item.added', entry);
res.json({ ...signEntry(entry, signingCfg), path: entry.path });
res.json({
...signEntry(entry, signingCfg, {
actorUserId: callerActor.user?.id,
}),
path: maskEntryPath(entry),
});
return;
}
if (operation === 'rename') {
@@ -1511,9 +1536,18 @@ export class LegacyFSController extends PuterController {
targetEntry.path,
'write',
);
const renamed = await this.services.fs.rename(targetEntry, newName);
const renamed = await this.services.fs.rename(
userId,
targetEntry,
newName,
);
await this.#emitGuiEvent('outer.gui.item.updated', renamed);
res.json({ ...signEntry(renamed, signingCfg), path: renamed.path });
res.json({
...signEntry(renamed, signingCfg, {
actorUserId: callerActor.user?.id,
}),
path: maskEntryPath(renamed),
});
return;
}
if (operation === 'delete' || operation === 'trash') {
@@ -1581,7 +1615,12 @@ export class LegacyFSController extends PuterController {
? { old_path: targetEntry.path }
: undefined,
);
res.json({ ...signEntry(result, signingCfg), path: result.path });
res.json({
...signEntry(result, signingCfg, {
actorUserId: callerActor.user?.id,
}),
path: maskEntryPath(result),
});
return;
}
@@ -1643,11 +1682,18 @@ export class LegacyFSController extends PuterController {
// Directory: return a signed listing of direct children.
// The caller only proved read access, so strip write_url from
// each child to prevent privilege escalation via /writeFile.
//
// Child paths are the owner's real ones — there is no actor here to
// mask for. Accepted: a signature is minted and handed out by the
// owner, so the layout it reveals is theirs to reveal, and the child
// signatures expire; the paths are the only part that outlives them.
if (entry.isDir) {
const children = await this.services.fs.listDirectory(entry.uuid);
const signedChildren = children.map((child) => {
const { write_url: _, ...rest } = signEntry(child, signingCfg);
return { ...rest, path: child.path };
const { write_url: _, ...rest } = signEntry(child, signingCfg, {
ttlSeconds: NON_OWNER_SIGNATURE_TTL_SECONDS,
});
return { ...rest, path: maskEntryPath(child) };
});
res.json(signedChildren);
return;
@@ -1741,6 +1787,7 @@ export class LegacyFSController extends PuterController {
// read. `/writeFile`'s ACL re-check would still block the write, but
// returning `write_url` to a read-only caller is the same
// privilege-leak shape that `/sign` and `/readdir` strip.
// Real path — see the note on the other `acl.check` above.
const writeOk = await this.services.acl.check(
actor,
{
@@ -1754,7 +1801,7 @@ export class LegacyFSController extends PuterController {
const suggested =
(await this.services.suggestedApps?.getSuggestedApps({
name: entry.name,
path: entry.path,
path: maskEntryPath(entry),
})) ?? [];
let token: string | null = null;
@@ -1777,12 +1824,14 @@ export class LegacyFSController extends PuterController {
}
const signingCfg = signingConfigFromAppConfig(this.config);
const signed = signEntry(entry, signingCfg);
const signed = signEntry(entry, signingCfg, {
actorUserId: actor.user?.id,
});
const signature = writeOk
? { ...signed, path: entry.path }
? { ...signed, path: maskEntryPath(entry) }
: (() => {
const { write_url: _, ...rest } = signed;
return { ...rest, path: entry.path };
return { ...rest, path: maskEntryPath(entry) };
})();
res.json({
signature,
@@ -1794,6 +1843,12 @@ export class LegacyFSController extends PuterController {
/**
* POST /auth/request-app-root-dir — an app-under-user requests stat on its
* own app root directory. The app must own itself.
*
* `check: true` answers whether the caller may claim it and stops there,
* for callers asking the question rather than acting on the answer —
* otherwise `puter.perms.check('appRootDir', …)` would provision a
* directory just by being asked. Answering is the whole response: a caller
* that may not claim it gets the 403 below either way.
*/
requestAppRootDir = async (req: Request, res: Response): Promise<void> => {
const actor = this.#requireActor(req);
@@ -1820,6 +1875,11 @@ export class LegacyFSController extends PuterController {
legacyCode: 'unauthorized',
});
if (getBoolean(body, 'check')) {
res.json({ allowed: true });
return;
}
const rootPath = `/${username}/AppData/${appUid}`;
// Auto-create the AppData/<uid> tree on first call.
const entry = await this.services.fs.mkdir(userId, {
@@ -2296,12 +2356,12 @@ export class LegacyFSController extends PuterController {
source.path,
'write',
);
await assertAccess(
await assertCanMoveInto(
this.services.acl,
this.services.fs,
actor,
destinationParent.path,
'write',
source,
destinationParent,
);
const moved = await this.services.fs.move(userId, {
source,
@@ -24,11 +24,13 @@ import {
getBoolean,
getString,
loadLegacyAssociatedApps,
signEntry,
signEntryThumbnail,
signingConfigFromAppConfig,
toLegacyEntry,
} from './legacyFsHelpers.js';
import type { FSEntry } from '../../stores/fs/FSEntry.js';
import { NON_OWNER_SIGNATURE_TTL_SECONDS } from '../../util/fileSigning.js';
const entryWithApp = (associatedAppId: number): FSEntry =>
({ associatedAppId }) as unknown as FSEntry;
@@ -444,3 +446,48 @@ describe('loadLegacyAssociatedApps short-circuit', () => {
expect(seen).toEqual([[4, 5]]);
});
});
describe('signEntry', () => {
const cfg = { secret: 'test-secret', apiBaseUrl: 'https://api.test' };
const entry = {
uuid: 'e-1',
name: 'x.txt',
isDir: false,
size: 1,
accessed: 0,
modified: 0,
created: 0,
userId: 7,
};
// signFile ceils its own clock read, so allow a second of slack.
const inSeconds = (expires: number) =>
expires - Math.ceil(Date.now() / 1000);
it('leaves an owner’s signature effectively permanent', () => {
const signed = signEntry(entry, cfg, { actorUserId: 7 });
expect(inSeconds(signed.expires)).toBeGreaterThan(
NON_OWNER_SIGNATURE_TTL_SECONDS * 10,
);
});
it('expires a signature over someone else’s entry', () => {
const signed = signEntry(entry, cfg, { actorUserId: 8 });
const ttl = inSeconds(signed.expires);
expect(ttl).toBeLessThanOrEqual(NON_OWNER_SIGNATURE_TTL_SECONDS);
expect(ttl).toBeGreaterThan(0);
});
it('keeps the old default when no actor is supplied', () => {
expect(inSeconds(signEntry(entry, cfg).expires)).toBeGreaterThan(
NON_OWNER_SIGNATURE_TTL_SECONDS * 10,
);
});
it('honours an explicit ttl over the ownership check', () => {
const signed = signEntry(entry, cfg, {
actorUserId: 7,
ttlSeconds: 30,
});
expect(inSeconds(signed.expires)).toBeLessThanOrEqual(30);
});
});
+66 -6
View File
@@ -30,10 +30,16 @@ import { HttpError } from '../../core/http/HttpError.js';
import {
resolveNode,
normalizeAbsolutePath,
isOwnersTrash,
joinChildPath,
expandTildePath,
} from '../../services/fs/resolveNode.js';
import {
maskEntryPath,
resolveSharePath,
} from '../../services/fs/sharePathMask.js';
import {
NON_OWNER_SIGNATURE_TTL_SECONDS,
signFile,
type SigningConfig,
type SignedFile,
@@ -93,6 +99,22 @@ export function getBoolean(
// username, read from the ALS-backed Context. Legacy clients send tilde-
// rooted paths (e.g. `~/AppData/<app-uid>/...`); FSController does the same
// expansion via its own `#normalizePath` helper.
/**
* Everything a client-authored path needs before it names a row: `~` expanded,
* and a masked share path turned back into the owner's real one.
*/
export async function expandClientPath(
fsEntryStore: FSEntryStore,
raw: string,
username?: string,
): Promise<string> {
return resolveSharePath(
fsEntryStore,
Context.get('actor'),
expandTildePath(raw, username),
);
}
export async function resolveV1Selector(
fsEntryStore: FSEntryStore,
raw: unknown,
@@ -106,7 +128,7 @@ export async function resolveV1Selector(
if (typeof raw === 'string') {
const isPath = raw.startsWith('/') || raw.startsWith('~');
const ref = isPath
? { path: expandTildePath(raw, username) }
? { path: await expandClientPath(fsEntryStore, raw, username) }
: { uid: raw };
const entry = await resolveNode(fsEntryStore, ref, { required: true });
if (!entry)
@@ -138,7 +160,7 @@ export async function resolveV1Selector(
const ref = {
path:
rawPath !== undefined
? expandTildePath(rawPath, username)
? await expandClientPath(fsEntryStore, rawPath, username)
: undefined,
uid:
typeof record.uid === 'string'
@@ -260,6 +282,24 @@ export async function assertCanCreate(
await assertAccess(aclService, fsService, actor, parentForCheck, 'write');
}
/** `write` on the destination parent, unless it is the entry's own Trash. */
export async function assertCanMoveInto(
aclService: ACLService,
fsService: FSService,
actor: Actor,
source: FSEntry,
destinationParent: FSEntry,
): Promise<void> {
if (isOwnersTrash(source, destinationParent)) return;
await assertAccess(
aclService,
fsService,
actor,
destinationParent.path,
'write',
);
}
// -- Response shaping ------------------------------------------------
type AppRowLookup = {
@@ -411,7 +451,10 @@ export async function toLegacyEntry(
appsById?: Map<number, Record<string, unknown>>;
} = {},
): Promise<Record<string, unknown>> {
const dirname = pathPosix.dirname(entry.path);
// Someone else's entry is published under its masked path; the owner's
// real one, and everything above the share, stays server-side.
const publishedPath = maskEntryPath(entry);
const dirname = pathPosix.dirname(publishedPath);
const mimeType = fsEntryMimeType(entry);
const pathComponents = entry.path.split('/');
@@ -424,7 +467,7 @@ export async function toLegacyEntry(
uuid: entry.uuid,
parent_id: entry.parentUid,
parent_uid: entry.parentUid,
path: entry.path,
path: publishedPath,
dirname,
dirpath: dirname,
name: entry.name,
@@ -520,7 +563,13 @@ export function signingConfigFromAppConfig(config: IConfig): SigningConfig {
return { secret, apiBaseUrl };
}
/** Convenience wrapper: turn an FSEntry into a signed-file response object. */
/**
* Convenience wrapper: turn an FSEntry into a signed-file response object.
*
* Pass `actorUserId` so a signature over someone else's entry — a shared file —
* expires rather than outliving the share (see
* NON_OWNER_SIGNATURE_TTL_SECONDS).
*/
export function signEntry(
entry: {
uuid: string;
@@ -530,8 +579,19 @@ export function signEntry(
accessed: number | null;
modified: number;
created: number | null;
userId?: number;
},
config: SigningConfig,
opts: { actorUserId?: number; ttlSeconds?: number } = {},
): SignedFile {
return signFile(entry as Parameters<typeof signFile>[0], config);
const isForeign =
typeof opts.actorUserId === 'number' &&
typeof entry.userId === 'number' &&
entry.userId !== opts.actorUserId;
const ttlSeconds =
opts.ttlSeconds ??
(isForeign ? NON_OWNER_SIGNATURE_TTL_SECONDS : undefined);
return signFile(entry as Parameters<typeof signFile>[0], config, {
...(ttlSeconds === undefined ? {} : { ttlSeconds }),
});
}
+19 -2
View File
@@ -661,6 +661,7 @@ if (window.opener) {
provider: string,
userinfo: { sub: string; email?: unknown; [k: string]: unknown },
referrer?: string | null,
attempt = 0,
): Promise<
| { error: string; code?: string; requestCode?: string }
| {
@@ -679,8 +680,12 @@ if (window.opener) {
const claimedEmail =
typeof userinfo.email === 'string' ? userinfo.email : null;
if (claimedEmail) {
const byEmail =
await this.services.oidc.findUserByEmail(claimedEmail);
// On the retry after a lost race, read the primary — the winning
// row may be younger than the replica snapshot.
const byEmail = await this.services.oidc.findUserByEmail(
claimedEmail,
{ force: attempt > 0 },
);
if (byEmail) {
if (!byEmail.email_confirmed) {
return {
@@ -707,6 +712,18 @@ if (window.opener) {
userinfo as { sub: string; email?: string },
referrer,
);
// A concurrent callback (a second tab, a provider retry) created the
// account between step 2 and the insert. Nothing went wrong for the
// user — start over and we'll find the winner at step 1 or 2. One retry
// only: a second miss means something other than a race is going on.
if (outcome.raced && attempt === 0) {
return this.#resolveOrCreateOIDCUser(
provider,
userinfo,
referrer,
attempt + 1,
);
}
if (!outcome.success || !outcome.user) {
return {
error: outcome.error ?? 'Account creation failed.',
@@ -0,0 +1,157 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
/**
* The vendor-compatible wire routes are subscriber-only, over real HTTP.
*
* The gate's decision is unit-tested in
* `services/metering/enforcement.test.ts`; what this covers is that the route
* option reaches the middleware chain on these four routes, that it sits behind
* the credential gates (a session token is still turned away for being the
* wrong credential, not for the plan), and that the model catalogue stays open
* — clients fetch it before they have any reason to have authenticated at all.
*/
describe('AI wire routes require a subscription', () => {
let env: PuterTestEnv;
const WIRE_ROUTES = [
'/puterai/openai/v1/chat/completions',
'/puterai/openai/v1/completions',
'/puterai/openai/v1/responses',
'/puterai/anthropic/v1/messages',
];
/**
* The plan the test user is on. A single resolver reads this, since
* resolvers are registered for the life of the service and can't be taken
* back off — flipping a variable is what keeps one test's plan out of the
* next one's.
*/
let plan: string | null = null;
let userUuid = '';
beforeAll(async () => {
env = await setupPuterTestEnv();
const row = await env.server.stores.user.getByUsername(
env.users.user.username,
);
userUuid = row!.uuid;
// Paid plans are registered by an extension in production, so a
// resolver alone resolves to nothing here — the policy has to exist
// for the id to be found.
env.server.services.metering.registerPolicy({
id: 'business',
monthUsageAllowance: 45 * 1_000_000 * 100,
monthlyStorageAllowance: 1024 ** 3,
});
env.server.services.metering.registerSubscriptionResolver((actor) =>
actor.user?.uuid === userUuid ? plan : null,
);
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const call = (route: string, token: string) =>
fetch(new URL(route, env.apiOrigin), {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${token}`,
Origin: env.apiOrigin,
},
body: JSON.stringify({
model: 'fake',
provider: 'fake-chat',
prompt: 'hi',
messages: [{ role: 'user', content: 'hi' }],
max_tokens: 16,
}),
});
/** Put the test user on a plan for the duration of one block. */
const withSubscription = async <T>(
id: string | null,
fn: () => Promise<T>,
): Promise<T> => {
const metering = env.server.services.metering;
plan = id;
metering.invalidateActorSubscription(userUuid);
try {
return await fn();
} finally {
plan = null;
metering.invalidateActorSubscription(userUuid);
}
};
it('turns a free account away with 402 subscription_required', async () => {
for (const route of WIRE_ROUTES) {
const res = await call(route, env.users.user.apiToken);
expect(res.status, route).toBe(402);
expect(await res.json(), route).toMatchObject({
code: 'subscription_required',
});
}
});
it('lets a subscriber past the gate', async () => {
await withSubscription('business', async () => {
for (const route of WIRE_ROUTES) {
const res = await call(route, env.users.user.apiToken);
// Whether the fake provider answers is the controller's
// business; all this asserts is that the plan gate is not
// what stopped the request.
expect(res.status, route).not.toBe(402);
}
});
});
it('answers a session token on credential shape, not plan', async () => {
// Credential gates run first, so a caller with the wrong token shape
// never learns anything about the account's plan.
const res = await call(WIRE_ROUTES[0]!, env.users.user.token);
expect(res.status).toBe(403);
expect(await res.json()).toMatchObject({
code: 'app_or_api_token_required',
});
});
it('leaves the model catalogue open', async () => {
const res = await fetch(new URL('/puterai/chat/models', env.apiOrigin));
expect(res.status).toBe(200);
});
it('leaves the video proxy outside the plan gate', async () => {
// Unsigned, so this is turned away on the signature — the point is
// that a free account gets that answer rather than a 402. The proxy
// delivers a video the account already paid to generate.
const res = await fetch(
new URL(
'/puterai/video/proxy?url=https://x.test/v.mp4',
env.apiOrigin,
),
);
expect(res.status).not.toBe(402);
});
});
@@ -195,7 +195,9 @@ describe('PuterAIController.registerRoutes', () => {
// `requireUserActor` keeps apps out, `allowFullAccessToken` admits
// the PAT, and `noUserSession` rejects the account session ("root")
// token. Each route also carries `requireVerified` (fresh accounts
// must confirm their email before using the AI wire surface) and
// must confirm their email before using the AI wire surface),
// `requireSubscription` (the vendor-compatible surface is a paid
// feature; free accounts reach the same models via /drivers/call) and
// the shared per-tier AI rate-limit / concurrency policy — these
// routes bypass the `/drivers/call` dispatch (where the
// driver-declared limits are enforced), so without the route gates
@@ -214,6 +216,7 @@ describe('PuterAIController.registerRoutes', () => {
allowFullAccessToken: true,
noUserSession: true,
requireVerified: true,
requireSubscription: true,
rateLimit: {
...AI_RATE_LIMIT.default,
scope: 'driver:puter-chat-completion:complete',
@@ -244,6 +247,38 @@ describe('PuterAIController.registerRoutes', () => {
});
});
it('leaves the catalogue and the video proxy outside the plan gate', () => {
const calls: Array<{ path: string; opts: RouteOptions }> = [];
const router = {
post: vi.fn(() => router),
get: vi.fn((path: string, opts: RouteOptions) => {
calls.push({ path, opts });
return router;
}),
};
controller.registerRoutes(router as never);
// The catalogue is fetched before a client has any reason to
// authenticate, and the proxy streams a video the account already
// paid to generate to whoever holds the signed link — neither has a
// plan to read, let alone one to charge for.
const openPaths = [
'/puterai/chat/models',
'/puterai/chat/models/details',
'/puterai/image/models',
'/puterai/image/models/details',
'/puterai/video/models',
'/puterai/video/models/details',
'/puterai/video/proxy',
];
for (const path of openPaths) {
const route = calls.find((c) => c.path === path);
expect(route, path).toBeDefined();
expect(route?.opts.requireSubscription, path).toBeUndefined();
}
});
it('keys the proxy-route AI limits by user uuid so they share the /drivers/call buckets', () => {
const calls: Array<{ path: string; opts: RouteOptions }> = [];
const router = {
@@ -49,7 +49,11 @@ const GEMINI_DOWNLOAD_BASE =
* All routes live on `subdomain: 'api'` and require a full-access API token
* minted from the dashboard (user-scoped worker tokens also pass — workers are
* never treated as root tokens). Apps, scoped tokens, and account session
* ("root") tokens are rejected.
* ("root") tokens are rejected. The vendor-compatible routes additionally
* require the account to be on a paid plan. The model listings stay open —
* clients fetch the catalogue before they have any reason to authenticate — and
* so does the video proxy, whose signed URLs are handed out by a generation the
* account already paid for.
*/
export class PuterAIController extends PuterController {
registerRoutes(router: PuterRouter): void {
@@ -79,6 +83,13 @@ export class PuterAIController extends PuterController {
allowFullAccessToken: true,
noUserSession: true,
requireVerified: true,
// The vendor-compatible wire surface is a paid feature: it is
// what makes Puter a drop-in for a metered vendor API, and an
// account on a free plan reaches the same models through
// `/drivers/call` (and puter.js) without it. Free accounts get
// 402 `subscription_required` here rather than a rate limit,
// so the answer is "upgrade", not "slow down".
requireSubscription: true,
rateLimit: {
...AI_RATE_LIMIT.default!,
scope: aiPolicyScope,
@@ -166,7 +177,9 @@ export class PuterAIController extends PuterController {
// -- Video URL proxy -----------------------------------------
// Reverse-proxies AI-generated video URLs that can't be given
// directly to the client (auth-gated provider downloads). The
// URL itself is HMAC-signed, so no additional auth gate.
// URL itself is HMAC-signed, so no additional auth gate — and no
// plan gate either: it delivers a video the account already paid to
// generate, to whoever holds the link.
router.get(
'/puterai/video/proxy',
{
@@ -0,0 +1,554 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
/**
* Route-level coverage for the sharing endpoints. The service unit tests drive
* the semantics; this suite exists to catch a route that was never registered,
* a gate that rejects a legitimate request, and anything the response shape
* leaks.
*/
describe('share endpoints over HTTP', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const post = (path: string, token: string, body: unknown) =>
fetch(new URL(path, env.apiOrigin), {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${token}`,
},
body: JSON.stringify(body),
});
const get = (path: string, token: string, params: Record<string, string>) => {
const url = new URL(path, env.apiOrigin);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
return fetch(url, { headers: { authorization: `Bearer ${token}` } });
};
/** A file in the owner's home. Written directly — these tests are about
* the share routes, not the upload path. */
const makeFile = async (owner: { username: string }) => {
const uid = crypto.randomUUID();
const name = `share-http-${uid.slice(0, 8)}.txt`;
const path = `/${owner.username}/${name}`;
const user = await env.server.stores.user.getByUsername(owner.username);
await env.server.clients.db.write(
'INSERT INTO `fsentries` (`uuid`, `name`, `path`, `user_id`, `is_dir`, `modified`) VALUES (?, ?, ?, ?, 0, ?)',
[uid, name, path, user!.id, Math.floor(Date.now() / 1000)],
);
return { uid, path, name };
};
// Masking hides where an item sits, not who may open it.
it('will not let a masked path reach an unshared sibling', async () => {
const owner = env.users.user;
const recipient = env.users.other;
const shared = await makeFile(owner);
const secret = await makeFile(owner);
await post('/share', owner.token, {
recipients: [recipient.username],
items: [{ uid: shared.uid }],
mode: 'read',
});
const read = (path: string) =>
fetch(
`${env.apiOrigin}/read?${new URLSearchParams({ file: path })}`,
{
headers: {
authorization: `Bearer ${recipient.token}`,
origin: env.apiOrigin,
},
},
);
const masked = `/${owner.username}/${shared.uid}/${shared.name}`;
expect((await read(masked)).status).toBe(200);
for (const attempt of [
// The shared item's root, renamed to the sibling.
`/${owner.username}/${shared.uid}/${secret.name}`,
// The sibling's own uuid, as if it had leaked.
`/${owner.username}/${secret.uid}/${secret.name}`,
// Back out of the root the uuid vouched for.
`/${owner.username}/${shared.uid}/${shared.name}/../${secret.name}`,
// The owner's real path, named outright.
secret.path,
]) {
const res = await read(attempt);
expect(res.status, `reachable: ${attempt}`).not.toBe(200);
}
});
it('shares an item, lists it for the recipient, then revokes it', async () => {
const owner = env.users.user;
const recipient = env.users.other;
const file = await makeFile(owner);
const shareRes = await post('/share', owner.token, {
recipients: [recipient.username],
items: [{ uid: file.uid }],
mode: 'read',
});
expect(shareRes.status).toBe(200);
const shareBody = (await shareRes.json()) as {
status: string;
results: Array<{ status: string; mode?: string; name?: string }>;
};
expect(shareBody.status).toBe('success');
expect(shareBody.results[0].mode).toBe('read');
expect(shareBody.results[0].name).toBe(file.name);
const listRes = await get(
'/share/shared-with-me',
recipient.token,
{ includeTotal: 'true' },
);
expect(listRes.status).toBe(200);
const listed = (await listRes.json()) as {
items: Array<Record<string, unknown>>;
total?: number;
};
const row = listed.items.find((i) => i.uid_entry === file.uid);
expect(row).toBeDefined();
expect(row?.issuer).toBe(owner.username);
expect(row?.mode).toBe('read');
expect(typeof listed.total).toBe('number');
// Nothing internal rides along in the response.
for (const key of ['issuer_user_id', 'holder_user_id', 'fsentry_id']) {
expect(row).not.toHaveProperty(key);
}
const revokeRes = await post('/share/revoke', owner.token, {
recipients: [recipient.username],
items: [{ uid: file.uid }],
});
expect(revokeRes.status).toBe(200);
expect(await revokeRes.json()).toMatchObject({ revoked: 1 });
const afterRes = await get('/share/shared-with-me', recipient.token, {});
const after = (await afterRes.json()) as {
items: Array<Record<string, unknown>>;
};
expect(after.items.find((i) => i.uid_entry === file.uid)).toBeUndefined();
});
it('revokes every item in the request, not just the first', async () => {
const owner = env.users.user;
const recipient = env.users.other;
const fileA = await makeFile(owner);
const fileB = await makeFile(owner);
for (const file of [fileA, fileB]) {
const res = await post('/share', owner.token, {
recipients: [recipient.username],
items: [{ uid: file.uid }],
mode: 'read',
});
expect(res.status).toBe(200);
}
// A truncated revoke is a silent security failure: the caller is told
// "success" while items after the first keep their grants.
const revokeRes = await post('/share/revoke', owner.token, {
recipients: [recipient.username],
items: [{ uid: fileA.uid }, { uid: fileB.uid }],
});
expect(revokeRes.status).toBe(200);
expect(await revokeRes.json()).toMatchObject({
status: 'success',
revoked: 2,
});
const afterRes = await get('/share/shared-with-me', recipient.token, {});
const after = (await afterRes.json()) as {
items: Array<Record<string, unknown>>;
};
for (const file of [fileA, fileB]) {
expect(
after.items.find((i) => i.uid_entry === file.uid),
).toBeUndefined();
}
});
it('accepts tilde-rooted paths the way the FS routes do', async () => {
const owner = env.users.user;
const recipient = env.users.other;
const file = await makeFile(owner);
const tildePath = `~/${file.path.split('/').pop()}`;
const shareRes = await post('/share', owner.token, {
recipients: [recipient.username],
items: [tildePath],
mode: 'read',
});
expect(shareRes.status).toBe(200);
expect(await shareRes.json()).toMatchObject({ status: 'success' });
const listRes = await get('/share/shares', owner.token, {
path: tildePath,
});
expect(listRes.status).toBe(200);
const listed = (await listRes.json()) as {
items: Array<{ holder: string }>;
};
expect(
listed.items.some((i) => i.holder === recipient.username),
).toBe(true);
const revokeRes = await post('/share/revoke', owner.token, {
recipients: [recipient.username],
items: [tildePath],
});
expect(revokeRes.status).toBe(200);
expect(await revokeRes.json()).toMatchObject({ revoked: 1 });
});
it('tells the recipient over the wire, once for the whole batch', async () => {
const owner = env.users.user;
const recipient = env.users.other;
const [a, b] = [await makeFile(owner), await makeFile(owner)];
// These accounts are shared between tests, so both things that would
// otherwise decide this outcome are cleared first: the budgets that
// silence a repeat interruption, and any notification still open for
// this recipient, which a new share folds into instead of creating one.
// What's left under test is the batching.
const [issuer, holder] = await Promise.all([
env.server.stores.user.getByUsername(owner.username),
env.server.stores.user.getByUsername(recipient.username),
]);
await env.server.clients.redis.del(
`rate:share:notify:pair:${issuer!.id}:${holder!.id}`,
`rate:share:notify:pair-day:${issuer!.id}:${holder!.id}`,
`rate:share:notify:to:${holder!.id}`,
`rate:share:notify:to-day:${holder!.id}`,
);
for (const row of await env.server.stores.notification.listByUserId(
holder!.id,
{ filter: 'unacknowledged' },
)) {
await env.server.stores.notification.markAcknowledged(
row.uid,
holder!.id,
);
}
const seen: Array<{ userIds: number[]; payload: Record<string, unknown> }> = [];
const notification = env.server.services.notification;
const original = notification.notify.bind(notification);
notification.notify = async (userIds, payload) => {
seen.push({ userIds, payload });
return original(userIds, payload);
};
try {
const res = await post('/share', owner.token, {
items: [{ path: a.path }, { path: b.path }],
recipients: [{ username: recipient.username }],
mode: 'read',
});
expect(res.status).toBe(200);
// Delivery is off the response path, so it may land just after.
await vi.waitFor(() => expect(seen.length).toBeGreaterThan(0), {
timeout: 5000,
});
} finally {
notification.notify = original;
}
// Two items, one recipient — one notification carrying the count.
expect(seen).toHaveLength(1);
expect(seen[0].payload).toMatchObject({
source: 'sharing',
fields: { count: 2 },
});
});
it('reports per-pair outcomes when only some recipients resolve', async () => {
const owner = env.users.user;
const file = await makeFile(owner);
const res = await post('/share', owner.token, {
recipients: [env.users.other.username, 'nosuchuser-zzz'],
items: [{ uid: file.uid }],
mode: 'read',
});
expect(res.status).toBe(200);
const body = (await res.json()) as {
status: string;
results: Array<{ status: string; recipient: string }>;
};
expect(body.status).toBe('mixed');
expect(body.results).toHaveLength(2);
expect(
body.results.find((r) => r.recipient === 'nosuchuser-zzz')?.status,
).toBe('error');
});
it('lists who can reach an item for its owner', async () => {
const owner = env.users.user;
const recipient = env.users.other;
const file = await makeFile(owner);
await post('/share', owner.token, {
recipients: [recipient.username],
items: [{ uid: file.uid }],
mode: 'write',
});
const res = await get('/share/shares', owner.token, { uid: file.uid });
expect(res.status).toBe(200);
const body = (await res.json()) as {
items: Array<{ holder: string; mode: string }>;
};
expect(body.items).toHaveLength(1);
expect(body.items[0].holder).toBe(recipient.username);
expect(body.items[0].mode).toBe('write');
});
it('hides an item from a stranger asking who can reach it', async () => {
const owner = env.users.user;
const file = await makeFile(owner);
const res = await get('/share/shares', env.users.other.token, {
uid: file.uid,
});
expect(res.status).toBe(404);
});
it('runs a duplicated pair once, and answers for both positions', async () => {
const owner = env.users.user;
const file = await makeFile(owner);
const invitee = `dup-${crypto.randomUUID().slice(0, 8)}@puter.local`;
// The same pair twice raced itself into two pending rows; it must
// execute once, with the one outcome reported in both positions.
const res = await post('/share', owner.token, {
recipients: [invitee],
items: [{ uid: file.uid }, { uid: file.uid }],
mode: 'read',
});
expect(res.status).toBe(200);
const body = (await res.json()) as {
status: string;
results: Array<{ status: string; uid?: string }>;
};
expect(body.results).toHaveLength(2);
expect(body.results[0].status).toBe('pending');
expect(body.results[1]).toEqual(body.results[0]);
const rows = await env.server.stores.share.listPendingByEmail(invitee);
expect(rows).toHaveLength(1);
});
it('rejects an unauthenticated share', async () => {
const res = await fetch(new URL('/share', env.apiOrigin), {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ recipients: ['x'], items: ['y'] }),
});
expect(res.status).toBeGreaterThanOrEqual(400);
expect(res.status).toBeLessThan(500);
});
it('caps how many recipients one request can reach', async () => {
const owner = env.users.user;
const file = await makeFile(owner);
const many = Array.from({ length: 64 }, (_, i) => `user-${i}`);
const res = await post('/share', owner.token, {
recipients: many,
items: [{ uid: file.uid }],
mode: 'read',
});
expect(res.status).toBe(400);
expect(await res.json()).toMatchObject({
code: 'too_many_recipients',
});
});
it('caps how many items one request can carry', async () => {
const owner = env.users.user;
const many = Array.from({ length: 128 }, () => ({
uid: crypto.randomUUID(),
}));
const res = await post('/share', owner.token, {
recipients: [env.users.other.username],
items: many,
mode: 'read',
});
expect(res.status).toBe(400);
expect(await res.json()).toMatchObject({ code: 'too_many_items' });
});
it('rejects a request with no recipients or no items', async () => {
const owner = env.users.user;
const file = await makeFile(owner);
expect(
(await post('/share', owner.token, { items: [{ uid: file.uid }] }))
.status,
).toBe(400);
expect(
(
await post('/share', owner.token, {
recipients: [env.users.other.username],
})
).status,
).toBe(400);
});
it('blocks a sender, refuses their share, then unblocks them', async () => {
const owner = env.users.user;
const recipient = env.users.other;
const file = await makeFile(owner);
const blocked = await post('/share/blocks', recipient.token, {
username: owner.username,
});
expect(blocked.status).toBe(200);
expect(await blocked.json()).toMatchObject({
username: owner.username,
blocked: true,
created: true,
});
const refused = await post('/share', owner.token, {
recipients: [recipient.username],
items: [{ uid: file.uid }],
mode: 'read',
});
// A per-pair outcome, so the envelope reports it rather than the status.
expect(refused.status).toBe(200);
expect(await refused.json()).toMatchObject({
status: 'aborted',
results: [{ status: 'error', code: 'recipient_not_accepting_shares' }],
});
const listed = await get('/share/blocks', recipient.token, {});
expect(listed.status).toBe(200);
const body = (await listed.json()) as {
items: Array<Record<string, unknown>>;
};
const row = body.items.find((i) => i.username === owner.username);
expect(row).toBeDefined();
expect(typeof row?.created_at).toBe('number');
// Nothing internal rides along.
for (const key of ['blocker_user_id', 'blocked_user_id', 'id']) {
expect(row).not.toHaveProperty(key);
}
const unblocked = await fetch(new URL('/share/blocks', env.apiOrigin), {
method: 'DELETE',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${recipient.token}`,
},
body: JSON.stringify({ username: owner.username }),
});
expect(unblocked.status).toBe(200);
expect(await unblocked.json()).toMatchObject({ unblocked: true });
const shared = await post('/share', owner.token, {
recipients: [recipient.username],
items: [{ uid: file.uid }],
mode: 'read',
});
expect(await shared.json()).toMatchObject({ status: 'success' });
});
it('requires a username to block', async () => {
const res = await post('/share/blocks', env.users.other.token, {});
expect(res.status).toBe(400);
expect(await res.json()).toMatchObject({ code: 'bad_request' });
});
it('refuses every sender while the blanket switch is on', async () => {
const owner = env.users.user;
const recipient = env.users.other;
const file = await makeFile(owner);
const del = (body: unknown) =>
fetch(new URL('/share/blocks', env.apiOrigin), {
method: 'DELETE',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${recipient.token}`,
},
body: JSON.stringify(body),
});
const on = await post('/share/blocks', recipient.token, { all: true });
expect(on.status).toBe(200);
expect(await on.json()).toMatchObject({ all: true, blocked: true });
const listed = await get('/share/blocks', recipient.token, {});
expect(await listed.json()).toMatchObject({ all: true, items: [] });
const refused = await post('/share', owner.token, {
recipients: [recipient.username],
items: [{ uid: file.uid }],
mode: 'read',
});
expect(await refused.json()).toMatchObject({
status: 'aborted',
results: [
{ status: 'error', code: 'recipient_not_accepting_shares' },
],
});
const off = await del({ all: true });
expect(off.status).toBe(200);
expect(await off.json()).toMatchObject({ all: false, blocked: false });
const shared = await post('/share', owner.token, {
recipients: [recipient.username],
items: [{ uid: file.uid }],
mode: 'read',
});
expect(await shared.json()).toMatchObject({ status: 'success' });
// Cleaned up so a later assertion on this recipient isn't reading
// state this test left behind.
expect((await del({ all: true })).status).toBe(200);
});
it('keeps one caller\'s blocklist out of another\'s', async () => {
const res = await get('/share/blocks', env.users.admin.token, {});
const body = (await res.json()) as { all: boolean; items: unknown[] };
expect(body).toEqual({ all: false, items: [] });
});
});
+557 -345
View File
@@ -17,381 +17,593 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
// import type { Request, Response } from 'express';
// import { HttpError } from '../../core/http/HttpError.js';
import type { PuterRouter } from '../../core/http/PuterRouter.js';
import type { Request, Response } from 'express';
import type { Actor } from '../../core/actor.js';
import { Controller, Delete, Get, Post } from '../../core/http/decorators.js';
import { HttpError, isHttpError } from '../../core/http/HttpError.js';
import type {
ResolvedShare,
ShareRecipient,
ShareTarget,
} from '../../services/share/ShareService.js';
import { expandTildePath } from '../../services/fs/resolveNode.js';
import { signEntryThumbnail } from '../fs/legacyFsHelpers.js';
import { runWithConcurrencyLimitSettled } from '../../util/concurrency.js';
import { normalizeLimit } from '../../util/pagination.js';
import { PuterController } from '../types.js';
// const SHARE_TOKEN_TYPE = 'share';
// const SHARE_TOKEN_EXPIRY = '14d';
/**
* Two windows: a burst ceiling, and a daily one so a slow drip can't add up to
* a mail-merge. Neither bounds _shares_ — one request carries many — which is
* what `ShareService`'s per-day quota is for.
*/
const SHARE_LIMIT = [
{ scope: 'share:mutate', limit: 60, window: 60_000, key: 'user' as const },
{
scope: 'share:mutate-daily',
limit: 500,
window: 24 * 60 * 60_000,
key: 'user' as const,
},
];
const SHARE_LIST_LIMIT = {
scope: 'share:list',
limit: 600,
window: 60_000,
key: 'user' as const,
};
/** Distinct (holder, item) pairs run together; see the note on grouping below. */
const SHARE_CONCURRENCY = 8;
const LIST_LIMIT_CAP = 200;
/**
* Share link endpoints — check, apply, and request access to pending shares.
* The main `POST /share` creation endpoint is also here.
*
* Shares are permission grants addressed to an email. When the recipient
* doesn't have a Puter account yet, the share row lives in the `share` table
* until they sign up and apply it. When they DO have an account, permissions
* are granted immediately and no row is stored.
* Caps on one request's fan-out. Recipients matter most: that number is how
* many people a single call can reach, so it stays small by default and only
* moves by configuration.
*/
export const DEFAULT_MAX_RECIPIENTS = 10;
export const DEFAULT_MAX_ITEMS = 50;
/**
* A success carries the created share; a failure carries why. `pending` is an
* invite: recorded, but not access anyone holds yet.
*/
interface ShareOutcome {
recipient: string;
status: 'success' | 'error' | 'pending';
path?: string;
uid?: string;
mode?: string;
uid_entry?: string;
is_dir?: boolean;
issuer?: string | null;
holder?: string | null;
created_at?: unknown;
message?: string;
code?: string;
}
/**
* Sharing endpoints. `ShareService` owns the semantics; this layer parses
* input, bounds fan-out, and shapes responses.
*
* Apps and tokens are admitted rather than gated out, because `ShareService`
* bounds them properly: authority to share comes from the user behind the
* actor, and the actor must additionally reach the node in its own right. An
* app therefore shares its own AppData and the files it was given, and nothing
* else its user happens to own.
*/
@Controller('/share')
export class ShareController extends PuterController {
registerRoutes(_router: PuterRouter): void {
// const api = { subdomain: 'api' } as const;
// router.post('/sharelink/check', api, this.#check);
// router.post(
// '/sharelink/apply',
// { ...api, requireAuth: true },
// this.#apply,
// );
// router.post(
// '/sharelink/request',
// { ...api, requireAuth: true },
// this.#request,
// );
// router.post('/share', { ...api, requireAuth: true }, this.#share);
/**
* POST /share — grant `mode` on one or more items to one or more
* recipients. Partial success is the contract: each pair reports its own
* outcome and the envelope summarizes.
*/
@Post('', {
subdomain: 'api',
requireVerified: true,
rateLimit: SHARE_LIMIT,
})
async createShares(req: Request, res: Response): Promise<void> {
const actor = this.#requireActor(req);
const body = this.#body(req);
const recipients = this.#recipients(body);
const items = this.#items(body, actor);
const mode = typeof body.mode === 'string' ? body.mode : 'read';
// Every (recipient, item) pair is a distinct (holder, entry) key, so
// they can run together. Two writes to the *same* pair could not —
// setUserUser is a read-modify-write, and a duplicated invite races
// itself into two pending rows — so duplicates in the request execute
// once and every original position reports that one outcome.
const pairs = recipients.flatMap((recipient) =>
items.map((item) => ({ recipient, item })),
);
const { unique, indexOf } = this.#dedupePairs(pairs);
const settledUnique = await runWithConcurrencyLimitSettled(
unique,
SHARE_CONCURRENCY,
async ({ recipient, item }) => {
const share = await this.services.share.share(actor, {
...item,
recipient,
mode: mode as never,
});
return share;
},
);
const settled = indexOf.map((i) => settledUnique[i]);
const results: ShareOutcome[] = await Promise.all(
settled.map(async (outcome, index) => {
const { recipient, item } = pairs[index];
const label = recipient.email ?? recipient.username ?? '';
if (outcome.status === 'fulfilled') {
// The whole share, not just an acknowledgement, so a caller
// needn't re-read to learn what it created.
const share = outcome.value as ResolvedShare;
return {
...(await this.#toClientShare(share)),
recipient: label,
status: share.pending ? 'pending' : 'success',
};
}
return {
recipient: label,
...(item.path ? { path: item.path } : {}),
status: 'error',
...this.#errorShape(outcome.reason),
};
}),
);
// Off the response path: a share that landed must not be reported as
// failed because telling the recipient didn't. Fanned out from the
// unique outcomes, so a duplicated pair is not counted twice.
void this.services.shareNotification.notifyShared(
actor,
settledUnique
.filter((o) => o.status === 'fulfilled')
.map((o) => (o as PromiseFulfilledResult<ResolvedShare>).value),
);
const succeeded = results.filter((r) => r.status !== 'error').length;
res.json({
status:
succeeded === results.length
? 'success'
: succeeded > 0
? 'mixed'
: 'aborted',
results,
});
}
// // -- POST /sharelink/check ---------------------------------------
// // Public — verify a share token from an email link.
/**
* POST /share/revoke — withdraw recipients' access to items. Same fan-out
* contract as POST /share: every (recipient, item) pair is its own revoke
* with its own outcome. Silently dropping pairs after the first would leave
* access standing that the caller believes is gone.
*/
@Post('/revoke', {
subdomain: 'api',
requireVerified: true,
rateLimit: SHARE_LIMIT,
})
async revokeShare(req: Request, res: Response): Promise<void> {
const actor = this.#requireActor(req);
const body = this.#body(req);
const recipients = this.#recipients(body);
const items = this.#items(body, actor);
// #check = async (req: Request, res: Response): Promise<void> => {
// const token = req.body?.token;
// if (typeof token !== 'string' || token.length === 0) {
// throw new HttpError(400, 'Missing `token`');
// }
const pairs = recipients.flatMap((recipient) =>
items.map((item) => ({ recipient, item })),
);
const { unique, indexOf } = this.#dedupePairs(pairs);
// let decoded: { uid?: string; type?: string };
// try {
// decoded = this.services.token.verify(SHARE_TOKEN_TYPE, token);
// } catch {
// throw new HttpError(400, 'Invalid or expired share token');
// }
// if (decoded.type !== `token:${SHARE_TOKEN_TYPE}` || !decoded.uid) {
// throw new HttpError(400, 'Invalid share token');
// }
const settledUnique = await runWithConcurrencyLimitSettled(
unique,
SHARE_CONCURRENCY,
({ recipient, item }) =>
this.services.share.unshare(actor, { ...item, recipient }),
);
const settled = indexOf.map((i) => settledUnique[i]);
// const share = await this.stores.share.getByUid(decoded.uid);
// if (!share) throw new HttpError(404, 'Share not found or expired');
// Totalled over the unique outcomes: a pair listed twice revoked its
// grants once.
const revoked = settledUnique.reduce(
(total, outcome) =>
outcome.status === 'fulfilled'
? total + outcome.value.revoked
: total,
0,
);
const results: ShareOutcome[] = settled.map((outcome, index) => {
const { recipient, item } = pairs[index];
const label = recipient.email ?? recipient.username ?? '';
if (outcome.status === 'fulfilled') {
return {
recipient: label,
...(item.path ? { path: item.path } : {}),
...(item.uid ? { uid: item.uid } : {}),
status: 'success',
};
}
return {
recipient: label,
...(item.path ? { path: item.path } : {}),
...(item.uid ? { uid: item.uid } : {}),
status: 'error',
...this.#errorShape(outcome.reason),
};
});
// res.json({
// $: 'api:share',
// uid: share.uid,
// email: share.recipient_email,
// });
// };
const succeeded = results.filter((r) => r.status === 'success').length;
res.json({
status:
succeeded === results.length
? 'success'
: succeeded > 0
? 'mixed'
: 'aborted',
revoked,
results,
});
}
// // -- POST /sharelink/apply ---------------------------------------
// // Auth required — apply a pending share's permissions to the caller.
/**
* GET /share/shared-with-me — paginated listing of what others have shared
* with the caller.
*/
@Get('/shared-with-me', {
subdomain: 'api',
requireVerified: true,
rateLimit: SHARE_LIST_LIMIT,
})
async listSharedWithMe(req: Request, res: Response): Promise<void> {
const actor = this.#requireActor(req);
const query = this.#query(req);
// #apply = async (req: Request, res: Response): Promise<void> => {
// const uid = req.body?.uid;
// if (typeof uid !== 'string') throw new HttpError(400, 'Missing `uid`');
const page = await this.services.share.listSharedWithMe(actor, {
limit: normalizeLimit(query.limit, { cap: LIST_LIMIT_CAP }),
cursor: typeof query.cursor === 'string' ? query.cursor : undefined,
includeTotal: query.includeTotal === 'true',
});
// const actor = req.actor;
// if (!actor?.user) throw new HttpError(401, 'Unauthorized');
res.json({
items: await Promise.all(
page.items.map((share) => this.#toClientShare(share)),
),
...(page.cursor ? { cursor: page.cursor } : {}),
...(page.total !== undefined ? { total: page.total } : {}),
});
}
// const share = await this.stores.share.getByUid(uid);
// if (!share) throw new HttpError(404, 'Share not found or expired');
/** GET /share/shares — who can reach one item. */
@Get('/shares', {
subdomain: 'api',
requireVerified: true,
rateLimit: SHARE_LIST_LIMIT,
})
async listSharesOf(req: Request, res: Response): Promise<void> {
const actor = this.#requireActor(req);
const query = this.#query(req);
const target: ShareTarget = {};
if (typeof query.uid === 'string') target.uid = query.uid;
if (typeof query.path === 'string')
target.path = expandTildePath(query.path, actor.user?.username);
if (!target.uid && !target.path) {
throw new HttpError(400, 'one of `uid` or `path` is required', {
legacyCode: 'bad_request',
});
}
// // Issuer must still exist
// const issuer = await this.stores.user.getById(share.issuer_user_id);
// if (!issuer)
// throw new HttpError(410, 'Share expired — issuer account gone');
const shares = await this.services.share.listSharesOf(actor, target);
res.json({
items: await Promise.all(
shares.map((share) => this.#toClientShare(share)),
),
});
}
// // Email must be confirmed
// if (
// actor.user.requires_email_confirmation &&
// !actor.user.email_confirmed
// ) {
// throw new HttpError(
// 403,
// 'Please confirm your email before applying shares',
// );
// }
// -- Blocking -----------------------------------------------------
// // Recipient email must match
// if (
// !actor.user.email ||
// actor.user.email.toLowerCase() !==
// share.recipient_email.toLowerCase()
// ) {
// throw new HttpError(
// 403,
// 'This share was sent to a different email address',
// );
// }
/**
* GET /share/blocks — who the caller is refusing shares from, and whether
* they are refusing everyone.
*/
@Get('/blocks', {
subdomain: 'api',
requireVerified: true,
rateLimit: SHARE_LIST_LIMIT,
})
async listBlocks(req: Request, res: Response): Promise<void> {
const actor = this.#requireActor(req);
const { all, items } =
await this.services.share.listBlockedSenders(actor);
res.json({
all,
items: items.map((item) => ({
username: item.username,
created_at: item.createdAt,
})),
});
}
// // Grant each permission
// const issuerActor = {
// user: {
// id: issuer.id,
// uuid: issuer.uuid,
// username: issuer.username,
// email: issuer.email ?? null,
// suspended: false,
// email_confirmed: true,
// requires_email_confirmation: false,
// },
// } as import('../../core/actor.js').Actor;
// const data = (share.data ?? {}) as {
// permissions?: Array<{
// permission: string;
// extra?: Record<string, unknown>;
// }>;
// };
// for (const perm of data.permissions ?? []) {
// try {
// await this.services.permission.grantUserUserPermission(
// issuerActor,
// actor.user.username ?? '',
// perm.permission,
// perm.extra ?? {},
// );
// } catch (err) {
// console.warn('[share] grant failed for', perm.permission, err);
// }
// }
/**
* POST /share/blocks — stop accepting shares. `{ all: true }` refuses
* everyone; `{ username }` refuses one person. Idempotent either way:
* blocking twice is the state the caller asked for.
*
* Access already granted is untouched — `POST /share/revoke` is what
* withdraws that.
*/
@Post('/blocks', {
subdomain: 'api',
requireVerified: true,
rateLimit: SHARE_LIMIT,
})
async createBlock(req: Request, res: Response): Promise<void> {
const actor = this.#requireActor(req);
const body = this.#body(req);
if (this.#isBlockAll(body)) {
const { all } = await this.services.share.setBlockAllSenders(
actor,
true,
);
res.json({ all, blocked: true });
return;
}
const { username, created } = await this.services.share.blockSender(
actor,
this.#username(body),
);
res.json({ username, blocked: true, created });
}
// // Share consumed — delete it
// await this.stores.share.deleteByUid(uid);
/**
* DELETE /share/blocks — accept shares again. `{ all: true }` lifts the
* blanket refusal, leaving the per-sender list as it was.
*/
@Delete('/blocks', {
subdomain: 'api',
requireVerified: true,
rateLimit: SHARE_LIMIT,
})
async deleteBlock(req: Request, res: Response): Promise<void> {
const actor = this.#requireActor(req);
const body = this.#body(req);
if (this.#isBlockAll(body)) {
const { all } = await this.services.share.setBlockAllSenders(
actor,
false,
);
res.json({ all, blocked: false });
return;
}
const { username, unblocked } = await this.services.share.unblockSender(
actor,
this.#username(body),
);
res.json({ username, blocked: false, unblocked });
}
// res.json({ $: 'api:status-report', status: 'success' });
// };
// -- Helpers ------------------------------------------------------
// // -- POST /sharelink/request -------------------------------------
// // Auth required — notify the issuer that someone is requesting access.
/**
* Collapse repeated (recipient, item) pairs to one execution. `indexOf`
* maps every original position onto its unique pair, so a request that
* names the same pair twice still gets a result in both positions — the
* same result, which is the truth of what happened.
*/
#dedupePairs<T extends { recipient: ShareRecipient; item: ShareTarget }>(
pairs: T[],
): { unique: T[]; indexOf: number[] } {
const unique: T[] = [];
const seen = new Map<string, number>();
const indexOf = pairs.map((pair) => {
const key = JSON.stringify([
pair.recipient.email ?? null,
pair.recipient.username ?? null,
pair.item.uid ?? null,
pair.item.path ?? null,
]);
let at = seen.get(key);
if (at === undefined) {
at = unique.length;
seen.set(key, at);
unique.push(pair);
}
return at;
});
return { unique, indexOf };
}
// #request = async (req: Request, res: Response): Promise<void> => {
// const uid = req.body?.uid;
// if (typeof uid !== 'string') throw new HttpError(400, 'Missing `uid`');
/**
* Whether this call is about the blanket switch rather than one person.
* Only the literal `true` counts, so a client sending `all: false`
* alongside a username still means the username.
*/
#isBlockAll(body: Record<string, unknown>): boolean {
return body.all === true;
}
// const actor = req.actor;
// if (!actor?.user) throw new HttpError(401, 'Unauthorized');
#username(body: Record<string, unknown>): string {
const username =
typeof body.username === 'string' ? body.username.trim() : '';
if (!username) {
throw new HttpError(400, '`username` is required', {
legacyCode: 'bad_request',
});
}
return username;
}
// const share = await this.stores.share.getByUid(uid);
// if (!share) throw new HttpError(404, 'Share not found or expired');
/**
* Only ever the username — never the internal id, and never an email the
* caller didn't already supply.
*
* `thumbnail` is stored as an `s3://bucket/key` URI, so it is swapped for a
* signed URL rather than emitted: the raw value names internal storage and
* no client can render it.
*/
async #toClientShare(share: ResolvedShare) {
const thumbnail =
share.thumbnail === undefined
? undefined
: await signEntryThumbnail(
this.clients.event,
share.entryUid,
share.thumbnail,
);
return {
uid: share.uid,
mode: share.mode,
path: share.path,
// A share listing has no fsentry behind it for a client to stat.
...(share.name === undefined ? {} : { name: share.name }),
...(share.type === undefined ? {} : { type: share.type }),
...(thumbnail === undefined ? {} : { thumbnail }),
...(share.owner === undefined
? {}
: { owner: share.owner.username }),
...(share.pending
? { pending: true, recipient_email: share.recipientEmail }
: {}),
uid_entry: share.entryUid,
is_dir: share.isDir,
issuer: share.issuer.username,
holder: share.holder.username,
created_at: share.createdAt,
issued_by_app: share.issuedByApp ?? null,
inherited_from: share.inheritedFrom ?? null,
modified: share.modified,
size: share.size,
};
}
// const issuer = await this.stores.user.getById(share.issuer_user_id);
// if (!issuer)
// throw new HttpError(410, 'Share expired — issuer account gone');
#requireActor(req: Request): Actor {
const actor = req.actor;
if (!actor?.user)
throw new HttpError(401, 'Unauthorized', {
legacyCode: 'unauthorized',
});
return actor;
}
// // If caller IS the intended recipient (confirmed email matches),
// // they should just /apply instead.
// if (
// actor.user.email_confirmed &&
// actor.user.email?.toLowerCase() ===
// share.recipient_email.toLowerCase()
// ) {
// throw new HttpError(
// 400,
// 'You are the intended recipient — use /sharelink/apply instead',
// );
// }
#body(req: Request): Record<string, unknown> {
const body = req.body;
if (!body || typeof body !== 'object' || Array.isArray(body)) {
throw new HttpError(400, 'body must be an object', {
legacyCode: 'bad_request',
});
}
return body as Record<string, unknown>;
}
// // Notify the issuer
// if (this.services.notification) {
// await this.services.notification.notify([issuer.id], {
// source: 'sharing',
// title: `User ${actor.user.username} is trying to open a share you sent to ${share.recipient_email}`,
// template: 'user-requesting-share',
// fields: {
// username: actor.user.username,
// intended_recipient: share.recipient_email,
// permissions:
// (share.data as Record<string, unknown>)?.permissions ??
// [],
// },
// });
// }
#query(req: Request): Record<string, unknown> {
return (req.query ?? {}) as Record<string, unknown>;
}
// res.json({ $: 'api:status-report', status: 'success' });
// };
#recipients(body: Record<string, unknown>): ShareRecipient[] {
const raw = body.recipients ?? body.recipient;
const list = Array.isArray(raw) ? raw : [raw];
const out: ShareRecipient[] = [];
for (const entry of list) {
if (typeof entry === 'string') {
const value = entry.trim();
if (!value) continue;
out.push(
value.includes('@')
? { email: value }
: { username: value },
);
continue;
}
if (entry && typeof entry === 'object') {
const rec = entry as Record<string, unknown>;
const email =
typeof rec.email === 'string' ? rec.email.trim() : '';
const username =
typeof rec.username === 'string' ? rec.username.trim() : '';
if (email || username) {
out.push(email ? { email } : { username });
}
}
}
if (out.length === 0) {
throw new HttpError(400, '`recipients` is required', {
legacyCode: 'bad_request',
});
}
const max = this.config.share_max_recipients ?? DEFAULT_MAX_RECIPIENTS;
if (out.length > max) {
throw new HttpError(400, `at most ${max} recipients per request`, {
legacyCode: 'too_many_recipients',
});
}
return out;
}
// // -- POST /share -------------------------------------------------
// // Auth required — create shares for recipients (users or emails).
#items(body: Record<string, unknown>, actor: Actor): ShareTarget[] {
const username = actor.user?.username;
const raw = body.items ?? body.item ?? body.path ?? body.uid;
const list = Array.isArray(raw) ? raw : [raw];
const out: ShareTarget[] = [];
for (const entry of list) {
if (typeof entry === 'string') {
const value = entry.trim();
if (!value) continue;
// Tilde-rooted strings are paths, as the legacy FS routes treat them.
const isPath = value.startsWith('/') || value.startsWith('~');
out.push(
isPath
? { path: expandTildePath(value, username) }
: { uid: value },
);
continue;
}
if (entry && typeof entry === 'object') {
const item = entry as Record<string, unknown>;
const path =
typeof item.path === 'string'
? expandTildePath(item.path, username)
: '';
const uid = typeof item.uid === 'string' ? item.uid : '';
if (path || uid) out.push(path ? { path } : { uid });
}
}
if (out.length === 0) {
throw new HttpError(400, '`items` is required', {
legacyCode: 'bad_request',
});
}
const max = this.config.share_max_items ?? DEFAULT_MAX_ITEMS;
if (out.length > max) {
throw new HttpError(400, `at most ${max} items per request`, {
legacyCode: 'too_many_items',
});
}
return out;
}
// #share = async (req: Request, res: Response): Promise<void> => {
// const actor = req.actor;
// if (!actor?.user) throw new HttpError(401, 'Unauthorized');
// const body = req.body ?? {};
// let recipients = body.recipients;
// let shares = body.shares;
// const dryRun = !!body.dry_run;
// if (!recipients) throw new HttpError(400, 'Missing `recipients`');
// if (!shares) throw new HttpError(400, 'Missing `shares`');
// if (!Array.isArray(recipients)) recipients = [recipients];
// if (!Array.isArray(shares)) shares = [shares];
// // Build the permissions list from share declarations.
// const permissions = this.#resolvePermissions(shares as unknown[]);
// const recipientResults: unknown[] = [];
// for (const recipient of recipients as unknown[]) {
// const recipientStr =
// typeof recipient === 'string' ? recipient.trim() : '';
// if (!recipientStr) {
// recipientResults.push({
// $: 'error',
// message: 'empty recipient',
// });
// continue;
// }
// try {
// // Try username first
// const targetUser =
// (await this.stores.user.getByUsername(recipientStr)) ??
// (recipientStr.includes('@')
// ? await this.stores.user.getByEmail(recipientStr)
// : null);
// if (targetUser) {
// // Direct grant — user exists
// if (!dryRun) {
// for (const perm of permissions) {
// try {
// await this.services.permission.grantUserUserPermission(
// actor,
// targetUser.username ?? '',
// perm.permission,
// perm.extra ?? {},
// );
// } catch (err) {
// console.warn(
// '[share] grant to user failed',
// perm.permission,
// err,
// );
// }
// }
// // Notify
// if (this.services.notification) {
// await this.services.notification.notify(
// [targetUser.id],
// {
// source: 'sharing',
// title: `${actor.user.username} shared items with you`,
// template: 'file-shared-with-you',
// fields: {
// username: actor.user.username,
// permissions: permissions.map(
// (p) => p.permission,
// ),
// },
// },
// );
// }
// }
// recipientResults.push({
// $: 'api:status-report',
// status: 'success',
// });
// } else if (recipientStr.includes('@')) {
// // Email recipient — store pending share
// if (!dryRun) {
// const share = await this.stores.share.create({
// issuerUserId: actor.user.id,
// recipientEmail: recipientStr.toLowerCase(),
// data: {
// permissions,
// metadata: body.metadata ?? {},
// },
// });
// // Sign a share token (14-day expiry)
// const token = this.services.token.sign(
// SHARE_TOKEN_TYPE,
// {
// type: `token:${SHARE_TOKEN_TYPE}`,
// uid: share.uid,
// },
// { expiresIn: SHARE_TOKEN_EXPIRY },
// );
// // Email the share link
// const origin = `https://${this.config.domain ?? 'puter.com'}`;
// try {
// await this.clients.email.sendRaw({
// to: recipientStr,
// subject: `${actor.user.username} shared something with you on Puter`,
// html: `<p>${actor.user.username} shared items with you.</p><p><a href="${origin}?share_token=${encodeURIComponent(token)}">Click here to accept</a></p>`,
// });
// } catch (err) {
// console.warn('[share] email send failed', err);
// }
// }
// recipientResults.push({
// $: 'api:status-report',
// status: 'success',
// });
// } else {
// recipientResults.push({
// $: 'error',
// message: 'User not found',
// });
// }
// } catch (err) {
// recipientResults.push({ $: 'error', message: String(err) });
// }
// }
// const allOk = recipientResults.every(
// (r: unknown) => (r as Record<string, unknown>).status === 'success',
// );
// const anyOk = recipientResults.some(
// (r: unknown) => (r as Record<string, unknown>).status === 'success',
// );
// res.json({
// $: 'api:share',
// $version: 'v0.0.0',
// status: allOk ? 'success' : anyOk ? 'mixed' : 'aborted',
// recipients: recipientResults,
// ...(dryRun ? { dry_run: true } : {}),
// });
// };
// // -- Helpers ------------------------------------------------------
// /**
// * Convert share declarations into a flat permission list.
// * Supports `fs-share` ({ path, access }) and `app-share` ({ uid, name }).
// */
// #resolvePermissions(
// shares: unknown[],
// ): Array<{ permission: string; extra?: Record<string, unknown> }> {
// const perms: Array<{
// permission: string;
// extra?: Record<string, unknown>;
// }> = [];
// for (const share of shares) {
// if (!share || typeof share !== 'object') continue;
// const s = share as Record<string, unknown>;
// if (s.$ === 'fs-share' || s.type === 'fs-share' || s.path) {
// const path = String(s.path ?? '');
// const access = String(s.access ?? 'read');
// if (path) {
// perms.push({ permission: `fs:${path}:${access}` });
// }
// } else if (
// s.$ === 'app-share' ||
// s.type === 'app-share' ||
// s.uid ||
// s.name
// ) {
// const appUid = String(s.uid ?? s.name ?? '');
// if (appUid) {
// perms.push({ permission: `app:uid#${appUid}:access` });
// }
// }
// }
// return perms;
// }
/**
* Report a failure without widening what the caller already knew. The
* service already decides 404-vs-403; anything unrecognized becomes a
* generic error rather than leaking an internal message.
*/
#errorShape(reason: unknown): { message: string; code?: string } {
if (!isHttpError(reason) || reason.statusCode >= 500) {
return { message: 'Request failed' };
}
const code = reason.legacyCode ?? reason.code;
return {
message: reason.message || 'Request failed',
...(code ? { code } : {}),
};
}
}
@@ -280,15 +280,19 @@ export class StaticPagesController extends PuterController {
(user.clean_email as string | null | undefined) ??
String(user.email ?? '').toLowerCase();
const [dupe] = (await this.clients.db.read(
`SELECT EXISTS(
SELECT 1 FROM \`user\` WHERE (\`email\` = ? OR \`clean_email\` = ?)
AND \`email_confirmed\` = ${this.clients.db.booleanLiteral(true)}
AND \`password\` IS NOT NULL
) AS email_exists`,
[user.email, cleanEmail],
)) as Array<{ email_exists: number }>;
if (dupe?.email_exists) {
// An account that already confirmed this address proved access
// to the inbox. The strip below would take it away from them,
// so refuse here instead. Password-less accounts count: an
// identity provider verified the address for those, and they
// are exactly what the old `password IS NOT NULL` clause let
// through.
const confirmedRival =
await this.stores.user.findConfirmedOtherByEmail(
user.id as number,
user.email as string,
cleanEmail,
);
if (confirmedRival) {
res.send(
err('This email was confirmed on a different account.'),
);
@@ -302,6 +306,15 @@ export class StaticPagesController extends PuterController {
[user.email],
);
// Take the address off every remaining row before confirming
// this one. The check above leaves only unconfirmed rows, and
// only one row may own an address once this one is confirmed.
await this.stores.user.unconfirmOthersByEmail(
user.id,
user.email as string,
cleanEmail,
);
await this.stores.user.update(user.id, {
email_confirmed: 1,
requires_email_confirmation: 0,
@@ -1,19 +1,21 @@
// This suite tests basic features of puter webdav. it is not a comprehensive webdav test suite unlike litmus
// but rather it performs some common sense checks to ensure that WebDAV support isn't irrevocably broken in puter
import type { Request, Response } from 'express';
import type { Request, RequestHandler, Response } from 'express';
import { Readable, Writable } from 'node:stream';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import { hash as bcryptHash } from 'bcrypt';
import { PuterRouter } from '../../core/http/PuterRouter.js';
import type { RouteDescriptor } from '../../core/http/types.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { runWithContext } from '../../core/context.js';
import { generateDefaultFsentries } from '../../util/userProvisioning.js';
import type { WebDAVController } from './WebDAVController.js';
let server: PuterServer;
let controller: WebDAVController;
let dispatchMiddleware: Function;
let routes: RouteDescriptor[];
beforeAll(async () => {
server = await setupTestServer();
@@ -21,12 +23,60 @@ beforeAll(async () => {
const router = new PuterRouter();
controller.registerRoutes(router);
// WebDAVController registers a single `use()` middleware on the `dav`
// subdomain. Grab it to call directly in tests.
dispatchMiddleware = router.routes[0]!.handler;
routes = router.routes;
});
/** Run one handler; true when it called `next` instead of answering. */
const runHandler = async (
handler: RequestHandler,
req: Request,
res: Response,
): Promise<boolean> => {
let advanced = false;
await handler(req, res, (() => {
advanced = true;
}) as never);
return advanced;
};
/**
* Stand in for express's router: walk the controller's collected routes in
* order and run the first one that matches the request's method, plus any
* `middleware` it declares.
*
* The options the real server materializes into gates — subdomain, rate limit,
* concurrency — are deliberately not applied here; those belong to the
* materializer's own tests, and reproducing them would make every case below
* share one DAV budget. `server.test.ts` covers the wiring on a real port.
*/
const dispatchMiddleware = async (
req: Request,
res: Response,
next: () => void,
): Promise<void> => {
const method = req.method.toLowerCase();
for (const route of routes) {
if (
route.method !== 'use' &&
route.method !== 'all' &&
route.method !== method
) {
continue;
}
let advanced = true;
for (const handler of [
...(route.options.middleware ?? []),
route.handler,
]) {
advanced = await runHandler(handler, req, res);
if (!advanced) break;
}
// A chain that stopped short of `next` answered the request.
if (!advanced) return;
}
next();
};
afterAll(async () => {
await server?.shutdown();
});
@@ -151,10 +201,45 @@ const makeUser = async () => {
describe('WebDAVController', () => {
describe('route registration', () => {
it('registers a single catch-all use() route', () => {
it('registers one catch-all per DAV verb, all on the dav subdomain', () => {
const router = new PuterRouter();
controller.registerRoutes(router);
expect(router.routes.length).toBeGreaterThanOrEqual(1);
for (const verb of [
'options',
'head',
'get',
'propfind',
'proppatch',
'mkcol',
'put',
'delete',
'copy',
'move',
'lock',
'unlock',
]) {
const route = router.routes.find((r) => r.method === verb);
expect(route, verb).toBeDefined();
expect(route!.path).toBe('/{*splat}');
expect(route!.options.subdomain).toBe('dav');
// The gates the materializer installs ahead of the handler.
expect(route!.options.rateLimit).toBeDefined();
expect(route!.options.concurrent).toBeDefined();
}
// `all` catches the verbs we don't implement, so it has to come
// after every route that does.
const allIndex = router.routes.findIndex((r) => r.method === 'all');
expect(allIndex).toBe(router.routes.length - 1);
});
it('registers HEAD ahead of GET so express does not fold the two', () => {
const router = new PuterRouter();
controller.registerRoutes(router);
const head = router.routes.findIndex((r) => r.method === 'head');
const get = router.routes.findIndex((r) => r.method === 'get');
expect(head).toBeLessThan(get);
});
});
@@ -240,6 +325,48 @@ describe('WebDAVController', () => {
});
});
describe('CORS preflight', () => {
it('answers a browser preflight before the auth gate', async () => {
const { res, captured } = makeRes();
await dispatchMiddleware(
makeReq({
method: 'OPTIONS',
path: '/someone/.vscode/settings.json',
headers: {
origin: 'https://code.puter.com',
'access-control-request-method': 'PROPFIND',
},
}),
res,
noop,
);
// A non-2xx here is what made every browser DAV client give up
// before it could send a token.
expect(captured.statusCode).toBe(200);
expect(captured.headers['dav']).toContain('1');
expect(captured.headers['allow']).toContain('PROPFIND');
expect(captured.headers['access-control-max-age']).toBe('86400');
expect(captured.headers['www-authenticate']).toBeUndefined();
});
it('still authenticates an OPTIONS that is not a preflight', async () => {
const { res, captured } = makeRes();
await dispatchMiddleware(
makeReq({
method: 'OPTIONS',
// An Origin alone isn't a preflight — only the
// `Access-Control-Request-Method` probe is.
headers: { origin: 'https://code.puter.com' },
}),
res,
noop,
);
expect(captured.statusCode).toBe(401);
// macOS opens a mount on this reply, so it keeps the DAV header.
expect(captured.headers['dav']).toContain('1');
});
});
describe('pending-verification gate', () => {
// WebDAV must enforce the same gate every other authenticated route
// gets from requireVerifiedAccount — it dispatches off a single use()
@@ -1122,6 +1249,54 @@ describe('WebDAVController verbs', () => {
);
});
it('reports timestamps in the present, not the epoch', async () => {
// Entry timestamps are stored as Unix seconds. Handing those to
// `Date` unscaled reads them as milliseconds and dates every file
// to January 1970, which clients sort and sync against.
const { actor, username } = await makeUser();
const path = `/${username}/Documents/dated.txt`;
await putFile(actor, path, 'x');
const captured = await dispatch({
method: 'PROPFIND',
path,
actor,
headers: { depth: '0' },
});
const xml = captured.body as string;
const modified = xml.match(
/<D:getlastmodified>(.*?)<\/D:getlastmodified>/,
)?.[1];
expect(modified).toBeDefined();
const skew = Math.abs(
new Date(modified as string).getTime() - Date.now(),
);
expect(skew).toBeLessThan(5 * 60 * 1000);
});
it('emits the same ETag for an entry over PROPFIND and GET', async () => {
// A client that sees the validator change between the two treats
// the file as modified and re-downloads it on every pass.
const { actor, username } = await makeUser();
const path = `/${username}/Documents/etag-match.txt`;
await putFile(actor, path, 'x');
const propfind = await dispatch({
method: 'PROPFIND',
path,
actor,
headers: { depth: '0' },
});
const propfindEtag = (propfind.body as string).match(
/<D:getetag>(.*?)<\/D:getetag>/,
)?.[1];
const get = await dispatch({ method: 'GET', path, actor });
expect(propfindEtag).toBeDefined();
expect(get.headers['etag']).toBe(propfindEtag);
});
it('omits children at depth 0', async () => {
const { actor, username } = await makeUser();
await putFile(actor, `/${username}/Documents/hidden.txt`, 'x');
@@ -1166,6 +1341,68 @@ describe('WebDAVController verbs', () => {
'<D:getcontenttype>text/markdown</D:getcontenttype>',
);
});
it("answers a share root's parent with a virtual collection", async () => {
const owner = await makeUser();
const holder = await makeUser();
await dispatch({
method: 'MKCOL',
path: `/${owner.username}/Documents/Album`,
actor: owner.actor,
});
const root = (await server.stores.fsEntry.getEntryByPath(
`/${owner.username}/Documents/Album`,
))!;
await server.services.acl.setUserUser(
owner.actor as never,
holder.actor as never,
{
path: root.path,
resolveAncestors: () =>
server.services.fs.getAncestorChain(root.path),
},
'read',
);
// `/<owner>/<uuid>` is where Up from a share root lands; it is not
// a real path, so it answers as a virtual collection whose only
// member is the share root at its masked path.
const captured = await dispatch({
method: 'PROPFIND',
path: `/${owner.username}/${root.uuid}`,
actor: holder.actor,
});
expect(captured.statusCode).toBe(207);
const xml = captured.body as string;
expect(xml).toContain(
`<D:href>/${owner.username}/${root.uuid}/</D:href>`,
);
expect(xml).toContain(
`<D:href>/${owner.username}/${root.uuid}/Album/</D:href>`,
);
// The owner's real tree never shows through.
expect(xml).not.toContain('/Documents/');
});
it("keeps a share root's parent a 404 for a caller with no share", async () => {
const owner = await makeUser();
const stranger = await makeUser();
await dispatch({
method: 'MKCOL',
path: `/${owner.username}/Documents/Private`,
actor: owner.actor,
});
const root = (await server.stores.fsEntry.getEntryByPath(
`/${owner.username}/Documents/Private`,
))!;
const captured = await dispatch({
method: 'PROPFIND',
path: `/${owner.username}/${root.uuid}`,
actor: stranger.actor,
});
expect(captured.statusCode).toBe(404);
});
});
describe('PROPPATCH', () => {
@@ -1653,4 +1890,368 @@ describe('WebDAVController verbs', () => {
expect(captured.body).toBe('Lock token does not match this path');
});
});
describe('shared-path masking', () => {
/** A `Secrets/plan.txt` folder in `owner`'s Documents, shared with `to`. */
const shareFolder = async (
owner: Awaited<ReturnType<typeof makeUser>>,
to: Awaited<ReturnType<typeof makeUser>>,
) => {
const dirUuid = uuidv4();
const dirPath = `/${owner.username}/Documents/Secrets`;
const now = Math.floor(Date.now() / 1000);
const parent = (await server.stores.fsEntry.getEntryByPath(
`/${owner.username}/Documents`,
))!;
await server.clients.db.write(
'INSERT INTO `fsentries` (`uuid`, `name`, `path`, `user_id`, `is_dir`, `modified`, `parent_id`, `parent_uid`) VALUES (?, ?, ?, ?, 1, ?, ?, ?)',
[
dirUuid,
'Secrets',
dirPath,
owner.userId,
now,
parent.id,
parent.uuid,
],
);
const dir = (await server.stores.fsEntry.getEntryByPath(dirPath))!;
await server.clients.db.write(
'INSERT INTO `fsentries` (`uuid`, `name`, `path`, `user_id`, `is_dir`, `modified`, `parent_id`, `parent_uid`) VALUES (?, ?, ?, ?, 0, ?, ?, ?)',
[
uuidv4(),
'plan.txt',
`${dirPath}/plan.txt`,
owner.userId,
now,
dir.id,
dir.uuid,
],
);
await runWithContext({ actor: owner.actor as never }, () =>
server.services.share.share(owner.actor as never, {
uid: dirUuid,
recipient: { username: to.username! },
mode: 'read',
}),
);
return { dirUuid, dirPath };
};
/**
* PROPFIND under the ALS wrap a real request gets, pre-auth actor
* empty.
*/
const propfind = async (path: string, actor: unknown, depth = '1') => {
const { res, captured } = makeRes();
await runWithContext({ actor: undefined }, () =>
dispatchMiddleware(
makeReq({
method: 'PROPFIND',
path,
headers: { depth },
actor,
}),
res,
noop,
),
);
return captured;
};
// The DAV controller authenticates in-controller, after the request
// context snapshotted an empty `req.actor` — so it has to place the
// actor into the context itself or outbound masking silently turns
// off and PROPFIND lists the owner's real paths.
it('lists a shared folder under its masked path, not the owner’s real one', async () => {
const owner = await makeUser();
const recipient = await makeUser();
const { dirUuid } = await shareFolder(owner, recipient);
const masked = `/${owner.username}/${dirUuid}/Secrets`;
const captured = await propfind(masked, recipient.actor);
expect(captured.statusCode).toBe(207);
const xml = String(captured.body);
expect(xml).toContain(`${masked}/plan.txt`);
expect(xml).not.toContain('/Documents/Secrets');
});
it('lists each sharer as a collection under the root', async () => {
const owner = await makeUser();
const recipient = await makeUser();
await shareFolder(owner, recipient);
const captured = await propfind('/', recipient.actor);
expect(captured.statusCode).toBe(207);
const xml = String(captured.body);
// The recipient's own home, plus the owner who shared with them.
expect(xml).toContain(`<D:href>/${recipient.username}/</D:href>`);
expect(xml).toContain(`<D:href>/${owner.username}/</D:href>`);
});
it('omits the sharers at depth 0', async () => {
const owner = await makeUser();
const recipient = await makeUser();
await shareFolder(owner, recipient);
const captured = await propfind('/', recipient.actor, '0');
expect(captured.statusCode).toBe(207);
expect(String(captured.body)).not.toContain(`/${owner.username}/`);
});
it('answers a sharer’s collection with the shares they issued', async () => {
const owner = await makeUser();
const recipient = await makeUser();
const { dirUuid } = await shareFolder(owner, recipient);
const captured = await propfind(
`/${owner.username}`,
recipient.actor,
);
expect(captured.statusCode).toBe(207);
const xml = String(captured.body);
expect(xml).toContain(`<D:href>/${owner.username}/</D:href>`);
// The `<uuid>` level, which the share root hangs off. It borrows
// the shared item's name so a client has something to show.
expect(xml).toContain(
`<D:href>/${owner.username}/${dirUuid}/</D:href>`,
);
expect(xml).toContain('<D:displayname>Secrets</D:displayname>');
// Nothing else of the owner's shows through — not their home
// directory's other children, not the folder the share sits in.
expect(xml).not.toContain('Documents');
});
it('keeps a sharer’s collection a 403 for a caller with no share', async () => {
const owner = await makeUser();
const stranger = await makeUser();
await shareFolder(owner, await makeUser());
const captured = await propfind(
`/${owner.username}`,
stranger.actor,
);
expect(captured.statusCode).toBe(403);
});
});
// The virtual collections invent two levels of path that no fsentry backs.
// Everything here checks that they only ever widen to what was actually
// shared — the rest of the owner's tree has to stay unreachable and
// unnamed.
describe('shared-space containment', () => {
/**
* `owner` shares `Shared/` (holding `in.txt`) with `to`, and keeps
* `Private/` (holding `out.txt`) to themselves.
*/
const shareOneOfTwo = async (
owner: Awaited<ReturnType<typeof makeUser>>,
to: Awaited<ReturnType<typeof makeUser>>,
) => {
const mk = async (
method: string,
path: string,
extra: Record<string, string> = {},
) => {
const { res, captured } = makeRes();
await runWithContext({ actor: undefined }, () =>
dispatchMiddleware(
makeReq({
method,
path,
actor: owner.actor,
headers: extra,
}),
res,
noop,
),
);
return captured;
};
await mk('MKCOL', `/${owner.username}/Documents/Shared`);
await mk('MKCOL', `/${owner.username}/Documents/Private`);
const shared = (await server.stores.fsEntry.getEntryByPath(
`/${owner.username}/Documents/Shared`,
))!;
const private_ = (await server.stores.fsEntry.getEntryByPath(
`/${owner.username}/Documents/Private`,
))!;
await runWithContext({ actor: owner.actor as never }, () =>
server.services.share.share(owner.actor as never, {
uid: shared.uuid,
recipient: { username: to.username! },
mode: 'read',
}),
);
return { shared, private: private_ };
};
const propfind = async (path: string, actor: unknown, depth = '1') => {
const { res, captured } = makeRes();
await runWithContext({ actor: undefined }, () =>
dispatchMiddleware(
makeReq({
method: 'PROPFIND',
path,
headers: { depth },
actor,
}),
res,
noop,
),
);
return captured;
};
it('lists only the items that were shared, not the owner’s siblings', async () => {
const owner = await makeUser();
const holder = await makeUser();
const { shared, private: hidden } = await shareOneOfTwo(
owner,
holder,
);
const captured = await propfind(`/${owner.username}`, holder.actor);
expect(captured.statusCode).toBe(207);
const xml = String(captured.body);
expect(xml).toContain(shared.uuid);
expect(xml).not.toContain(hidden.uuid);
expect(xml).not.toContain('Private');
});
it('does not recurse past the uuid level at Depth: infinity', async () => {
const owner = await makeUser();
const holder = await makeUser();
const { shared } = await shareOneOfTwo(owner, holder);
const captured = await propfind(
`/${owner.username}`,
holder.actor,
'infinity',
);
expect(captured.statusCode).toBe(207);
const xml = String(captured.body);
// The share's own href lives one segment deeper; asking for the
// world must not walk into it.
expect(xml).toContain(`/${shared.uuid}/`);
expect(xml).not.toContain(`/${shared.uuid}/Shared`);
});
it('refuses every way out of a share', async () => {
const owner = await makeUser();
const holder = await makeUser();
const { shared, private: hidden } = await shareOneOfTwo(
owner,
holder,
);
const root = `/${owner.username}/${shared.uuid}/Shared`;
for (const escape of [
// Up and sideways out of the shared subtree.
`${root}/../Private`,
`${root}/../../Documents`,
// Percent-encoded, because the path is decoded before it is
// resolved — the guard has to sit after that, not before.
`${root}/%2e%2e/Private`,
// The owner's real path, named directly rather than walked to.
`/${owner.username}/Documents/Private`,
`/${owner.username}/Documents`,
// A mask pointing at an entry the holder was never given —
// both a bad tail under a uuid they do hold, and the uuid of
// one they don't.
`/${owner.username}/${shared.uuid}/Private`,
`/${owner.username}/${hidden.uuid}`,
`/${owner.username}/${hidden.uuid}/Private`,
]) {
const captured = await propfind(escape, holder.actor);
expect(captured.statusCode, escape).toBeGreaterThanOrEqual(400);
expect(captured.statusCode, escape).toBeLessThan(500);
expect(String(captured.body), escape).not.toContain('Private');
}
});
it('does not leak one holder’s shares to another', async () => {
const owner = await makeUser();
const holder = await makeUser();
const outsider = await makeUser();
await shareOneOfTwo(owner, holder);
const captured = await propfind('/', outsider.actor);
expect(captured.statusCode).toBe(207);
expect(String(captured.body)).not.toContain(owner.username);
});
it('drops the sharer once the share is revoked', async () => {
const owner = await makeUser();
const holder = await makeUser();
const { shared } = await shareOneOfTwo(owner, holder);
expect(String((await propfind('/', holder.actor)).body)).toContain(
`/${owner.username}/`,
);
await runWithContext({ actor: owner.actor as never }, () =>
server.services.share.unshare(owner.actor as never, {
uid: shared.uuid,
recipient: { username: holder.username! },
}),
);
const root = await propfind('/', holder.actor);
expect(String(root.body)).not.toContain(`/${owner.username}/`);
// And the level it used to stand in for is a plain 403 again.
const owned = await propfind(`/${owner.username}`, holder.actor);
expect(owned.statusCode).toBe(403);
});
it('refuses writes and reads aimed at the invented levels', async () => {
const owner = await makeUser();
const holder = await makeUser();
const { shared } = await shareOneOfTwo(owner, holder);
const attempt = async (method: string, path: string) => {
const { res, captured } = makeRes();
await runWithContext({ actor: undefined }, () =>
dispatchMiddleware(
makeReq({
method,
path,
actor: holder.actor,
headers: { 'content-length': '3' },
}),
res,
noop,
),
);
return captured.statusCode;
};
// A collection, so GET is a 400 either way — what matters is that
// neither the owner level nor the uuid level accepts a write.
expect(await attempt('GET', `/${owner.username}`)).not.toBe(200);
expect(
await attempt('MKCOL', `/${owner.username}/intruder`),
).toBeGreaterThanOrEqual(400);
expect(
await attempt(
'MKCOL',
`/${owner.username}/${shared.uuid}/intruder`,
),
).toBeGreaterThanOrEqual(400);
expect(
await attempt('DELETE', `/${owner.username}/${shared.uuid}`),
).toBeGreaterThanOrEqual(400);
// Nothing landed in the owner's tree.
expect(
await server.stores.fsEntry.getEntryByPath(
`/${owner.username}/intruder`,
),
).toBeFalsy();
});
});
});
+534 -231
View File
@@ -27,9 +27,31 @@ import {
assertNotSuspended,
assertVerifiedAccount,
} from '../../core/http/middleware/gates.js';
import type { PuterRouter } from '../../core/http/PuterRouter.js';
import {
All,
Controller,
Copy,
Delete,
Get,
Head,
Lock,
Mkcol,
Move,
Options,
Propfind,
Proppatch,
Put,
Unlock,
} from '../../core/http/decorators.js';
import type { RouteOptions } from '../../core/http/types.js';
import { verify as verifyOtp } from '../../services/auth/OTPUtil.js';
import { expandTildePath } from '../../services/fs/resolveNode.js';
import {
maskEntryPath,
parseMaskedSharePath,
resolveSharePath,
} from '../../services/fs/sharePathMask.js';
import { Context } from '../../core/context.js';
import type { FSEntry } from '../../stores/fs/FSEntry.js';
import { toLegacyEntry } from '../fs/legacyFsHelpers.js';
import { PuterController } from '../types.js';
@@ -43,12 +65,8 @@ import {
refreshLock,
} from './locks.js';
import { DAV_CONCURRENT, DAV_LIMIT } from '../fs/limits.js';
import {
acquireConcurrent,
checkRateLimit,
computeNetworkFingerprint,
} from '../../core/http/middleware/rateLimit.js';
import { assertActorHasCredits } from '../../services/metering/enforcement.js';
import type { ResolvedShare } from '../../services/share/ShareService.js';
const DAV_HEADERS = {
DAV: '1, 2, ordered-collections',
@@ -62,11 +80,41 @@ const ALLOW_METHODS =
const MACOS_JUNK_REGEX = /(?:^\.DS_Store$|^\._)/;
/**
* Verbs that move file content or duplicate it in the object store, and so are
* refused to an account with nothing left of its budget. HEAD is here with GET
* because a client asking for headers is a client about to fetch the body.
* Every DAV verb mounts on the same catch-all: on a DAV host the path _is_ the
* resource, so there is nothing else to route on. `{*splat}` rather than plain
* `*splat` because only the braced form also matches `/`, which is the first
* collection a client PROPFINDs.
*/
const CREDIT_GATED_DAV_METHODS = new Set(['GET', 'HEAD', 'PUT', 'COPY']);
const DAV_ROUTE = '/{*splat}';
/**
* Everything the route table declares. The subdomain keeps DAV off every other
* host, and both limits key on the network fingerprint because nothing has
* authenticated yet when they run — which is the point of putting them here
* rather than inside the handler. A DAV client sends credentials on every
* request anyway, so there is no unauthenticated browsing phase that a per-user
* key would protect.
*/
const DAV_ROUTE_OPTIONS: RouteOptions = {
subdomain: 'dav',
rateLimit: DAV_LIMIT,
concurrent: DAV_CONCURRENT,
};
/** How long a browser may cache a DAV preflight. */
const PREFLIGHT_MAX_AGE = '86400';
/** How many shared items the virtual share collections list. */
const SHARE_LISTING_CAP = 200;
/** What `#context` resolved for one request, handed to every verb handler. */
interface DavContext {
actor: Actor;
/** The real path being addressed, with `~` and any share mask resolved. */
davPath: string;
/** Lock token from `If:` / `Lock-Token:`, when the client sent one. */
lockToken: string | null;
}
/**
* WebDAV controller — full RFC 4918 surface on the `dav.*` subdomain.
@@ -79,93 +127,162 @@ const CREDIT_GATED_DAV_METHODS = new Set(['GET', 'HEAD', 'PUT', 'COPY']);
* `-token` username for token-based auth). Falls back to the global authProbe's
* `req.actor` if a session cookie is present.
*/
@Controller()
export class WebDAVController extends PuterController {
registerRoutes(router: PuterRouter): void {
// Single catch-all on the `dav` subdomain. We dispatch by req.method
// inside the handler because WebDAV uses non-standard HTTP verbs that
// Express doesn't have first-class router methods for in all versions.
//
// The rate limit is applied inside the handler rather than through
// `RouteOptions`. For a `use` mount the subdomain check lives in the
// handler wrapper, not in the middleware chain — so a `rateLimit`
// here would run for every request on every subdomain and count
// non-DAV traffic against the DAV budget.
router.use(
{ subdomain: 'dav' },
async (req: Request, res: Response, _next) => {
try {
if (!(await this.#admit(req, res))) return;
await this.#dispatch(req, res);
} catch (err) {
if (err instanceof HttpError) {
res.status(err.statusCode).send(err.message);
return;
}
console.error('[webdav] unhandled error', err);
res.status(500).send('Internal Server Error');
}
// Don't call next — we always handle or error.
},
);
// -- Routes ------------------------------------------------------
//
// One route per verb, every one of them the same catch-all. Declaration
// order is registration order: HEAD before GET, because express falls a
// HEAD request back onto a GET route when it meets one first, and `@All`
// last, because it matches any method.
//
// `DAV_ROUTE_OPTIONS` is the whole declarative half of the chain. The
// credential work happens in `#serve` instead, which is what keeps the
// limits ahead of it: HTTP Basic means a bcrypt compare per request, and a
// caller already over budget shouldn't get one.
@Options(DAV_ROUTE, DAV_ROUTE_OPTIONS)
options(req: Request, res: Response): Promise<void> {
// Before the auth gate, and only for a real preflight — see
// `answerPreflight`.
if (answerPreflight(req, res)) return Promise.resolve();
return this.#serve(req, res, () => this.#options(res));
}
// GET/HEAD/PUT/COPY move file content or duplicate it in the object store,
// so they carry the same budget gate the FS routes declare with
// `requireCredits`: DAV serves the same files over a metered host, and
// leaving it out would make mounting the drive the way around enforcement.
// The verbs that only describe or remove things stay open, as they do over
// HTTP. HEAD is metered with GET because a client asking for headers is a
// client about to fetch the body.
//
// `requireCredits` itself can't be used for this: it reads `req.actor`, and
// on the DAV host nothing has authenticated by the time route options run.
@Head(DAV_ROUTE, DAV_ROUTE_OPTIONS)
head(req: Request, res: Response): Promise<void> {
return this.#serve(req, res, (ctx) => this.#get(req, res, ctx, true), {
credits: true,
});
}
@Get(DAV_ROUTE, DAV_ROUTE_OPTIONS)
get(req: Request, res: Response): Promise<void> {
return this.#serve(req, res, (ctx) => this.#get(req, res, ctx, false), {
credits: true,
});
}
@Propfind(DAV_ROUTE, DAV_ROUTE_OPTIONS)
propfind(req: Request, res: Response): Promise<void> {
return this.#serve(req, res, (ctx) => this.#propfind(req, res, ctx));
}
@Proppatch(DAV_ROUTE, DAV_ROUTE_OPTIONS)
proppatch(req: Request, res: Response): Promise<void> {
return this.#serve(req, res, (ctx) => this.#proppatch(res, ctx));
}
@Mkcol(DAV_ROUTE, DAV_ROUTE_OPTIONS)
mkcol(req: Request, res: Response): Promise<void> {
return this.#serve(req, res, (ctx) => this.#mkcol(req, res, ctx));
}
@Put(DAV_ROUTE, DAV_ROUTE_OPTIONS)
put(req: Request, res: Response): Promise<void> {
return this.#serve(req, res, (ctx) => this.#put(req, res, ctx), {
credits: true,
});
}
@Delete(DAV_ROUTE, DAV_ROUTE_OPTIONS)
delete(req: Request, res: Response): Promise<void> {
return this.#serve(req, res, (ctx) => this.#delete(res, ctx));
}
@Copy(DAV_ROUTE, DAV_ROUTE_OPTIONS)
copy(req: Request, res: Response): Promise<void> {
return this.#serve(req, res, (ctx) => this.#copy(req, res, ctx), {
credits: true,
});
}
@Move(DAV_ROUTE, DAV_ROUTE_OPTIONS)
move(req: Request, res: Response): Promise<void> {
return this.#serve(req, res, (ctx) => this.#move(req, res, ctx));
}
@Lock(DAV_ROUTE, DAV_ROUTE_OPTIONS)
lock(req: Request, res: Response): Promise<void> {
return this.#serve(req, res, (ctx) => this.#lock(req, res, ctx));
}
@Unlock(DAV_ROUTE, DAV_ROUTE_OPTIONS)
unlock(req: Request, res: Response): Promise<void> {
return this.#serve(req, res, (ctx) => this.#unlock(req, res, ctx));
}
/** POST, TRACE, an extension verb we don't implement. */
@All(DAV_ROUTE, DAV_ROUTE_OPTIONS)
unsupported(req: Request, res: Response): Promise<void> {
return this.#serve(req, res, () => {
res.status(405)
.set('Allow', ALLOW_METHODS)
.send('Method Not Allowed');
});
}
// -- Request plumbing ---------------------------------------------
/**
* Rate + concurrency gate for the whole DAV surface. Returns false when the
* request was rejected (429 already sent).
*
* Runs before `#dispatch` authenticates, so it keys on the network
* fingerprint rather than an actor. That is the coarser bucket, but a DAV
* client sends credentials on every request anyway — there is no
* unauthenticated browsing phase to protect a per-user key from.
* Authenticate, gate, resolve the target path, and run `handler` — or
* answer the request instead when any of that fails. A thrown `HttpError`
* becomes the plain-text reply a DAV client reads; nothing calls `next`,
* because a request that gets here is ours to answer.
*/
async #admit(req: Request, res: Response): Promise<boolean> {
const key = computeNetworkFingerprint(req);
if (
!(await checkRateLimit(
`${DAV_LIMIT.scope}:${key}`,
DAV_LIMIT.limit,
DAV_LIMIT.window,
))
) {
res.status(429).send('Too many requests.');
return false;
async #serve(
req: Request,
res: Response,
handler: (ctx: DavContext) => void | Promise<void>,
opts: { credits?: boolean } = {},
): Promise<void> {
try {
const ctx = await this.#context(req, res, opts);
if (!ctx) return; // 401 already sent
await handler(ctx);
} catch (err) {
sendDavError(res, err);
}
const slot = await acquireConcurrent(
`${DAV_CONCURRENT.scope}:${key}`,
DAV_CONCURRENT.limit,
);
if (!slot.ok) {
res.status(429).send('Too many concurrent requests.');
return false;
}
// `finish` and `close` can both fire; release is once-only.
res.once('finish', () => void slot.release());
res.once('close', () => void slot.release());
return true;
}
async #dispatch(req: Request, res: Response): Promise<void> {
// Authenticate
/** The actor, path and lock token for one request; null once 401'd. */
async #context(
req: Request,
res: Response,
opts: { credits?: boolean },
): Promise<DavContext | null> {
const actor = await this.#resolveActor(req, res);
if (!actor) return; // 401 already sent
if (!actor) return null;
// Apply the same suspension + pending-verification gates every other
// authenticated route gets from `requireAuthGate` / `requireVerifiedAccount`.
// WebDAV dispatches every method off a single `router.use` with no route
// options, so that middleware is never inserted into its chain — without
// these calls a suspended account (or one still pending email / phone /
// card verification) could read, write, and delete its entire filesystem
// over the `dav` subdomain, bypassing the gates. Both throw a 403
// HttpError, surfaced by the catch in registerRoutes.
// The same suspension + pending-verification gates every other
// authenticated route gets from `requireAuthGate` /
// `requireVerifiedAccount`. DAV authenticates itself rather than
// through the auth probe, so that middleware is never in its chain —
// without these calls a suspended account (or one still pending email /
// phone / card verification) could read, write, and delete its entire
// filesystem over the `dav` subdomain, bypassing the gates.
assertNotSuspended(actor.user);
assertVerifiedAccount(actor.user);
// And the same budget gate the FS routes declare with
// `requireCredits`, for the verbs that move content — DAV serves the
// same files over a metered host, so leaving it out would make mounting
// the drive the way around enforcement. The verbs that only describe or
// remove things stay open, as they do over HTTP.
if (CREDIT_GATED_DAV_METHODS.has(req.method.toUpperCase())) {
// The actor was absent when the auth probe ran, so anything that reads
// it off the request or the context — shared-path masking, egress
// metering, which bills the `dav` host — is blind until it's published
// here.
req.actor = actor;
if (Context.current()) Context.set('actor', actor);
if (opts.credits) {
await assertActorHasCredits(
this.services.metering,
actor,
@@ -176,51 +293,20 @@ export class WebDAVController extends PuterController {
// Expand `~`/`~/...` against the authenticated actor's username.
// WebDAV doesn't standardize `~`, but some clients do — and the
// pre-existing behaviour silently expanded it via the FS store.
const davPath = expandTildePath(
decodeURIComponent(req.path),
actor.user.username,
);
const redis = this.clients.redis;
const lockToken = extractLockToken(
(req.headers['if'] as string | undefined) ??
(req.headers['lock-token'] as string | undefined),
const davPath = await resolveSharePath(
this.stores.fsEntry,
actor,
expandTildePath(decodeURIComponent(req.path), actor.user.username),
);
switch (req.method.toUpperCase()) {
case 'OPTIONS':
return this.#options(res);
case 'HEAD':
case 'GET':
return this.#get(
req,
res,
actor,
davPath,
req.method === 'HEAD',
);
case 'PROPFIND':
return this.#propfind(req, res, actor, davPath);
case 'PROPPATCH':
return this.#proppatch(res, davPath, redis, lockToken);
case 'MKCOL':
return this.#mkcol(req, res, actor, davPath, redis, lockToken);
case 'PUT':
return this.#put(req, res, actor, davPath, redis, lockToken);
case 'DELETE':
return this.#delete(res, actor, davPath, redis, lockToken);
case 'COPY':
return this.#copy(req, res, actor, davPath, redis, lockToken);
case 'MOVE':
return this.#move(req, res, actor, davPath, redis, lockToken);
case 'LOCK':
return this.#lock(req, res, actor, davPath, redis, lockToken);
case 'UNLOCK':
return this.#unlock(req, res, davPath, redis);
default:
res.status(405)
.set('Allow', ALLOW_METHODS)
.send('Method Not Allowed');
}
return {
actor,
davPath,
lockToken: extractLockToken(
(req.headers['if'] as string | undefined) ??
(req.headers['lock-token'] as string | undefined),
),
};
}
// -- Auth ---------------------------------------------------------
@@ -349,8 +435,7 @@ export class WebDAVController extends PuterController {
async #get(
req: Request,
res: Response,
actor: Actor,
davPath: string,
{ actor, davPath }: DavContext,
headOnly: boolean,
): Promise<void> {
const entry = await this.stores.fsEntry.getEntryByPath(davPath);
@@ -363,15 +448,14 @@ export class WebDAVController extends PuterController {
await this.#assertRead(actor, davPath);
const etag = `"${entry.uuid}-${Math.floor(entry.modified ?? entry.created ?? 0)}"`;
const modified = entry.modified ?? entry.created ?? 0;
const etag = entryEtag(entry.uuid, modified);
const size = entry.size ?? 0;
res.set({
'Accept-Ranges': 'bytes',
'Content-Length': String(size),
'Last-Modified': new Date(
entry.modified ?? entry.created ?? 0,
).toUTCString(),
'Last-Modified': entryDate(modified).toUTCString(),
ETag: etag,
});
@@ -399,43 +483,172 @@ export class WebDAVController extends PuterController {
async #propfind(
req: Request,
res: Response,
actor: Actor,
davPath: string,
{ actor, davPath }: DavContext,
): Promise<void> {
const depth = req.headers.depth ?? '1';
const entry =
davPath === '/'
? null // root always exists
: await this.stores.fsEntry.getEntryByPath(davPath);
if (davPath !== '/' && !entry)
if (davPath === '/') {
await this.#assertRead(actor, davPath);
this.#sendMultistatus(res, await this.#rootPropfind(actor, depth));
return;
}
const entry = await this.stores.fsEntry.getEntryByPath(davPath);
// The two levels a share mask invents — `/<owner>` and
// `/<owner>/<uuid>` — have no entry behind them, so they answer as
// virtual collections. Tried before the ACL check because `/<owner>`
// _is_ a real path (the owner's home directory) that the recipient
// can't read: a 403 there would hide every share they do have.
const virtual =
(await this.#shareOwnerPropfind(actor, davPath, entry, depth)) ??
(entry
? null
: await this.#shareRootParentPropfind(actor, davPath, depth));
if (virtual) {
this.#sendMultistatus(res, virtual);
return;
}
if (!entry) {
throw new HttpError(404, 'Not Found', { legacyCode: 'not_found' });
}
await this.#assertRead(actor, davPath);
const isDir = davPath === '/' || !!entry?.isDir;
const responses = [propfindEntry(davPath, entry, isDir)];
// `davPath` is the resolved real path; the href has to be the masked
// one, like every child below it, or the self-entry names the owner's
// real folder.
const responses = [
propfindEntry(maskEntryPath(entry), entry, entry.isDir),
];
if (depth !== '0' && isDir && entry) {
if (depth !== '0' && entry.isDir) {
const children = await this.services.fs.listDirectory(
entry.uuid,
{},
);
for (const child of children) {
responses.push(propfindEntry(child.path, child, child.isDir));
}
} else if (depth !== '0' && davPath === '/') {
// Root: list top-level user directories
const rootEntry = await this.stores.fsEntry.getEntryByPath(
`/${actor.user!.username}`,
);
if (rootEntry) {
responses.push(
propfindEntry(rootEntry.path, rootEntry, rootEntry.isDir),
propfindEntry(maskEntryPath(child), child, child.isDir),
);
}
}
this.#sendMultistatus(res, responses);
}
/**
* The root collection: the caller's own home directory, plus one collection
* per person who has shared something with them.
*
* A share is addressed `/<owner>/<uuid>/<name>`, where the uuid stands in
* for the folder the owner keeps it in. Only the `<owner>` segment is a
* child of the root, so without this level nothing shared is reachable from
* a mount at all — which is why a Finder mount showed only the caller's own
* files.
*/
async #rootPropfind(actor: Actor, depth: string | string[]) {
const responses = [propfindEntry('/', null, true)];
if (depth === '0') return responses;
const home = await this.stores.fsEntry.getEntryByPath(
`/${actor.user!.username}`,
);
if (home) {
responses.push(
propfindEntry(maskEntryPath(home), home, home.isDir),
);
}
const owners = new Set<string>();
for (const share of await this.#sharedWithMe(actor)) {
const owner = share.owner?.username;
// Self-owned entries never appear in the share index, but the home
// directory above already covers them if one ever did.
if (owner && owner !== actor.user!.username) owners.add(owner);
}
for (const owner of owners) {
responses.push(propfindEntry(`/${owner}`, { name: owner }, true));
}
return responses;
}
/**
* PROPFIND responses for the virtual collection at `/<owner>`, listing one
* child per item that owner shared with the actor, or null when `davPath`
* isn't a bare username, is the actor's own, is readable for real, or the
* actor holds nothing of theirs — the caller then continues down the normal
* path and gets the 403 or 404 it would otherwise have given.
*
* Children are the `<uuid>` level rather than the shared item itself: the
* item's own href sits one segment deeper, and a collection may only report
* its direct members. Nothing here reaches past that level, so `Depth:
* infinity` on a sharer's collection still enumerates only what they
* shared, and everything below it goes through the ordinary resolve + ACL
* path.
*/
async #shareOwnerPropfind(
actor: Actor,
davPath: string,
entry: FSEntry | null,
depth: string | string[],
): Promise<string[] | null> {
const segments = davPath.split('/').filter(Boolean);
if (segments.length !== 1) return null;
const owner = segments[0]!;
if (owner === actor.user?.username) return null;
// Every provisioned user's home directory is a real entry and a
// sharer is necessarily provisioned, so no directory here means no
// share to stand in for. Checked before the share lookup so an
// unknown name costs a 404 rather than a listing.
if (!entry?.isDir) return null;
// An owner who shared their whole home directory has a real listing
// here, and it beats the virtual one.
if (await this.#canRead(actor, davPath)) return null;
const shares = (await this.#sharedWithMe(actor)).filter(
(share) => share.owner?.username === owner,
);
if (shares.length === 0) return null;
const responses = [propfindEntry(davPath, { name: owner }, true)];
if (depth !== '0') {
for (const share of shares) {
responses.push(
propfindEntry(
`/${owner}/${share.entryUid}`,
{
name: share.name,
uuid: share.entryUid,
modified: share.modified,
},
true,
),
);
}
}
return responses;
}
/**
* What has been shared with the actor, for the virtual collections above.
* One page: WebDAV has no way to ask for the next one, so a caller holding
* more shares than {@link SHARE_LISTING_CAP} sees the oldest that many.
*
* Listing also teaches the request's path masker every root it returns, so
* an entry reached through one keeps that masked prefix on the way back out
* instead of being masked against itself.
*/
async #sharedWithMe(actor: Actor): Promise<ResolvedShare[]> {
const page = await this.services.share.listSharedWithMe(actor, {
limit: SHARE_LISTING_CAP,
});
return page.items;
}
#sendMultistatus(res: Response, responses: string[]): void {
res.status(207)
.set({ 'Content-Type': 'application/xml; charset=utf-8' })
.send(wrapMultistatus(responses.join('\n')));
@@ -445,16 +658,10 @@ export class WebDAVController extends PuterController {
async #proppatch(
res: Response,
davPath: string,
redis: unknown,
lockToken: string | null,
{ davPath, lockToken }: DavContext,
): Promise<void> {
if (
!(await hasWritePermission(
redis as import('ioredis').Cluster,
davPath,
lockToken,
))
!(await hasWritePermission(this.clients.redis, davPath, lockToken))
) {
throw new HttpError(423, 'Locked', { legacyCode: 'conflict' });
}
@@ -470,10 +677,7 @@ export class WebDAVController extends PuterController {
async #mkcol(
req: Request,
res: Response,
actor: Actor,
davPath: string,
redis: unknown,
lockToken: string | null,
{ actor, davPath, lockToken }: DavContext,
): Promise<void> {
if (davPath === '/')
throw new HttpError(403, 'Cannot create at root', {
@@ -488,11 +692,7 @@ export class WebDAVController extends PuterController {
});
}
if (
!(await hasWritePermission(
redis as import('ioredis').Cluster,
davPath,
lockToken,
))
!(await hasWritePermission(this.clients.redis, davPath, lockToken))
) {
throw new HttpError(423, 'Locked', { legacyCode: 'conflict' });
}
@@ -520,10 +720,7 @@ export class WebDAVController extends PuterController {
async #put(
req: Request,
res: Response,
actor: Actor,
davPath: string,
redis: unknown,
lockToken: string | null,
{ actor, davPath, lockToken }: DavContext,
): Promise<void> {
const name = pathPosix.basename(davPath);
if (MACOS_JUNK_REGEX.test(name)) {
@@ -531,11 +728,7 @@ export class WebDAVController extends PuterController {
return;
}
if (
!(await hasWritePermission(
redis as import('ioredis').Cluster,
davPath,
lockToken,
))
!(await hasWritePermission(this.clients.redis, davPath, lockToken))
) {
throw new HttpError(423, 'Locked', { legacyCode: 'conflict' });
}
@@ -580,13 +773,11 @@ export class WebDAVController extends PuterController {
);
const fe = writeResult.fsEntry;
const etag = `"${fe.uuid}-${Math.floor(fe.modified ?? fe.created ?? 0)}"`;
const modified = fe.modified ?? fe.created ?? 0;
res.status(existing ? 204 : 201)
.set({
ETag: etag,
'Last-Modified': new Date(
fe.modified ?? fe.created ?? 0,
).toUTCString(),
ETag: entryEtag(fe.uuid, modified),
'Last-Modified': entryDate(modified).toUTCString(),
})
.end();
}
@@ -595,17 +786,10 @@ export class WebDAVController extends PuterController {
async #delete(
res: Response,
actor: Actor,
davPath: string,
redis: unknown,
lockToken: string | null,
{ actor, davPath, lockToken }: DavContext,
): Promise<void> {
if (
!(await hasWritePermission(
redis as import('ioredis').Cluster,
davPath,
lockToken,
))
!(await hasWritePermission(this.clients.redis, davPath, lockToken))
) {
throw new HttpError(423, 'Locked', { legacyCode: 'conflict' });
}
@@ -626,18 +810,11 @@ export class WebDAVController extends PuterController {
async #copy(
req: Request,
res: Response,
actor: Actor,
davPath: string,
redis: unknown,
lockToken: string | null,
{ actor, davPath, lockToken }: DavContext,
): Promise<void> {
const destPath = this.#parseDestination(req);
const destPath = await this.#parseDestination(req);
if (
!(await hasWritePermission(
redis as import('ioredis').Cluster,
destPath,
lockToken,
))
!(await hasWritePermission(this.clients.redis, destPath, lockToken))
) {
throw new HttpError(423, 'Locked', { legacyCode: 'conflict' });
}
@@ -684,13 +861,10 @@ export class WebDAVController extends PuterController {
async #move(
req: Request,
res: Response,
actor: Actor,
davPath: string,
redis: unknown,
lockToken: string | null,
{ actor, davPath, lockToken }: DavContext,
): Promise<void> {
const destPath = this.#parseDestination(req);
const r = redis as import('ioredis').Cluster;
const destPath = await this.#parseDestination(req);
const r = this.clients.redis;
if (!(await hasWritePermission(r, davPath, lockToken)))
throw new HttpError(423, 'Locked', { legacyCode: 'conflict' });
if (!(await hasWritePermission(r, destPath, lockToken)))
@@ -740,12 +914,9 @@ export class WebDAVController extends PuterController {
async #lock(
req: Request,
res: Response,
actor: Actor,
davPath: string,
redis: unknown,
headerToken: string | null,
{ actor, davPath, lockToken: headerToken }: DavContext,
): Promise<void> {
const r = redis as import('ioredis').Cluster;
const r = this.clients.redis;
// ACL must succeed before any lock state is touched — otherwise
// an authenticated user could lock paths they don't own (e.g. `/`)
@@ -807,10 +978,9 @@ export class WebDAVController extends PuterController {
async #unlock(
req: Request,
res: Response,
davPath: string,
redis: unknown,
{ davPath }: DavContext,
): Promise<void> {
const r = redis as import('ioredis').Cluster;
const r = this.clients.redis;
const tokenHeader = req.headers['lock-token'] as string | undefined;
const token = extractLockToken(tokenHeader);
if (!token)
@@ -833,15 +1003,65 @@ export class WebDAVController extends PuterController {
res.status(204).end();
}
/**
* PROPFIND responses for the virtual collection at `/<owner>/<uuid>`, or
* null when `davPath` isn't that shape, the uuid doesn't resolve to an
* entry of `owner`'s, or the actor cannot read the share — the caller falls
* through to 404 in every null case, so an unauthorized probe learns
* nothing about whether the uuid exists.
*/
async #shareRootParentPropfind(
actor: Actor,
davPath: string,
depth: string | string[],
): Promise<string[] | null> {
const parsed = parseMaskedSharePath(davPath);
if (!parsed || parsed.tail !== '') return null;
if (parsed.ownerUsername === actor.user?.username) return null;
const root = await this.stores.fsEntry.getEntryByUuid(parsed.rootUuid);
if (!root) return null;
// Same guard as resolveSharePath: the mask names the owner, so the
// uuid must be theirs.
if (root.path.split('/')[1] !== parsed.ownerUsername) return null;
const allowed = await this.services.acl.check(
actor,
{
path: root.path,
resolveAncestors: () =>
this.services.fs.getAncestorChain(root.path),
},
'read',
);
if (!allowed) return null;
const maskedRoot = `/${parsed.ownerUsername}/${root.uuid}/${root.name}`;
// The uuid segment has no name of its own, so it borrows the shared
// item's — a client that reads `displayname` shows something
// recognizable rather than a bare uuid.
const responses = [propfindEntry(davPath, { name: root.name }, true)];
if (depth !== '0') {
responses.push(propfindEntry(maskedRoot, root, root.isDir));
}
return responses;
}
// -- ACL helpers -------------------------------------------------
async #canRead(actor: Actor, path: string): Promise<boolean> {
return await this.services.acl.check(
actor,
{
path,
resolveAncestors: () => this.services.fs.getAncestorChain(path),
},
'read',
);
}
async #assertRead(actor: Actor, path: string): Promise<void> {
const descriptor = {
path,
resolveAncestors: () => this.services.fs.getAncestorChain(path),
};
const ok = await this.services.acl.check(actor, descriptor, 'read');
if (!ok)
if (!(await this.#canRead(actor, path)))
throw new HttpError(403, 'Permission denied', {
legacyCode: 'permission_denied',
});
@@ -890,21 +1110,72 @@ export class WebDAVController extends PuterController {
// -- Misc helpers ------------------------------------------------
#parseDestination(req: Request): string {
async #parseDestination(req: Request): Promise<string> {
const dest = req.headers.destination as string | undefined;
if (!dest)
throw new HttpError(400, 'Missing Destination header', {
legacyCode: 'bad_request',
});
let raw: string;
try {
const url = new URL(dest, `http://${req.headers.host}`);
return decodeURIComponent(url.pathname);
raw = decodeURIComponent(url.pathname);
} catch {
return decodeURIComponent(dest);
raw = decodeURIComponent(dest);
}
// The destination is addressed the same way as the request path, so a
// client that browsed into a share names its target the same way too.
return resolveSharePath(this.stores.fsEntry, Context.get('actor'), raw);
}
}
// -- Middleware helpers -----------------------------------------------
/**
* Answer a CORS preflight before anything can reject it. True when this request
* was one and has been answered.
*
* A browser sends a credential-less `OPTIONS` ahead of any cross-origin
* PROPFIND/GET/PUT and treats a non-2xx reply as a failed preflight, never
* sending the real request. The authenticated OPTIONS path answers 401, which
* blocked every browser DAV client — the VS Code app's mount included — before
* it could present a token.
*
* Only an actual preflight is answered here: it's the one OPTIONS that carries
* `Access-Control-Request-Method`. A native client's plain OPTIONS still goes
* through auth, so macOS keeps getting the 401-with-`DAV:` reply it opens a
* mount with.
*/
function answerPreflight(req: Request, res: Response): boolean {
if (!req.headers.origin || !req.headers['access-control-request-method']) {
return false;
}
// `Allow-Origin`/`-Methods`/`-Headers` were set on the way in by the
// server's CORS middleware. `Max-Age` is what belongs here specifically:
// without it a browser re-preflights every URL every few seconds, doubling
// the request count of a mount and its share of the DAV rate limit.
res.status(200)
.set({
...DAV_HEADERS,
Allow: ALLOW_METHODS,
'Access-Control-Max-Age': PREFLIGHT_MAX_AGE,
'Content-Length': '0',
})
.end();
return true;
}
/** DAV clients read a status and a plain-text reason, not our JSON error shape. */
function sendDavError(res: Response, err: unknown): void {
if (res.headersSent) return;
if (err instanceof HttpError) {
res.status(err.statusCode).send(err.message);
return;
}
console.error('[webdav] unhandled error', err);
res.status(500).send('Internal Server Error');
}
// -- XML helpers ------------------------------------------------------
function escapeXml(text: string): string {
@@ -920,9 +1191,42 @@ function wrapMultistatus(inner: string): string {
return `<?xml version="1.0" encoding="utf-8"?>\n<D:multistatus xmlns:D="DAV:">\n${inner}\n</D:multistatus>`;
}
/**
* FSEntry timestamps are Unix seconds. Entries with nothing stored fall back to
* an ISO string literal, so both forms have to be accepted here.
*/
function toEpochSeconds(ts: number | string): number {
return typeof ts === 'number'
? Math.floor(ts)
: Math.floor(new Date(ts).getTime() / 1000);
}
/** `Date` takes milliseconds, so entry seconds must be scaled to format them. */
function entryDate(ts: number | string): Date {
return new Date(toEpochSeconds(ts) * 1000);
}
/**
* Opaque validator. Built from seconds so every verb emits the same ETag for an
* entry — a client that gets one value from PROPFIND and another from GET
* treats the resource as changed and re-fetches it on every pass.
*/
function entryEtag(uid: string, ts: number | string): string {
return `"${uid}-${toEpochSeconds(ts)}"`;
}
/**
* What a `<D:response>` needs to describe one resource. An {@link FSEntry}
* satisfies it; the virtual collections a share mask invents pass whatever they
* know, which is sometimes only a name.
*/
type PropfindTarget = Partial<
Pick<FSEntry, 'name' | 'uuid' | 'modified' | 'created' | 'size'>
>;
function propfindEntry(
href: string,
entry: FSEntry | null,
entry: PropfindTarget | null,
isDir: boolean,
): string {
const encodedHref =
@@ -932,14 +1236,13 @@ function propfindEntry(
const created = entry?.created ?? '2025-01-01T00:00:00Z';
const name = entry?.name ?? (pathPosix.basename(href) || '/');
const uid = entry?.uuid ?? 'root';
const modTs = Math.floor(new Date(modified as string).getTime());
let props = `
<D:displayname>${escapeXml(String(name))}</D:displayname>
<D:getlastmodified>${new Date(modified as string).toUTCString()}</D:getlastmodified>
<D:creationdate>${new Date(created as string).toISOString()}</D:creationdate>
<D:getlastmodified>${entryDate(modified).toUTCString()}</D:getlastmodified>
<D:creationdate>${entryDate(created).toISOString()}</D:creationdate>
<D:resourcetype>${isDir ? '<D:collection/>' : ''}</D:resourcetype>
<D:getetag>"${uid}-${modTs}"</D:getetag>
<D:getetag>${entryEtag(uid, modified)}</D:getetag>
<D:supportedlock>
<D:lockentry><D:lockscope><D:exclusive/></D:lockscope><D:locktype><D:write/></D:locktype></D:lockentry>
<D:lockentry><D:lockscope><D:shared/></D:lockscope><D:locktype><D:write/></D:locktype></D:lockentry>
@@ -949,7 +1252,7 @@ function propfindEntry(
if (!isDir && entry) {
props += `\n <D:getcontentlength>${entry.size ?? 0}</D:getcontentlength>`;
const mime = mimeFromExt(pathPosix.extname(entry.name));
const mime = mimeFromExt(pathPosix.extname(name));
props += `\n <D:getcontenttype>${escapeXml(mime)}</D:getcontenttype>`;
}
+33 -19
View File
@@ -61,23 +61,22 @@ export type LegacyErrorCodes =
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
* Puter is free software: you can redistribute it and/or modify it under the
* terms of the GNU Affero General Public License as published by the Free
* Software Foundation, either version 3 of the License, or (at your option) any
* later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
* details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
* along with this program. If not, see
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
*/
/**
* Options accepted by `HttpError`. All optional.
*/
/** Options accepted by `HttpError`. All optional. */
export interface HttpErrorOptions {
/** Underlying error. Set as the standard `Error.cause`. */
cause?: unknown;
@@ -89,25 +88,38 @@ export interface HttpErrorOptions {
legacyCode?: LegacyErrorCodes | (string & {});
/**
* Modern, structured error code. If both `legacyCode` and `code` are set,
* the legacy one takes the `code` slot in the response body and `code`
* is emitted as `errorCode`, so clients keying on either field find
* what they expect.
* the legacy one takes the `code` slot in the response body and `code` is
* emitted as `errorCode`, so clients keying on either field find what they
* expect.
*/
code?: string;
/** Additional fields merged into the response body. */
fields?: Record<string, unknown>;
/**
* Skip the terminal alarm gate for this error. Set it only where the call
* site already knows the failure is expected and arrives in proportion to
* traffic — an upstream rate limit on a zero-cost model, say. Never
* serialized to the client.
*/
noAlarm?: boolean;
}
/**
* The single error type controllers and services throw to surface an HTTP
* failure. The terminal `errorHandler` middleware catches it, serializes a
* JSON body, and sets the response status.
* failure. The terminal `errorHandler` middleware catches it, serializes a JSON
* body, and sets the response status.
*
* Usage:
*
* ```ts
* throw new HttpError(404, 'Item not found');
* throw new HttpError(409, 'Cannot overwrite directory', { legacyCode: 'is_directory' });
* throw new HttpError(403, 'Forbidden', { legacyCode: 'forbidden', fields: { target } });
* throw new HttpError(409, 'Cannot overwrite directory', {
* legacyCode: 'is_directory',
* });
* throw new HttpError(403, 'Forbidden', {
* legacyCode: 'forbidden',
* fields: { target },
* });
* ```
*
* Express 5 forwards thrown errors (sync and async) to error-handling
@@ -118,6 +130,7 @@ export class HttpError extends Error {
readonly legacyCode?: LegacyErrorCodes | (string & {});
readonly code?: string;
readonly fields?: Record<string, unknown>;
readonly noAlarm?: boolean;
constructor(
statusCode: number,
@@ -133,6 +146,7 @@ export class HttpError extends Error {
this.legacyCode = options.legacyCode;
this.code = options.code;
this.fields = options.fields;
this.noAlarm = options.noAlarm;
}
}
+79
View File
@@ -0,0 +1,79 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { describe, expect, it } from 'vitest';
import { Controller, Get, Propfind } from './decorators.ts';
import { PuterRouter } from './PuterRouter.ts';
interface Registrable {
registerRoutes: (router: PuterRouter) => void;
}
const collect = (controller: Registrable) => {
const router = new PuterRouter();
controller.registerRoutes(router);
return router.routes;
};
@Controller('/demo')
class DemoController {
@Get('/thing')
thing() {}
@Propfind('/{*splat}', { subdomain: 'dav' })
browse() {}
}
describe('route decorators', () => {
it('collects each decorated method once, in declaration order', () => {
const routes = collect(new DemoController() as unknown as Registrable);
expect(routes.map((r) => r.method)).toEqual(['get', 'propfind']);
expect(routes[1]!.options).toEqual({ subdomain: 'dav' });
});
it('does not re-collect when a second instance is constructed', () => {
// Method initializers run per instance but collect onto the shared
// prototype, so without a guard a process that builds two servers
// registers every route on the second one twice over.
const first = collect(new DemoController() as unknown as Registrable);
const second = collect(new DemoController() as unknown as Registrable);
const third = collect(new DemoController() as unknown as Registrable);
expect(second).toHaveLength(first.length);
expect(third).toHaveLength(first.length);
expect(third.map((r) => r.method)).toEqual(['get', 'propfind']);
});
it('binds handlers to the instance that registered them', () => {
@Controller()
class Bound {
marker = 'mine';
seen: string | undefined;
@Get('/x')
handle() {
this.seen = this.marker;
}
}
const instance = new Bound();
const [route] = collect(instance as unknown as Registrable);
(route!.handler as unknown as () => void)();
expect(instance.seen).toBe('mine');
});
});
+15 -1
View File
@@ -143,7 +143,21 @@ const createMethodDecorator = (method: Exclude<RouteMethod, 'use'>) => {
const proto = Object.getPrototypeOf(
this as object,
) as DecoratedPrototype;
getOrInitRoutes(proto).push({
const routes = getOrInitRoutes(proto);
// Initializers run once per *instance* but collect onto the
// shared prototype, so a process that builds a second server
// (every multi-server test file) would otherwise register
// every route twice. The decorated method and the path literal
// are the same references each time, so identity recognizes a
// repeat without comparing option objects.
const already = routes.some(
(r) =>
r.handler === (target as unknown as RequestHandler) &&
r.method === method &&
r.path === path,
);
if (already) return;
routes.push({
method,
path,
options,
+2 -2
View File
@@ -21,7 +21,7 @@ import type { RequestHandler } from 'express';
import type { IConfig } from '../../../types';
import {
assertActorHasCredits,
type CreditMeteringLike,
type CreditMetering,
} from '../../../services/metering/enforcement.js';
import '../expressAugmentation';
@@ -38,7 +38,7 @@ import '../expressAugmentation';
* affordable on routes that are called hundreds of times a minute.
*/
export const requireCreditsGate = (
metering: CreditMeteringLike | undefined,
metering: CreditMetering | undefined,
config: IConfig,
): RequestHandler => {
return (req, _res, next) => {
@@ -28,6 +28,8 @@ import {
assertNotUserSession,
noUserSessionGate,
requireAuthGate,
requireCardVerifiedGate,
requirePhoneVerifiedGate,
requireVerifiedAccount,
requireNonAccessTokenGate,
requireUserActorGate,
@@ -803,3 +805,70 @@ describe('allowedAppIdsGate', () => {
expectHttpError(got, 403, 'forbidden');
});
});
// ── requirePhoneVerifiedGate / requireCardVerifiedGate ──────────────
describe('requirePhoneVerifiedGate', () => {
it('passes a user with a verified number on file', () => {
const got = runGate(requirePhoneVerifiedGate(), {
actor: { user: { uuid: 'u-1', phone: '+15550000000' } },
});
expect(got).toBeUndefined();
});
it('rejects a user who was never asked to verify a phone', () => {
// The pending flag is clear, so the default-on account gate lets this
// user through — an opt-in route asking for the factor itself must not.
const got = runGate(requirePhoneVerifiedGate(), {
actor: { user: { uuid: 'u-1' } },
});
expectHttpError(got, 403, 'phone_verification_required');
});
it('rejects a user still mid-verification, stale number and all', () => {
const got = runGate(requirePhoneVerifiedGate(), {
actor: {
user: {
uuid: 'u-1',
phone: '+15550000000',
requires_phone_verification: true,
},
},
});
expectHttpError(got, 403, 'phone_verification_required');
});
it('rejects when there is no actor at all', () => {
const got = runGate(requirePhoneVerifiedGate(), {});
expectHttpError(got, 403, 'phone_verification_required');
});
});
describe('requireCardVerifiedGate', () => {
it('passes a user with a verified card on file', () => {
const got = runGate(requireCardVerifiedGate(), {
actor: { user: { uuid: 'u-1', card_fingerprint: 'fp_1' } },
});
expect(got).toBeUndefined();
});
it('rejects a user who never verified a card', () => {
const got = runGate(requireCardVerifiedGate(), {
actor: { user: { uuid: 'u-1' } },
});
expectHttpError(got, 403, 'card_verification_required');
});
it('rejects a user still carrying the card gate', () => {
const got = runGate(requireCardVerifiedGate(), {
actor: {
user: {
uuid: 'u-1',
card_fingerprint: 'fp_1',
requires_card_verification: true,
},
},
});
expectHttpError(got, 403, 'card_verification_required');
});
});
+76 -2
View File
@@ -378,17 +378,91 @@ export const assertVerifiedAccount = (
403,
'Please verify your phone number to continue',
{
legacyCode: 'phone_verification_required' as never,
legacyCode: 'phone_verification_required',
},
);
}
if (user?.requires_card_verification) {
throw new HttpError(403, 'Please verify your card to continue', {
legacyCode: 'card_verification_required' as never,
legacyCode: 'card_verification_required',
});
}
};
// -- Per-verification gates ------------------------------------------
//
// `requireVerifiedAccount` above is the default-on gate: it turns away an
// account that is *pending* a verification the abuse harness asked of it, and
// says nothing about accounts that were never asked. The two gates below are
// the opt-in counterparts of `requireVerified` (email) for the other two
// factors: they require the verification to have actually happened, whether or
// not this account was ever flagged for it.
//
// Neither factor has a boolean "verified" column — the pending flag is cleared
// on success and the proof is the artifact left behind. `phone` is written only
// once a code has been confirmed, and `card_fingerprint` only once a card has
// been checked (both are also what `UserAccountService` reads as the
// phone-verified / card-verified signals). Pending flag clear AND artifact
// present is therefore the honest reading of "verified", and an account still
// mid-flow fails on the flag rather than slipping through on a stale artifact.
/**
* Reject unless the user has a phone number verified on file. 403
* `phone_verification_required`, the same code the pending-verification gate
* uses, so a client that already prompts for the SMS flow prompts here too.
*/
export const assertPhoneVerified = (
user:
{ phone?: unknown; requires_phone_verification?: unknown } | undefined,
): void => {
if (user?.phone && !user?.requires_phone_verification) return;
throw new HttpError(403, 'Please verify your phone number to continue', {
legacyCode: 'phone_verification_required',
});
};
/** Route-option form of {@link assertPhoneVerified} (`requirePhoneVerified`). */
export const requirePhoneVerifiedGate = (): RequestHandler => {
return (req, _res, next) => {
try {
assertPhoneVerified(req.actor?.user);
} catch (err) {
next(err);
return;
}
next();
};
};
/**
* Reject unless the user has a card verified on file. 403
* `card_verification_required`, matching the pending-verification gate so the
* client shows the card flow it already has.
*/
export const assertCardVerified = (
user:
| { card_fingerprint?: unknown; requires_card_verification?: unknown }
| undefined,
): void => {
if (user?.card_fingerprint && !user?.requires_card_verification) return;
throw new HttpError(403, 'Please verify your card to continue', {
legacyCode: 'card_verification_required',
});
};
/** Route-option form of {@link assertCardVerified} (`requireCardVerified`). */
export const requireCardVerifiedGate = (): RequestHandler => {
return (req, _res, next) => {
try {
assertCardVerified(req.actor?.user);
} catch (err) {
next(err);
return;
}
next();
};
};
export const assertNotSuspended = (
user: { suspended?: unknown } | undefined,
): void => {
@@ -1206,3 +1206,166 @@ describe('createPuterSiteMiddleware — .puter_site_config', () => {
expect(String(out.body)).toContain('Not Found');
});
});
// ── __workers/ (worker source) ──────────────────────────────────────
//
// `__workers/` at a site root holds server-side worker source. It is the
// one thing under a site root that is NOT public: users' backend code can
// carry trade secrets, so every route to those bytes — direct request,
// folder fallback, custom error page — must dead-end in the same 404 a
// missing path produces.
describe('createPuterSiteMiddleware — __workers/ is never served', () => {
const setupSiteWithWorker = async () => {
const owner = await makeUserWithHome();
const homePath = `/${owner.username}`;
const homeEntry = await server.stores.fsEntry.getEntryByPath(homePath);
const sub = `dw-${Math.random().toString(36).slice(2, 8)}`;
await server.stores.subdomain.create({
userId: owner.id,
subdomain: sub,
rootDirId: homeEntry!.id,
});
await server.services.fs.mkdir(owner.id, {
path: `${homePath}/__workers`,
createMissingParents: true,
});
await writeFile(
owner.id,
`${homePath}/__workers/matchmaking.worker.js`,
Buffer.from('export default { secret: true };'),
'application/javascript',
);
return { owner, homePath, sub };
};
const request = async (sub: string, path: string) => {
const mw = buildMiddleware();
const { res, out } = makeRes();
await mw(
makeReq({ hostname: `${sub}.site.puter.localhost`, path }),
res,
vi.fn(),
);
await new Promise<void>((resolve) => setImmediate(resolve));
return out;
};
it('404s a direct request for a worker source file that exists', async () => {
const { sub } = await setupSiteWithWorker();
const out = await request(sub, '/__workers/matchmaking.worker.js');
expect(out.statusCode).toBe(404);
expect(String(out.body)).not.toContain('secret');
});
it('404s the folder itself — no index.html fallback into it', async () => {
const { owner, homePath, sub } = await setupSiteWithWorker();
await writeFile(
owner.id,
`${homePath}/__workers/index.html`,
Buffer.from('<html>listing</html>'),
'text/html',
);
for (const path of ['/__workers', '/__workers/']) {
const out = await request(sub, path);
expect(out.statusCode).toBe(404);
expect(String(out.body)).not.toContain('listing');
}
});
it('404s case variants and URL-encoded spellings of the folder', async () => {
const { sub } = await setupSiteWithWorker();
for (const path of [
'/__Workers/matchmaking.worker.js',
'/__WORKERS/matchmaking.worker.js',
'/%5F%5Fworkers/matchmaking.worker.js',
'/a/../__workers/matchmaking.worker.js',
]) {
const out = await request(sub, path);
expect(out.statusCode).toBe(404);
expect(String(out.body)).not.toContain('secret');
}
});
it('404s __workers/ at any depth — a site rooted above another site must not serve the inner one', async () => {
// Same FS layout, but the serving site is rooted one level up:
// the worker source now sits at /inner/__workers/… from its
// point of view, and must be just as unreachable.
const owner = await makeUserWithHome();
const homePath = `/${owner.username}`;
await server.services.fs.mkdir(owner.id, {
path: `${homePath}/inner/__workers`,
createMissingParents: true,
});
await writeFile(
owner.id,
`${homePath}/inner/__workers/matchmaking.worker.js`,
Buffer.from('export default { secret: true };'),
'application/javascript',
);
const homeEntry = await server.stores.fsEntry.getEntryByPath(homePath);
const sub = `dwo-${Math.random().toString(36).slice(2, 8)}`;
await server.stores.subdomain.create({
userId: owner.id,
subdomain: sub,
rootDirId: homeEntry!.id,
});
const out = await request(
sub,
'/inner/__workers/matchmaking.worker.js',
);
expect(out.statusCode).toBe(404);
expect(String(out.body)).not.toContain('secret');
});
it('refuses a custom error page that points into __workers/', async () => {
// `errors.404.file` aimed at worker source would publish it on
// every missing path — the rule is ignored and the default 404
// page served instead.
const { owner, homePath, sub } = await setupSiteWithWorker();
await writeFile(
owner.id,
`${homePath}/.puter_site_config`,
Buffer.from(
JSON.stringify({
errors: {
'404': {
file: '/__workers/matchmaking.worker.js',
status: 200,
},
},
}),
),
'application/json',
);
const out = await request(sub, '/no-such-page');
expect(out.statusCode).toBe(404);
expect(String(out.body)).not.toContain('secret');
expect(String(out.body)).toContain('Not Found');
});
it('still applies an SPA fallback to __workers/ requests when the target is outside the folder', async () => {
// The block hides the folder; it does not exempt those URLs from
// the site's own 404 handling. A fallback pointing at a public
// file keeps working — the fiction is "this path does not exist",
// and this is exactly what a missing path serves.
const { owner, homePath, sub } = await setupSiteWithWorker();
const shell = Buffer.from('<html>spa-shell</html>');
await writeFile(owner.id, `${homePath}/index.html`, shell, 'text/html');
await writeFile(
owner.id,
`${homePath}/.puter_site_config`,
Buffer.from(
JSON.stringify({
errors: { '404': { file: '/index.html', status: 200 } },
}),
),
'application/json',
);
const out = await request(sub, '/__workers/matchmaking.worker.js');
expect(out.statusCode).toBe(200);
const piped = out.body as Buffer | undefined;
expect(Buffer.isBuffer(piped)).toBe(true);
expect(piped!.equals(shell)).toBe(true);
});
});
+33 -4
View File
@@ -71,6 +71,24 @@ import {
* serving.
*/
/**
* Reserved folder at a site root holding server-side worker source
* (`__workers/<name>.worker.js`). Its contents are backend code — users' trade
* secrets live there — so nothing under it is ever served: not directly, not
* via the folder→index.html fallback, and not as a custom error page target.
* Matched on the decoded, normalized path, and case-insensitively, so no
* alternate spelling of the same entry can reach the bytes.
*
* Matched at ANY depth, not just the site root: site roots can nest, and a site
* rooted above another site would otherwise serve the inner site's `__workers/`
* as ordinary content under `/inner/__workers/...`.
*/
const WORKERS_FOLDER = '__workers';
const isWorkersSourcePath = (urlPath: string): boolean =>
urlPath
.split('/')
.some((segment) => segment.toLowerCase() === WORKERS_FOLDER);
const SUBDOMAIN_404 = `<div style="font-size: 20px;
text-align: center;
height: calc(100vh);
@@ -493,9 +511,14 @@ export const createPuterSiteMiddleware = (
// Subdomain hosting bypasses ACL by design: anything the owner placed
// under the registered root_dir is treated as public. Path traversal
// is blocked above by `pathPosix.normalize` anchoring at `/`.
let entry = isConfigRequest
? null
: await layers.stores.fsEntry.getEntryByPath(filePath);
// `__workers/` is carved out of that bypass: worker source is the one
// thing under a site root that is NOT public. Treated exactly like the
// config file — no lookup at all, so the folder→index.html fallback
// below can't resolve into it either.
let entry =
isConfigRequest || isWorkersSourcePath(resolvedUrlPath)
? null
: await layers.stores.fsEntry.getEntryByPath(filePath);
if (entry?.isDir) {
// Folder request → fall back to <folder>/index.html, the same
// way `/` is rewritten to `/index.html` at the site root above.
@@ -513,7 +536,13 @@ export const createPuterSiteMiddleware = (
let statusOverride: number | undefined;
if (!entry || entry.isDir) {
const errorTarget = resolveErrorTarget(siteConfig, 404, rootPath);
if (errorTarget) {
// A config pointing `errors.404.file` into `__workers/` would
// publish worker source through the error page — refuse it the
// same way a nonexistent target is refused.
if (
errorTarget &&
!isWorkersSourcePath(errorTarget.absPath.slice(rootPath.length))
) {
const candidate = await layers.stores.fsEntry.getEntryByPath(
errorTarget.absPath,
);
+42 -18
View File
@@ -116,6 +116,20 @@ async function checkMemory(key, limit, windowMs) {
// -- Redis backend ---------------------------------------------------
// ioredis MULTI/EXEC reports per-command failures inside the exec() result
// (as `[err, res]` pairs) rather than throwing, so a failed command's result
// reads as `undefined` — and `Number(undefined)` is NaN, which every
// comparison below treats as "under the limit". Pull results through this so
// a command failure surfaces like a thrown one instead of silently admitting.
function multiResult(results, i) {
const entry = results[i];
if (Array.isArray(entry)) {
if (entry[0]) throw entry[0];
return entry[1];
}
return entry;
}
async function checkRedis(
/** @type {import('ioredis').Cluster} */
redis,
@@ -144,9 +158,7 @@ async function checkRedis(
.pexpire(redisKey, windowMs)
.exec();
const count = Number(
Array.isArray(results[2]) ? results[2][1] : results[2],
);
const count = Number(multiResult(results, 2));
if (count > limit) {
await redis.zrem(redisKey, member);
return false;
@@ -216,7 +228,7 @@ async function acquireMemoryConcurrent(key, limit) {
};
}
async function acquireRedisConcurrent(redis, key, limit) {
async function acquireRedisConcurrent(redis, key, limit, retried = false) {
const redisKey = `concurrent:${key}`;
const member = `${Date.now()}-${crypto.randomUUID()}`;
// One sorted-set member per held slot, scored by acquire time — the same
@@ -231,20 +243,32 @@ async function acquireRedisConcurrent(redis, key, limit) {
// nobody is actually using. Here the sweep below drops each slot on its own
// age, so a leak drains on schedule no matter how hard anyone retries.
const now = Date.now();
const results = await redis
.multi()
// Slots older than the orphan window belonged to a process that died
// before releasing; drop them before counting.
.zremrangebyscore(redisKey, 0, now - ORPHAN_SAFETY_TTL_MS)
.zadd(redisKey, now, member)
.zcard(redisKey)
// Key-level TTL is only garbage collection for a bucket that goes
// quiet — the per-member sweep above is what bounds a live one.
.expire(redisKey, ORPHAN_SAFETY_TTL_SEC)
.exec();
const count = Number(
Array.isArray(results[2]) ? results[2][1] : results[2],
);
let count;
try {
const results = await redis
.multi()
// Slots older than the orphan window belonged to a process that
// died before releasing; drop them before counting.
.zremrangebyscore(redisKey, 0, now - ORPHAN_SAFETY_TTL_MS)
.zadd(redisKey, now, member)
.zcard(redisKey)
// Key-level TTL is only garbage collection for a bucket that goes
// quiet — the per-member sweep above is what bounds a live one.
.expire(redisKey, ORPHAN_SAFETY_TTL_SEC)
.exec();
count = Number(multiResult(results, 2));
} catch (err) {
// Keys left behind by the INCR-counter version of this backend are
// plain strings, so every zset command above fails WRONGTYPE — while
// the EXPIRE at the end still succeeds, meaning steady traffic keeps
// refreshing the stale key and it never ages out on its own. Drop the
// legacy key and count against a clean one.
if (!retried && /WRONGTYPE/.test(err?.message ?? '')) {
await redis.del(redisKey);
return acquireRedisConcurrent(redis, key, limit, true);
}
throw err;
}
if (count > limit) {
await redis.zrem(redisKey, member);
return { ok: false };
@@ -959,6 +959,28 @@ describe('acquireConcurrent — orphan recovery (redis)', () => {
expect((await acquireConcurrent(key, 1)).ok).toBe(true);
});
it('heals a legacy string counter key and enforces the limit again', async () => {
// The pre-zset implementation stored these buckets as INCR string
// counters. Against such a key every zset command fails WRONGTYPE
// inside the MULTI while the trailing EXPIRE succeeds — so traffic
// kept the stale key alive indefinitely, and the undefined zcard
// result (NaN) admitted every caller unbounded.
// The file-level afterEach resets wiring to memory after every test,
// so re-point the default at this suite's redis for this test.
configureRateLimit({ default: 'redis', redis });
const key = 'legacy-counter';
await redis.set(`concurrent:${key}`, '7');
const first = await acquireConcurrent(key, 1);
expect(first.ok).toBe(true);
// The key was rebuilt as a zset and the cap is live again.
expect(await redis.type(`concurrent:${key}`)).toBe('zset');
expect((await acquireConcurrent(key, 1)).ok).toBe(false);
await first.release();
expect((await acquireConcurrent(key, 1)).ok).toBe(true);
});
});
// ── concurrency: subscription-based limits ──────────────────────────
@@ -0,0 +1,81 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, Response } from 'express';
import { describe, expect, it, vi } from 'vitest';
import type { IConfig } from '../../../types';
import { isHttpError } from '../HttpError';
import { requireSubscriptionGate } from './subscription';
// The decision itself is covered in `services/metering/enforcement.test.ts`;
// what matters here is that the gate resolves it and hands the outcome to
// `next()` rather than leaving the request hanging or throwing async.
const runGate = (
gate: ReturnType<typeof requireSubscriptionGate>,
plan: string,
): Promise<unknown> =>
new Promise((resolve) => {
gate(
{ actor: { user: { uuid: 'u-1' } } } as unknown as Request,
{} as Response,
resolve as (err?: unknown) => void,
);
});
const metering = (id: string) => ({
getActorSubscription: vi.fn().mockResolvedValue({ id }),
});
const config = {} as IConfig;
describe('requireSubscriptionGate', () => {
it('calls next() with nothing for a subscriber', async () => {
const got = await runGate(
requireSubscriptionGate(metering('business'), config, true),
'business',
);
expect(got).toBeUndefined();
});
it('calls next(err) with a 402 for a free account', async () => {
const got = await runGate(
requireSubscriptionGate(metering('user_free'), config, true),
'user_free',
);
expect(isHttpError(got)).toBe(true);
expect(got).toMatchObject({
statusCode: 402,
legacyCode: 'subscription_required',
});
});
it('passes a rejected lookup to next() instead of leaving it unhandled', async () => {
const broken = {
getActorSubscription: vi
.fn()
.mockRejectedValue(new Error('store down')),
};
const got = await runGate(
requireSubscriptionGate(broken, config, true),
'irrelevant',
);
expect(got).toBeInstanceOf(Error);
});
});
@@ -0,0 +1,55 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { RequestHandler } from 'express';
import {
assertActorHasSubscription,
type SubscriptionMetering,
type SubscriptionRequirement,
} from '../../../services/metering/enforcement.js';
import type { IConfig } from '../../../types';
import '../expressAugmentation';
/**
* Reject a caller whose plan doesn't include this surface, for routes that opt
* in with `requireSubscription`. Off unless a route asks for it — no surface is
* subscriber-only by default.
*
* `true` accepts any non-free plan; an array of policy ids accepts only those.
* Either way the decision is `assertActorHasSubscription`, which the driver
* dispatch path calls directly — `/drivers/call` is one shared route, so a
* per-driver requirement can't ride on the route chain.
*/
export const requireSubscriptionGate = (
metering: SubscriptionMetering | undefined,
config: IConfig,
requirement: SubscriptionRequirement,
): RequestHandler => {
return (req, _res, next) => {
assertActorHasSubscription(
metering,
req.actor,
requirement,
config,
).then(
() => next(),
(err) => next(err),
);
};
};
+49 -3
View File
@@ -67,8 +67,9 @@ export type RoutePath = string | RegExp | Array<string | RegExp>;
* middleware chain in this order:
*
* subdomain → requireAuth (+ suspended) → emailConfirmed →
* requireUserActor → adminOnly → allowedAppIds → rateLimit →
* requireCredits → concurrent → caller `middleware: []` → handler
* requireUserActor → adminOnly → allowedAppIds → phoneVerified →
* cardVerified → requireSubscription → rateLimit → requireCredits →
* concurrent → caller `middleware: []` → handler
*
* `requireUserActor`, `adminOnly`, and `allowedAppIds` all imply `requireAuth`;
* the materializer dedupes so only one auth gate ends up in the chain.
@@ -191,6 +192,24 @@ export interface RouteOptions {
*/
requireVerified?: boolean;
/**
* Reject unless the user has a phone number verified on file (verified at
* some point, not merely never asked to). 403
* `phone_verification_required`. Implies `requireAuth`; independent of the
* default-on pending-verification gate, which only turns away accounts the
* abuse harness flagged.
*
* Opt-in per route, for surfaces worth the friction of an SMS round trip.
*/
requirePhoneVerified?: boolean;
/**
* Reject unless the user has a card verified on file. 403
* `card_verification_required`. Same shape as `requirePhoneVerified` — the
* opt-in, "must have actually done it" counterpart of the pending gate.
*/
requireCardVerified?: boolean;
/**
* Per-route JSON body parsing override. By default the global parser
* handles every `application/json` request with a 50mb limit and stashes
@@ -328,6 +347,24 @@ export interface RouteOptions {
*/
requireCredits?: boolean;
/**
* Reject a caller whose plan doesn't include this route with 402
* `subscription_required`.
*
* `true` accepts any plan that isn't one of the free ones, so a plan
* registered by an extension counts without being named here. An array of
* `SubscriptionPolicy.id`s accepts only those, for a feature that belongs
* to specific plans (`requireSubscription: ['business', 'pro']`).
*
* Opt-in: nothing is subscriber-only unless it says so. Implies
* `requireAuth` — there is no plan to read off an anonymous caller. The
* answer comes from the metering service's per-actor subscription cache, so
* the check costs a map lookup on a warm actor. Distinct from
* `requireCredits`, which asks whether an account has budget left rather
* than which plan it is on; a route may want both.
*/
requireSubscription?: boolean | string[];
// Reserved — wire as the corresponding features/services land:
// bodyFiles?: string[]; // multer-style multipart fields
// responseTimeout?: number;
@@ -393,7 +430,16 @@ export type AuthRequired<O extends RouteOptions> = O extends {
? true
: O extends { noUserSession: true }
? true
: false;
: O extends { requirePhoneVerified: true }
? true
: O extends { requireCardVerified: true }
? true
: O extends {
requireSubscription:
true | readonly string[] | string[];
}
? true
: false;
/** Express `Request` with `actor` narrowed based on the route's options. */
export type TypedRequest<O extends RouteOptions> = Omit<Request, 'actor'> & {
@@ -22,6 +22,6 @@ import { toMicroCents } from '../../services/metering/utils.js';
export const REGISTERED_USER_FREE = {
id: DEFAULT_FREE_SUBSCRIPTION,
monthUsageAllowance: toMicroCents(0.25),
monthUsageAllowance: toMicroCents(0.5),
monthlyStorageAllowance: 100 * 1024 * 1024, // 100MiB
};
@@ -62,6 +62,7 @@ const FULL_PROVIDER_CONFIG = {
'azure-openai': { apiKey: 'k', apiURL: 'https://azure.test/openai/v1' },
'openai-completion': { apiKey: 'k' },
gemini: { apiKey: 'k' },
meta: { apiKey: 'k' },
groq: { apiKey: 'k' },
deepseek: { apiKey: 'k' },
mistral: { apiKey: 'k' },
@@ -73,6 +74,7 @@ const FULL_PROVIDER_CONFIG = {
'together-ai': { apiKey: 'k' },
openrouter: { apiKey: 'k', apiBaseUrl: 'https://openrouter.test' },
infron: { apiKey: 'k' },
byteplus: { apiKey: 'k' },
neuralwatt: { apiKey: 'k' },
// Suppress auto-discovery of a developer's local Ollama.
ollama: { enabled: false },
@@ -148,10 +150,13 @@ const completeFake = (args: Record<string, unknown>) =>
} as never),
);
const errorFor = async (thrown: unknown): Promise<HttpError> => {
const errorFor = async (
thrown: unknown,
args: Record<string, unknown> = {},
): Promise<HttpError> => {
vi.spyOn(FakeChatProvider.prototype, 'complete').mockRejectedValue(thrown);
try {
await completeFake({});
await completeFake(args);
} catch (e) {
return e as HttpError;
}
@@ -170,6 +175,7 @@ describe('ChatCompletionDriver provider registration', () => {
'azure-openai',
'openai-completion',
'gemini',
'meta',
'groq',
'deepseek',
'mistral',
@@ -181,6 +187,7 @@ describe('ChatCompletionDriver provider registration', () => {
'together-ai',
'openrouter',
'infron',
'byteplus',
'neuralwatt',
'fake-chat',
]) {
@@ -257,6 +264,32 @@ describe('ChatCompletionDriver exhausted-chain classification', () => {
expect(err.message).toBe('AI provider rate limit exceeded');
});
it('mutes the alarm when every rate-limited attempt was on a free model', async () => {
// `fake` is priced at zero throughout: an upstream throttle there is
// expected and costs nobody anything, so the caller still gets the
// 429 but nothing is recorded.
const err = await errorFor(
Object.assign(new Error('slow down'), { status: 429 }),
);
expect(err.noAlarm).toBe(true);
});
it('keeps the alarm when a paid model is the one being rate limited', async () => {
const err = await errorFor(
Object.assign(new Error('slow down'), { status: 429 }),
{ model: 'costly' },
);
expect(err).toMatchObject({ legacyCode: 'upstream_rate_limited' });
expect(err.noAlarm).toBe(false);
});
it('leaves non-rate-limit failures on a free model alarming as usual', async () => {
const err = await errorFor(
Object.assign(new Error('invalid api key'), { status: 401 }),
);
expect(err.noAlarm).toBeFalsy();
});
it('classifies a rate limit reported only in the message text', async () => {
const err = await errorFor(new Error('Quota exceeded for this key'));
expect(err.statusCode).toBe(429);
@@ -411,9 +444,9 @@ describe('ChatCompletionDriver cross-provider fallback', () => {
Object.assign(new Error('azure down'), { status: 503 }),
);
const openai = vi.spyOn(OpenAiChatProvider.prototype, 'complete');
vi.spyOn(server.services.metering, 'hasEnoughCredits')
.mockResolvedValueOnce(true) // pre-flight
.mockResolvedValue(false); // drained by a parallel request
vi.spyOn(server.services.metering, 'getRemainingUsage')
.mockResolvedValueOnce(1_000_000) // pre-flight
.mockResolvedValue(0); // drained by a parallel request
await expect(completeShared()).rejects.toMatchObject({
statusCode: 402,
@@ -0,0 +1,508 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* What a chat completion is charged when the usual path doesn't complete.
*
* Providers meter from the usage they hand to `chatStream.end`, which is the
* last thing a stream does — so every way a stream can stop early is a way a
* completion the upstream billed us for reaches the account as free. These
* tests pin the driver's backstop: output that was produced gets charged, and
* output that was never produced doesn't.
*/
import type { Readable } from 'node:stream';
import {
afterAll,
afterEach,
beforeAll,
beforeEach,
describe,
expect,
it,
vi,
} from 'vitest';
import type { UsageInput } from '../../services/metering/types.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { withTestActor } from '../integrationTestUtil.js';
import { ChatCompletionDriver } from './ChatCompletionDriver.js';
import { FakeChatProvider } from './providers/FakeChatProvider.js';
import type { IChatCompleteResult } from './types.js';
import type { AIChatStream } from './utils/Streaming.js';
let server: PuterServer;
const PRICED_MODEL = {
id: 'priced',
aliases: [],
costs_currency: 'usd-cents',
costs: { input_tokens: 1000, output_tokens: 2000 },
max_tokens: 8192,
};
const makeDriver = async () => {
const d = new ChatCompletionDriver(
{ providers: { ollama: { enabled: false } } } as never,
server.clients,
server.stores,
server.services,
);
d.onServerStart();
for (let i = 0; i < 200; i++) {
const m = await d.models();
if (m.length > 0) return d;
await new Promise((r) => setTimeout(r, 5));
}
throw new Error('ChatCompletionDriver model map never populated in test');
};
const drain = async (stream: Readable): Promise<void> => {
for await (const _chunk of stream as AsyncIterable<Buffer>) {
void _chunk;
}
// The driver meters in the `finally` of the fire-and-forget stream pump,
// which can land a tick after the last byte.
await new Promise((r) => setTimeout(r, 20));
};
/** Every usage entry the driver recorded, flattened across batches. */
const meteredUsages = (
spy: ReturnType<typeof vi.spyOn>,
): UsageInput[] =>
spy.mock.calls.flatMap((call) => (call[1] as UsageInput[]) ?? []);
beforeAll(async () => {
server = await setupTestServer();
});
afterAll(async () => {
await server?.shutdown();
});
let driver: ChatCompletionDriver;
beforeEach(async () => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValue([
PRICED_MODEL,
] as never);
driver = await makeDriver();
});
afterEach(() => {
vi.restoreAllMocks();
});
const streamOf = (
init: (args: { chatStream: AIChatStream }) => Promise<void>,
): IChatCompleteResult =>
({
init_chat_stream: init,
stream: true,
}) as unknown as IChatCompleteResult;
const startStream = async (driverUnderTest: ChatCompletionDriver) =>
(await withTestActor(() =>
driverUnderTest.complete({
model: 'priced',
messages: [{ role: 'user', content: 'write me something' }],
stream: true,
}),
)) as unknown as { stream: Readable };
describe('ChatCompletionDriver streaming metering backstop', () => {
it('charges an estimate when a stream dies after producing output', async () => {
const metered = vi.spyOn(server.services.metering, 'batchIncrementUsages');
vi.spyOn(FakeChatProvider.prototype, 'complete').mockResolvedValueOnce(
streamOf(async ({ chatStream }) => {
const message = chatStream.message();
const block = message.contentBlock({ type: 'text' });
block.addText('x'.repeat(4000));
throw new Error('upstream died mid-response');
}) as never,
);
const result = await startStream(driver);
await drain(result.stream);
const estimated = meteredUsages(metered).filter((u) =>
u.usageType.includes('estimated_'),
);
expect(estimated.length).toBe(2);
const output = estimated.find((u) =>
u.usageType.endsWith('estimated_output_tokens'),
)!;
// 4000 characters at 4 chars/token, priced at 2000 ucents/token.
expect(output.usageAmount).toBe(1000);
expect(output.costOverride).toBe(2_000_000);
});
it('charges an estimate when a stream completes without a usage report', async () => {
const metered = vi.spyOn(server.services.metering, 'batchIncrementUsages');
vi.spyOn(FakeChatProvider.prototype, 'complete').mockResolvedValueOnce(
streamOf(async ({ chatStream }) => {
const message = chatStream.message();
const block = message.contentBlock({ type: 'text' });
block.addText('y'.repeat(2000));
// Provider sent no usage chunk — `end` with nothing to report.
chatStream.end(undefined as never);
}) as never,
);
const result = await startStream(driver);
await drain(result.stream);
const estimated = meteredUsages(metered).filter((u) =>
u.usageType.includes('estimated_'),
);
expect(estimated.length).toBe(2);
});
it('does not charge when the stream failed before producing anything', async () => {
const metered = vi.spyOn(server.services.metering, 'batchIncrementUsages');
vi.spyOn(FakeChatProvider.prototype, 'complete').mockResolvedValueOnce(
streamOf(async () => {
throw new Error('upstream refused the request');
}) as never,
);
const result = await startStream(driver);
await drain(result.stream);
expect(
meteredUsages(metered).filter((u) =>
u.usageType.includes('estimated_'),
),
).toHaveLength(0);
});
// The provider metered, then died before `chatStream.end` — the one
// path where the backstop used to charge a second, estimated time on
// top of the real usage.
it('does not double-charge a stream whose provider metered before it threw', async () => {
const metered = vi.spyOn(server.services.metering, 'batchIncrementUsages');
vi.spyOn(FakeChatProvider.prototype, 'complete').mockResolvedValueOnce(
streamOf(async ({ chatStream }) => {
const message = chatStream.message();
const block = message.contentBlock({ type: 'text' });
block.addText('w'.repeat(4000));
// Real usage was metered here (providers report the moment
// they meter)...
chatStream.reportUsage({ input_tokens: 10, output_tokens: 1000 });
// ...then the stream died before reaching chatStream.end —
// e.g. a malformed tool-call payload failing to parse.
throw new Error('malformed tool-call payload');
}) as never,
);
const result = await startStream(driver);
await drain(result.stream);
expect(
meteredUsages(metered).filter((u) =>
u.usageType.includes('estimated_'),
),
).toHaveLength(0);
});
// `end({})` meters nothing — an empty usage object must not pass for a
// usage report, or a stream full of output goes out billed at zero.
it('charges the estimate when the usage report is empty', async () => {
const metered = vi.spyOn(server.services.metering, 'batchIncrementUsages');
vi.spyOn(FakeChatProvider.prototype, 'complete').mockResolvedValueOnce(
streamOf(async ({ chatStream }) => {
const message = chatStream.message();
const block = message.contentBlock({ type: 'text' });
block.addText('v'.repeat(2000));
chatStream.end({} as never);
}) as never,
);
const result = await startStream(driver);
await drain(result.stream);
expect(
meteredUsages(metered).filter((u) =>
u.usageType.includes('estimated_'),
),
).toHaveLength(2);
});
it('leaves a provider-reported stream alone', async () => {
const metered = vi.spyOn(server.services.metering, 'batchIncrementUsages');
vi.spyOn(FakeChatProvider.prototype, 'complete').mockResolvedValueOnce(
streamOf(async ({ chatStream }) => {
const message = chatStream.message();
const block = message.contentBlock({ type: 'text' });
block.addText('z'.repeat(2000));
chatStream.end({ input_tokens: 10, output_tokens: 500 });
}) as never,
);
const result = await startStream(driver);
await drain(result.stream);
expect(
meteredUsages(metered).filter((u) =>
u.usageType.includes('estimated_'),
),
).toHaveLength(0);
});
});
const freeUser = () => ({
user: {
uuid: `chat-gate-${Math.random().toString(36).slice(2)}`,
username: 'chat-gate-user',
email: '[email protected]',
},
});
// The gate against the real MeteringService, no mocks: a free account whose
// month has already outrun its allowance must not reach a provider again.
describe('ChatCompletionDriver credit gate against real metering', () => {
it('rejects a free account that has already spent its allowance', async () => {
const actor = freeUser() as never;
const metering = server.services.metering;
const allowance = (await metering.getActorSubscription(actor))
.monthUsageAllowance;
await metering.incrementUsage(
actor,
'test:prior-spend',
1,
allowance + 1,
);
expect(await metering.getRemainingUsage(actor)).toBe(0);
const completeSpy = vi.spyOn(FakeChatProvider.prototype, 'complete');
await expect(
withTestActor(
() =>
driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'one more' }],
}),
actor,
),
).rejects.toMatchObject({
statusCode: 402,
legacyCode: 'insufficient_funds',
});
expect(completeSpy).not.toHaveBeenCalled();
});
// The incident this exists for: usage is recorded when a completion
// finishes, so a second request that starts while the first is still
// running used to read a balance that had nothing in flight subtracted
// from it, and was told it could spend the whole thing too. Concurrency,
// not budget, was what bounded the spend.
it('does not let a second request spend a balance the first already has in flight', async () => {
const actor = freeUser() as never;
const metering = server.services.metering;
// Spent down to where one completion's worst case is the whole of
// what's left — the shape of an expensive model against a small
// allowance, which is when parallel requests overshoot.
const allowance = (await metering.getActorSubscription(actor))
.monthUsageAllowance;
await metering.incrementUsage(
actor,
'test:prior-spend',
1,
Math.floor(allowance * 0.9),
);
let releaseFirst: (v: unknown) => void = () => {};
const firstInFlight = new Promise((r) => {
releaseFirst = r;
});
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockImplementationOnce(
async () =>
firstInFlight.then(() => ({
message: {
role: 'assistant',
content: [{ type: 'text', text: 'ok' }],
},
usage: { input_tokens: 1, output_tokens: 1 },
finish_reason: 'stop',
})) as never,
);
const first = withTestActor(
() =>
driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'first request' }],
}),
actor,
);
// Let the first request clear the gate and take its hold.
await vi.waitFor(() => expect(completeSpy).toHaveBeenCalledTimes(1));
await expect(
withTestActor(
() =>
driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'second request' }],
}),
actor,
),
).rejects.toMatchObject({
statusCode: 402,
legacyCode: 'insufficient_funds',
});
// The second request never reached a provider.
expect(completeSpy).toHaveBeenCalledTimes(1);
releaseFirst(undefined);
await first;
// And the hold is given back, so the account can spend again.
expect(
await server.services.metering.getRemainingUsage(actor),
).toBeGreaterThan(0);
});
it('caps output to the balance left, so one call cannot run away with the month', async () => {
const actor = freeUser() as never;
const metering = server.services.metering;
const allowance = (await metering.getActorSubscription(actor))
.monthUsageAllowance;
// Nine tenths spent — a tenth of the allowance is left to bound the
// next completion's output.
await metering.incrementUsage(
actor,
'test:prior-spend',
1,
Math.floor(allowance * 0.9),
);
const remaining = await metering.getRemainingUsage(actor);
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockResolvedValueOnce({
message: {
role: 'assistant',
content: [{ type: 'text', text: 'ok' }],
},
usage: { input_tokens: 1, output_tokens: 1 },
finish_reason: 'stop',
} as never);
await withTestActor(
() =>
driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'go' }],
max_tokens: 100_000,
}),
actor,
);
const passed = completeSpy.mock.calls[0]![0] as { max_tokens?: number };
// 2000 ucents per output token — the cap has to fit what's left.
expect(passed.max_tokens).toBeDefined();
expect(passed.max_tokens! * 2000).toBeLessThanOrEqual(remaining);
});
});
describe('ChatCompletionDriver credit gate on multimodal prompts', () => {
it('prices attachments into the pre-flight affordability check', async () => {
const actor = freeUser() as never;
const metering = server.services.metering;
const allowance = (await metering.getActorSubscription(actor))
.monthUsageAllowance;
// Leave half of what the frame below estimates to: a plain-text
// prompt is still affordable, the same prompt carrying the frame is
// not. (~150KB of base64 payload ≈ 1000 tokens at 1000 ucents each.)
await metering.incrementUsage(
actor,
'test:prior-spend',
1,
allowance - 500_000,
);
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockResolvedValue({
message: {
role: 'assistant',
content: [{ type: 'text', text: 'ok' }],
},
usage: { input_tokens: 1, output_tokens: 1 },
finish_reason: 'stop',
} as never);
// Text alone clears the gate...
await withTestActor(
() =>
driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'describe' }],
max_tokens: 10,
}),
actor,
);
expect(completeSpy).toHaveBeenCalledTimes(1);
// ...the same prompt carrying a frame that used to price as ~nothing
// does not.
await expect(
withTestActor(
() =>
driver.complete({
model: 'priced',
messages: [
{
role: 'user',
content: [
{ type: 'text', text: 'describe' },
{
type: 'image_url',
image_url: {
url: `data:image/jpeg;base64,${'A'.repeat(200_000)}`,
},
},
],
},
],
max_tokens: 10,
}),
actor,
),
).rejects.toMatchObject({
statusCode: 402,
legacyCode: 'insufficient_funds',
});
expect(completeSpy).toHaveBeenCalledTimes(1);
});
});
@@ -128,6 +128,10 @@ const OPENROUTER_CATALOG = [
'deepseek/deepseek-v4-pro',
'deepseek-ai/deepseek-v4-pro',
'google/gemini-2.5-flash',
// OpenRouter really does carry Muse Spark under Meta's own id, which is
// also the alias the Meta provider publishes — the two have to land in
// one bucket rather than becoming separate models.
'meta/muse-spark-1.2',
].map((id) => ({
id,
name: `${id} (via OpenRouter)`,
@@ -158,6 +162,7 @@ beforeAll(async () => {
providers: {
gemini: { apiKey: 'test-key' },
deepseek: { apiKey: 'test-key' },
meta: { apiKey: 'test-key' },
infron: { apiKey: 'test-key' },
openrouter: { apiKey: 'test-key' },
ollama: { enabled: false },
@@ -250,6 +255,26 @@ describe('ChatCompletionDriver gemini routing', () => {
});
});
describe('ChatCompletionDriver muse spark routing', () => {
it('serves Muse Spark from Meta, with OpenRouter only as fallback', async () => {
const attempts = await attemptsFor('muse-spark-1.2');
expect(attempts[0]).toMatchObject({
provider: 'meta',
model: 'muse-spark-1.2',
});
expect(attempts[1]).toMatchObject({
provider: 'openrouter',
model: 'openrouter:meta/muse-spark-1.2',
});
});
it('routes the vendor-qualified alias to Meta too', async () => {
const attempts = await attemptsFor('meta/muse-spark-1.2');
expect(attempts[0].provider).toBe('meta');
});
});
describe('ChatCompletionDriver duplicate-model fallback', () => {
// deepseek-v4-pro is served directly by DeepSeek, by Infron, and twice by
// OpenRouter (two upstream orgs) — four routes in one bucket.
@@ -293,6 +318,73 @@ describe('ChatCompletionDriver duplicate-model fallback', () => {
});
});
// Each attempt in a fallback chain is a whole completion at that model's
// prices, so each one has to clear the credit gate on its own. The chain used
// to re-check with a nominal 1-microcent amount, which any account with a
// fraction of a credit left passed — three attempts could cost three times
// what the balance allowed.
describe('ChatCompletionDriver credit gate across the fallback chain', () => {
it('runs the full gate once per attempt, not a nominal re-check', async () => {
// Observed, not stubbed — the counter that shows each attempt did a
// real balance read of its own.
const remaining = vi.spyOn(
server.services.metering,
'getRemainingUsage',
);
const attempts = await attemptsFor('deepseek-v4-pro');
expect(attempts).toHaveLength(3);
// The balance is read for each attempt because each one's
// affordability and output cap are decided at that model's prices
// against what's actually left.
expect(remaining.mock.calls.length).toBeGreaterThanOrEqual(
attempts.length,
);
remaining.mockRestore();
});
it('aborts the chain when the balance runs out mid-fallback', async () => {
const actor = {
user: {
uuid: `routing-gate-${Math.random().toString(36).slice(2)}`,
username: 'routing-gate-user',
email: '[email protected]',
},
} as never;
const metering = server.services.metering;
// The first attempt fails upstream, and while it does, a "parallel
// request" spends the rest of the month's allowance — real usage
// rows, not a stubbed balance — so the next attempt's gate reads an
// account with nothing left.
createMock.mockImplementation(async () => {
const { remaining } = await metering.getAllowedUsage(actor);
await metering.incrementUsage(
actor,
'test:parallel-spend',
1,
remaining,
);
throw new Error('upstream down');
});
await expect(
withTestActor(
() =>
driver.complete({
model: 'deepseek-v4-pro',
messages: [{ role: 'user', content: 'hi' }],
}),
actor,
),
).rejects.toMatchObject({
statusCode: 402,
legacyCode: 'insufficient_funds',
});
});
});
describe('ChatCompletionDriver unhealthy-route skipping', () => {
it('skips a route marked by an earlier failure and serves the next one', async () => {
markRouteUnhealthy('deepseek', 'deepseek-v4-pro');
@@ -481,8 +481,8 @@ describe('ChatCompletionDriver.complete validation event routing', () => {
describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
it('throws 402 `insufficient_funds` when the actor has no remaining credits', async () => {
vi.spyOn(server.services.metering, 'hasEnoughCredits').mockResolvedValue(
false,
vi.spyOn(server.services.metering, 'getRemainingUsage').mockResolvedValue(
0,
);
await expect(
@@ -562,11 +562,8 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
]);
const d = await makeDriver();
// Pass the cheap pre-flight gate but leave a balance too small to
// Pass the cheap pre-flight check but leave a balance too small to
// afford a single 2000-microcent output token.
vi.spyOn(server.services.metering, 'hasEnoughCredits').mockResolvedValue(
true,
);
vi.spyOn(server.services.metering, 'getRemainingUsage').mockResolvedValue(
100,
);
@@ -610,10 +607,6 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
] as never);
const d = await makeDriver();
vi.spyOn(
server.services.metering,
'hasEnoughCredits',
).mockResolvedValue(true);
vi.spyOn(
server.services.metering,
'getRemainingUsage',
@@ -658,9 +651,6 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
]);
const d = await makeDriver();
// Plenty of credits so the credit gate doesn't intercept first.
vi.spyOn(server.services.metering, 'hasEnoughCredits').mockResolvedValue(
true,
);
vi.spyOn(server.services.metering, 'getRemainingUsage').mockResolvedValue(
1_000_000,
);
@@ -768,19 +758,20 @@ describe('ChatCompletionDriver.complete fallback and error envelope', () => {
});
});
it('re-checks `hasEnoughCredits` between fallback attempts so a parallel request that drains the wallet aborts the chain', async () => {
// The primary provider throws; the fallback loop checks credits
// before its next upstream hit. We force `false` on the second
// check to verify the 402 short-circuit, even though no actual
// fallback model is wired (the loop bails on the credit gate
// before `#findFallback` decides there's nowhere to go).
it('re-reads the balance between fallback attempts so a parallel request that drains the wallet aborts the chain', async () => {
// The primary provider throws; the fallback loop runs the full gate
// (one balance read per attempt) before its next upstream hit. We
// force an empty balance on the second read to verify the 402
// short-circuit, even though no actual fallback model is wired (the
// loop bails on the credit gate before `#findFallback` decides
// there's nowhere to go).
vi.spyOn(FakeChatProvider.prototype, 'complete').mockRejectedValueOnce(
new Error('boom'),
);
const credits = vi
.spyOn(server.services.metering, 'hasEnoughCredits')
.mockResolvedValueOnce(true) // pre-flight
.mockResolvedValueOnce(false); // mid-fallback re-check
const remaining = vi
.spyOn(server.services.metering, 'getRemainingUsage')
.mockResolvedValueOnce(1_000_000) // pre-flight
.mockResolvedValueOnce(0); // drained mid-fallback
// No second provider serves `fake`, so `#findFallback` returns
// null and the loop exits before reaching the credit re-check.
@@ -795,7 +786,7 @@ describe('ChatCompletionDriver.complete fallback and error envelope', () => {
}),
),
).rejects.toMatchObject({ statusCode: 500 });
expect(credits.mock.calls.length).toBeGreaterThanOrEqual(1);
expect(remaining.mock.calls.length).toBeGreaterThanOrEqual(1);
});
});
@@ -20,24 +20,29 @@
import crypto from 'node:crypto';
import { PassThrough } from 'node:stream';
import { EventMap } from '../../clients/event/types.js';
import type { Actor } from '../../core/actor.js';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { HttpError, isHttpError } from '../../core/http/HttpError.js';
import {
DEFAULT_FREE_SUBSCRIPTION,
DEFAULT_TEMP_SUBSCRIPTION,
} from '../../services/metering/consts.js';
import type { CreditHold } from '../../services/metering/types.js';
import { NO_CREDIT_HOLD } from '../../services/metering/types.js';
import type { DriverStreamResult } from '../meta.js';
import { PuterDriver } from '../types.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../util/aiLimits.js';
import { AlibabaProvider } from './providers/alibaba/AlibabaProvider.js';
import { AzureChatProvider } from './providers/azure/AzureChatProvider.js';
import { AzureResponsesProvider } from './providers/azure/AzureResponsesProvider.js';
import { BytePlusProvider } from './providers/byteplus/BytePlusProvider.js';
import { ClaudeProvider } from './providers/claude/ClaudeProvider.js';
import { DeepSeekProvider } from './providers/deepseek/DeepSeekProvider.js';
import { FakeChatProvider } from './providers/FakeChatProvider.js';
import { GeminiChatProvider } from './providers/gemini/GeminiChatProvider.js';
import { GroqAIProvider } from './providers/groq/GroqAIProvider.js';
import { InfronProvider } from './providers/infron/InfronProvider.js';
import { MetaProvider } from './providers/meta/MetaProvider.js';
import { MiniMaxProvider } from './providers/minimax/MiniMaxProvider.js';
import { MistralAIProvider } from './providers/mistral/MistralAiProvider.js';
import { MoonshotProvider } from './providers/moonshot/MoonshotProvider.js';
@@ -57,7 +62,6 @@ import type {
} from './types.js';
import { normalize_tools_object } from './utils/FunctionCalling.js';
import {
extract_text,
normalize_messages,
normalize_single_message,
} from './utils/Messages.js';
@@ -66,14 +70,35 @@ import {
isIdentityKey,
normalizeModelKey,
} from './utils/modelRouting.js';
import { costKeys, isFreeModel } from './utils/pricing.js';
import {
isRouteUnhealthy,
markRouteUnhealthy,
} from './utils/providerHealth.js';
import { AIChatStream } from './utils/Streaming.js';
import {
estimateOutputTokens,
estimatePromptTokens,
} from './utils/usageEstimate.js';
const MAX_ATTEMPTS = 3; // the first attempt plus two fallbacks
/**
* How often a streaming completion renews its credit hold. Holds default to a
* 10-minute TTL; a long generation (a reasoning model with a large
* `max_tokens`) can stream past that, and a hold that expires mid-stream
* reopens the overspend window it exists to close.
*/
const HOLD_RENEW_INTERVAL_MS = 5 * 60 * 1000;
/**
* A moderation refusal is a completion that was produced and charged, then
* withheld — not a route failure. Retrying it on a fallback provider would bill
* the account again for another completion the user will never see.
*/
const isModerationRefusal = (e: unknown): boolean =>
isHttpError(e) && e.code === 'moderation_flagged';
type ProviderAttempt = {
model: string;
provider: string;
@@ -157,13 +182,17 @@ const routeId = (provider: string, modelId: string) => `${provider}:${modelId}`;
*
* Per-class rules (see also alarm gate in server.ts):
*
* - All rate-limited → 429 `upstream_rate_limited` (paged: forced alert)
* - All rate-limited → 429 `upstream_rate_limited` (alerted, unless every attempt
* was on a free model — see `allModelsFree`)
* - All auth failures → 500 `upstream_auth_failed` (paged: our config)
* - All upstream 5xx → 400 `upstream_provider_unavailable` (no page)
* - All upstream 4xx (other) → 400 `upstream_bad_request` (no page)
* - Mixed → 400 `upstream_failed` (no page)
*/
const classifyAttempts = (attempts: ProviderAttempt[]): HttpError => {
const classifyAttempts = (
attempts: ProviderAttempt[],
{ allModelsFree = false } = {},
): HttpError => {
const fields = { attempts };
if (attempts.length === 0) {
return new HttpError(500, 'No providers attempted', {
@@ -176,6 +205,11 @@ const classifyAttempts = (attempts: ProviderAttempt[]): HttpError => {
return new HttpError(429, 'AI provider rate limit exceeded', {
legacyCode: 'upstream_rate_limited',
fields,
// A free model getting throttled upstream is the deal we took
// when we picked it up for nothing: there's no billing at stake
// and nothing to act on, and the volume tracks traffic. The
// caller still gets the 429; we just don't record it.
noAlarm: allModelsFree,
});
}
if (attempts.every(isAuthFailure)) {
@@ -334,6 +368,15 @@ export class ChatCompletionDriver extends PuterDriver {
normalize_tools_object(args.tools);
}
// Both estimated once, before any attempt: providers rewrite
// `args.messages` in place (tool_use blocks move out of `content`), so
// an estimate taken after a failed attempt would undercount the same
// prompt — and the gate writes each attempt's output cap into
// `args.max_tokens`, so the user's requested value has to be kept
// apart from what the previous attempt was capped to.
const promptTokenEstimate = estimatePromptTokens(args.messages ?? []);
const requestedMaxTokens = args.max_tokens;
const completionId = crypto
.randomUUID()
.replaceAll('-', '')
@@ -376,89 +419,20 @@ export class ChatCompletionDriver extends PuterDriver {
}
}
// -- Credit / subscription gates (metering) --------------------
// Cheap pre-flight: reject when the user can't afford even the
// approximate input cost, keep subscriber-only models gated, and
// cap `max_tokens` so output can't exceed remaining credits.
// Skipped for blocked requests since fake-chat is free and the
// user shouldn't see a billing error in place of the abuse page.
// Skipped for blocked requests since fake-chat is free and the user
// shouldn't see a billing error in place of the abuse page.
//
// The gate hands back a hold on what this attempt could cost, which
// stands in for its usage until the real numbers land. It is released
// on every way out of this method — including the streaming path,
// where "done" is the stream draining rather than this method
// returning.
let hold: CreditHold = NO_CREDIT_HOLD;
if (!blocked) {
const metering = this.services.metering;
const inputCostKey =
(model.input_cost_key as string | undefined) ?? 'input_tokens';
const outputCostKey =
(model.output_cost_key as string | undefined) ??
'output_tokens';
const inputTokenCost = Number(model.costs?.[inputCostKey] ?? 0);
const outputTokenCost = Number(model.costs?.[outputCostKey] ?? 0);
const text = extract_text(args.messages ?? []);
// Rough estimator from v1 — avg of char/4 and word*(4/3), halved.
// See https://help.openai.com/en/articles/4936856
const approximateTokenCount = Math.floor(
(text.length / 4 + text.split(/\s+/).length * (4 / 3)) / 2,
);
const approximateInputCost = approximateTokenCount * inputTokenCost;
const minimumCredits = Number(model.minimumCredits || 1);
const usageAllowed = await metering.hasEnoughCredits(
actor,
Math.max(approximateInputCost, minimumCredits),
);
if (!usageAllowed) {
throw new HttpError(402, 'No usage left for request.', {
legacyCode: 'insufficient_funds',
});
}
if (model.subscriberOnly) {
const subscription = await metering.getActorSubscription(actor);
const isDefaultPolicy =
subscription.id === DEFAULT_FREE_SUBSCRIPTION ||
subscription.id === DEFAULT_TEMP_SUBSCRIPTION;
if (isDefaultPolicy) {
throw new HttpError(
403,
`The model ${model.id} is only available to subscribers. Please subscribe to access this model.`,
{ legacyCode: 'permission_denied' },
);
}
}
if (outputTokenCost > 0) {
const remainingCredits =
await metering.getRemainingUsage(actor);
const maxAllowedOutputUcents =
remainingCredits - approximateInputCost;
const maxAllowedOutputTokens =
maxAllowedOutputUcents / outputTokenCost;
// A provider may not know a model's output ceiling. Drop the
// term rather than let a missing value drive the cap: `null`
// coerces to 0, so the subtraction goes negative instead of
// NaN and the user is told they're out of credits.
const modelOutputCeiling =
Number.isFinite(model.max_tokens) && model.max_tokens > 0
? model.max_tokens - approximateTokenCount
: Number.POSITIVE_INFINITY;
const cap = Math.floor(
Math.min(
args.max_tokens ?? Number.POSITIVE_INFINITY,
maxAllowedOutputTokens,
modelOutputCeiling,
),
);
// `cap` is the credit-bounded ceiling on output tokens. When
// it drops below 1 the user can't afford even a single output
// token, so reject the request. Crucially we must NOT leave
// `max_tokens` unset here: an undefined max_tokens lets the
// provider run to the model's full output limit (e.g. 128k for
// Claude), billing far past the user's remaining balance.
if (cap < 1) {
throw new HttpError(402, 'No usage left for request.', {
legacyCode: 'insufficient_funds',
});
}
args.max_tokens = cap;
}
hold = await this.#applyCreditGate(actor, model, args, {
promptTokenEstimate,
requestedMaxTokens,
});
}
// First attempt
@@ -473,18 +447,18 @@ export class ChatCompletionDriver extends PuterDriver {
const attempts: ProviderAttempt[] = [];
let res: IChatCompleteResult | undefined;
// Tracked across the chain so the classifier can tell a chain that
// only ever touched free models from one that cost the user something.
let allModelsFree = true;
// A failed route is remembered briefly so the next request skips it
// rather than paying its timeout again.
const recordFailure = (
modelId: string,
providerId: string,
err: unknown,
) => {
const attempt = toAttempt(modelId, providerId, err);
const recordFailure = (failed: IChatModel, err: unknown) => {
const attempt = toAttempt(failed.id, failed.provider!, err);
attempts.push(attempt);
if (!isFreeModel(failed)) allModelsFree = false;
if (isRouteLevelFailure(attempt)) {
markRouteUnhealthy(providerId, modelId);
markRouteUnhealthy(failed.provider!, failed.id);
}
};
@@ -495,7 +469,15 @@ export class ChatCompletionDriver extends PuterDriver {
provider: model.provider,
});
} catch (e) {
recordFailure(model.id, model.provider!, e);
// This attempt is over and cost whatever it cost; the next one
// takes a hold of its own.
await hold.release();
hold = NO_CREDIT_HOLD;
// A withheld completion was still a completion — charged, final,
// not a route failure worth another (billed) attempt elsewhere.
if (isModerationRefusal(e)) throw e;
recordFailure(model, e);
// Fallback loop — the bucket holds every provider that serves this
// model, ranked by `compareModelPreference`, so each miss walks one
@@ -511,14 +493,18 @@ export class ChatCompletionDriver extends PuterDriver {
const fbProvider = this.#providers[fallback.provider!];
if (!fbProvider) break;
// Credits can be exhausted mid-fallback by parallel requests;
// re-check before another upstream hit. Same bail as the
// pre-flight above.
const fallbackUsageAllowed =
await this.services.metering.hasEnoughCredits(actor, 1);
if (!fallbackUsageAllowed) {
throw new HttpError(402, 'No usage left for request.', {
legacyCode: 'insufficient_funds',
// Every attempt is a whole completion the account pays for, so
// each one goes through the full gate again rather than a
// token "do they have anything left" check: the balance may
// have been spent by a parallel request, and the fallback is
// a different model at a different price, whose output has to
// be capped against what is actually left.
// The previous attempt released its hold when it failed, so
// this one starts from nothing held.
if (!blocked) {
hold = await this.#applyCreditGate(actor, fallback, args, {
promptTokenEstimate,
requestedMaxTokens,
});
}
@@ -533,14 +519,18 @@ export class ChatCompletionDriver extends PuterDriver {
model = fallback;
lastError = null;
} catch (fbErr) {
await hold.release();
hold = NO_CREDIT_HOLD;
if (isModerationRefusal(fbErr)) throw fbErr;
lastError = fbErr as Error;
recordFailure(fallback.id, fallback.provider!, fbErr);
recordFailure(fallback, fbErr);
}
}
}
if (!res) {
throw classifyAttempts(attempts);
await hold.release();
throw classifyAttempts(attempts, { allModelsFree });
}
const username = actor.user?.username;
@@ -588,6 +578,13 @@ export class ChatCompletionDriver extends PuterDriver {
return originalEnd(enrichedUsage!);
};
// The hold lives for the whole stream, which can outlast its TTL —
// keep pushing the deadline out until the pump is done.
const renewHold = setInterval(() => {
void hold.extend?.();
}, HOLD_RENEW_INTERVAL_MS);
renewHold.unref?.();
// Fire-and-forget — the stream writes happen async while the
// response is being piped to the client.
(async () => {
@@ -602,6 +599,26 @@ export class ChatCompletionDriver extends PuterDriver {
);
passthrough.end();
} finally {
clearInterval(renewHold);
// Providers report usage the moment they meter it (see
// `AIChatStream.reportUsage`); a stream that never got
// there was never charged for.
if (!blocked && !chatStream.reportedUsage) {
this.#meterUnreportedStream({
actor,
chatStream,
model,
promptTokenEstimate,
completionId,
username,
intendedProvider,
});
}
// Held until the generation is actually over: for a
// stream, the provider returns as soon as it has a
// populator, and everything the account pays for happens
// after that.
await hold.release();
if (cleanup) await cleanup();
}
})();
@@ -615,6 +632,10 @@ export class ChatCompletionDriver extends PuterDriver {
return streamResult as unknown as IChatCompleteResult;
}
// The provider recorded this completion's usage before returning it,
// so the hold has served its purpose.
await hold.release();
// -- Post-completion audit event ------------------------------
// Only for non-streaming results (streaming emits from the
// `chatStream.end` wrapper above). Extensions like prompt_block /
@@ -676,10 +697,7 @@ export class ChatCompletionDriver extends PuterDriver {
outputMicroCents: number;
totalMicroCents: number;
} | null {
const inputKey =
(model.input_cost_key as string | undefined) ?? 'input_tokens';
const outputKey =
(model.output_cost_key as string | undefined) ?? 'output_tokens';
const { inputKey, outputKey } = costKeys(model);
const costs = model.costs;
if (!costs) return null;
@@ -763,6 +781,199 @@ export class ChatCompletionDriver extends PuterDriver {
};
}
/**
* The credit and subscription gate for one upstream attempt.
*
* Runs before every attempt, not once per request: each attempt is a whole
* completion the account pays for, at that model's prices, against whatever
* balance is left by the time it starts.
*
* Rejects when the account can't afford the approximate input cost, keeps
* subscriber-only models gated, and tightens `args.max_tokens` so the
* output this attempt can produce is bounded by the remaining balance.
*
* Returns a hold on what the attempt can cost at worst, so requests this
* account is running in parallel see the spend before it is recorded. The
* caller releases it once the attempt is done.
*/
async #applyCreditGate(
actor: Actor,
model: IChatModel,
args: ICompleteArguments,
estimates: {
/**
* Prompt tokens, estimated once before any attempt — counts
* attachments as well as text, and predates any in-place message
* rewriting a previous attempt's provider did.
*/
promptTokenEstimate: number;
/**
* What the user asked for, kept apart from `args.max_tokens`, which
* carries the previous attempt's cap: a cheap fallback must not
* inherit the ceiling computed at an expensive model's price.
*/
requestedMaxTokens: number | undefined;
},
): Promise<CreditHold> {
const metering = this.services.metering;
const { promptTokenEstimate, requestedMaxTokens } = estimates;
const { inputKey, outputKey } = costKeys(model);
// `|| 0` also catches NaN from a malformed cost table.
const inputTokenCost = Number(model.costs?.[inputKey] ?? 0) || 0;
const outputTokenCost = Number(model.costs?.[outputKey] ?? 0) || 0;
const approximateInputCost = promptTokenEstimate * inputTokenCost;
const minimumCredits = Number(model.minimumCredits || 1);
// One balance read serves the whole gate: the affordability check
// here and the output cap below.
const remainingCredits = await metering.getRemainingUsage(actor);
if (remainingCredits < Math.max(approximateInputCost, minimumCredits)) {
throw new HttpError(402, 'No usage left for request.', {
legacyCode: 'insufficient_funds',
});
}
if (model.subscriberOnly) {
const subscription = await metering.getActorSubscription(actor);
const isDefaultPolicy =
subscription.id === DEFAULT_FREE_SUBSCRIPTION ||
subscription.id === DEFAULT_TEMP_SUBSCRIPTION;
if (isDefaultPolicy) {
throw new HttpError(
403,
`The model ${model.id} is only available to subscribers. Please subscribe to access this model.`,
{ legacyCode: 'permission_denied' },
);
}
}
if (outputTokenCost > 0) {
const maxAllowedOutputUcents =
remainingCredits - approximateInputCost;
const maxAllowedOutputTokens =
maxAllowedOutputUcents / outputTokenCost;
// A provider may not know a model's output ceiling. Drop the term
// rather than let a missing value drive the cap: `null` coerces to
// 0, so the subtraction goes negative instead of NaN and the user
// is told they're out of credits.
const modelOutputCeiling =
Number.isFinite(model.max_tokens) && model.max_tokens > 0
? model.max_tokens - promptTokenEstimate
: Number.POSITIVE_INFINITY;
const cap = Math.floor(
Math.min(
requestedMaxTokens ?? Number.POSITIVE_INFINITY,
maxAllowedOutputTokens,
modelOutputCeiling,
),
);
// `cap` is the credit-bounded ceiling on output tokens. When it
// drops below 1 the user can't afford even a single output token,
// so reject the request. Crucially we must NOT leave `max_tokens`
// unset here: an undefined max_tokens lets the provider run to the
// model's full output limit (e.g. 128k for Claude), billing far
// past the user's remaining balance.
if (cap < 1) {
throw new HttpError(402, 'No usage left for request.', {
legacyCode: 'insufficient_funds',
});
}
args.max_tokens = cap;
} else {
// No output price, nothing to bound — but a previous attempt may
// have written its cap here; give this one the user's own value.
args.max_tokens = requestedMaxTokens;
}
// What this attempt can cost at worst: the prompt, plus output run to
// the cap just set. Capped output is what makes the number finite —
// for a model with no output price the output term is zero and the
// prompt estimate stands alone.
const worstCaseCost =
approximateInputCost + (args.max_tokens ?? 0) * outputTokenCost;
return this.services.metering.reserveCredits(
actor,
Math.max(worstCaseCost, minimumCredits),
);
}
/**
* Charge a stream that produced output but never reported usage.
*
* Providers meter from the usage they hand to `chatStream.end`, at the very
* end of the stream — so anything that stops the stream short of that point
* (an upstream error mid-response, a malformed tool-call payload, a
* provider that never sends a usage chunk) leaves a completion the upstream
* has already billed us for and the account has paid nothing for. This is
* the backstop: what the stream actually emitted, priced off the model's
* own cost table.
*
* Only when there was output. A stream that failed before producing
* anything cost the user nothing, and charging an estimated prompt to
* someone whose request we failed to serve is worse than the leak.
*
* Recorded under `estimated_*` usage keys so the numbers stay separable
* from provider-reported ones in the usage breakdown.
*/
#meterUnreportedStream(params: {
actor: Actor;
chatStream: AIChatStream;
model: IChatModel;
/** Estimated before any provider rewrote `args.messages` in place. */
promptTokenEstimate: number;
completionId: string;
username?: string;
intendedProvider: string;
}): void {
const {
actor,
chatStream,
model,
promptTokenEstimate,
completionId,
username,
intendedProvider,
} = params;
const outputTokens = estimateOutputTokens(chatStream.outputChars ?? 0);
if (outputTokens <= 0) return;
const { inputKey, outputKey } = costKeys(model);
const inputTokens = promptTokenEstimate;
const usage = {
[inputKey]: inputTokens,
[outputKey]: outputTokens,
};
const cost = this.#computeCost(usage, model);
this.services.metering.utilRecordUsageObject(
{
[`estimated_${inputKey}`]: inputTokens,
[`estimated_${outputKey}`]: outputTokens,
},
actor,
`${model.provider}:${model.id}`,
{
// Undefined when the model has no cost table: the entry is
// recorded unpriced rather than free.
[`estimated_${inputKey}`]: cost?.inputMicroCents,
[`estimated_${outputKey}`]: cost?.outputMicroCents,
},
);
console.warn(
`[ai-chat] stream ended without usage; charged an estimate (${completionId}, ${model.provider}:${model.id}, ~${inputTokens} in / ~${outputTokens} out)`,
);
this.#emitCostCalculated({
completionId,
username,
usage,
model,
intendedProvider,
});
}
// Add `usd_cents` to the usage object. Skips if the provider already
// set an authoritative value (e.g. OpenRouter's `usage.cost`).
// Sets `null` when cost data is unavailable for the model.
@@ -795,10 +1006,7 @@ export class ChatCompletionDriver extends PuterDriver {
params;
const cost = this.#computeCost(usage, model);
const inputKey =
(model.input_cost_key as string | undefined) ?? 'input_tokens';
const outputKey =
(model.output_cost_key as string | undefined) ?? 'output_tokens';
const { inputKey, outputKey } = costKeys(model);
const inputTokens = cost?.inputTokens ?? 0;
const outputTokens = cost?.outputTokens ?? 0;
const inputMicroCents = cost?.inputMicroCents ?? 0;
@@ -926,6 +1134,23 @@ export class ChatCompletionDriver extends PuterDriver {
});
}
const meta = providers['meta'];
const metaKey = readKey(meta);
if (metaKey) {
this.#providers['meta'] = new MetaProvider(
metering,
{
fsEntry: this.stores.fsEntry,
s3Object: this.stores.s3Object,
},
this.services.fs,
{
apiKey: metaKey,
apiBaseUrl: meta?.apiBaseUrl as string | undefined,
},
);
}
const groqKey = readKey(providers['groq']);
if (groqKey) {
this.#providers['groq'] = new GroqAIProvider(
@@ -1002,14 +1227,6 @@ export class ChatCompletionDriver extends PuterDriver {
);
}
const togetherKey = readKey(providers['together-ai']);
if (togetherKey) {
this.#providers['together-ai'] = new TogetherAIProvider(
{ apiKey: togetherKey },
metering,
);
}
// Ollama — auto-discover local instance unless `enabled: false`.
const ollama = providers['ollama'];
if (ollama?.enabled !== false) {
@@ -1021,6 +1238,18 @@ export class ChatCompletionDriver extends PuterDriver {
);
}
const infron = providers['infron'];
const infronKey = readKey(infron);
if (infronKey) {
this.#providers['infron'] = new InfronProvider(
{
apiKey: infronKey,
apiBaseUrl: infron?.apiBaseUrl as string | undefined,
},
metering,
);
}
const openrouter = providers['openrouter'];
const openrouterKey = readKey(openrouter);
if (openrouterKey) {
@@ -1033,13 +1262,21 @@ export class ChatCompletionDriver extends PuterDriver {
);
}
const infron = providers['infron'];
const infronKey = readKey(infron);
if (infronKey) {
this.#providers['infron'] = new InfronProvider(
const togetherKey = readKey(providers['together-ai']);
if (togetherKey) {
this.#providers['together-ai'] = new TogetherAIProvider(
{ apiKey: togetherKey },
metering,
);
}
const byteplus = providers['byteplus'];
const byteplusKey = readKey(byteplus);
if (byteplusKey) {
this.#providers['byteplus'] = new BytePlusProvider(
{
apiKey: infronKey,
apiBaseUrl: infron?.apiBaseUrl as string | undefined,
apiKey: byteplusKey,
apiBaseUrl: byteplus?.apiBaseUrl as string | undefined,
},
metering,
);
@@ -157,6 +157,93 @@ export const ALIBABA_MODELS: IChatModel[] = [
},
},
{
puterId: 'alibaba:qwen/qwen3.6-flash',
id: 'qwen3.6-flash',
modalities: { input: ['text', 'image', 'video'], output: ['text'] },
open_weights: false,
tool_call: true,
release_date: '2026-04-27',
name: 'Qwen3.6 Flash',
aliases: ['qwen/qwen3.6-flash'],
context: 1_000_000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 19,
completion_tokens: 113,
},
max_tokens: 65_536,
},
{
puterId: 'alibaba:qwen/qwen3.7-plus',
id: 'qwen3.7-plus',
modalities: { input: ['text', 'image', 'video'], output: ['text'] },
open_weights: false,
tool_call: true,
release_date: '2026-06-02',
name: 'Qwen3.7 Plus',
aliases: ['qwen/qwen3.7-plus'],
context: 1_000_000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 50,
completion_tokens: 300,
cached_tokens: 5,
},
max_tokens: 65_536,
},
{
puterId: 'alibaba:qwen/qwen3.7-max',
id: 'qwen3.7-max',
modalities: { input: ['text'], output: ['text'] },
open_weights: false,
tool_call: true,
release_date: '2026-05-21',
name: 'Qwen3.7 Max',
aliases: ['qwen/qwen3.7-max'],
context: 1_000_000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 250,
completion_tokens: 750,
cached_tokens: 50,
},
max_tokens: 65_536,
},
{
puterId: 'alibaba:qwen/qwen3.8-max',
id: 'qwen3.8-max',
modalities: {
input: ['text', 'image', 'video', 'pdf'],
output: ['text'],
},
open_weights: false,
tool_call: true,
release_date: '2026-08-03',
name: 'Qwen3.8 Max',
aliases: ['qwen/qwen3.8-max'],
context: 1_000_000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 200,
completion_tokens: 600,
cached_tokens: 25,
},
max_tokens: 131_072,
},
// -- Turbo / Flash tier -----------------------------------------
{
puterId: 'alibaba:qwen/qwen-turbo',
@@ -270,29 +357,6 @@ export const ALIBABA_MODELS: IChatModel[] = [
cached_tokens: 0,
},
},
{
puterId: 'alibaba:qwen/qwen3-coder-30b-a3b-instruct',
id: 'qwen3-coder-30b-a3b-instruct',
name: 'Qwen3-Coder 30B-A3B Instruct',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: true,
knowledge: '2025-04',
release_date: '2025-04',
aliases: ['qwen/qwen3-coder-30b-a3b-instruct'],
context: 262_144,
max_tokens: 65_536,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 45,
completion_tokens: 225,
cached_tokens: 0,
},
},
// -- Reasoning models -------------------------------------------
{
puterId: 'alibaba:qwen/qwq-plus',
@@ -666,28 +730,6 @@ export const ALIBABA_MODELS: IChatModel[] = [
cached_tokens: 0,
},
},
{
puterId: 'alibaba:qwen/qwen3-vl-30b-a3b',
id: 'qwen3-vl-30b-a3b',
name: 'Qwen3-VL 30B-A3B',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: true,
tool_call: true,
knowledge: '2025-04',
release_date: '2025-04',
aliases: ['qwen/qwen3-vl-30b-a3b'],
context: 131_072,
max_tokens: 32_768,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 20,
completion_tokens: 80,
cached_tokens: 0,
},
},
{
puterId: 'alibaba:qwen/qvq-max',
id: 'qvq-max',
@@ -186,7 +186,7 @@ describe('AzureChatProvider model catalog', () => {
expect(ids).toContain('openai/gpt-4o');
expect(ids).toContain('x-ai/grok-4-20-non-reasoning');
// Codex is Responses-only and must not be advertised here.
expect(ids).not.toContain('gpt-5-codex');
expect(ids).not.toContain('gpt-5.3-codex');
});
});
@@ -81,7 +81,7 @@ export class AzureChatProvider implements IChatProvider {
baseURL: config.apiURL,
});
}
checkModeration(_text: string): { flagged: boolean; categories: string[] } {
checkModeration(_text: string) {
throw new Error('Method not implemented.');
}
@@ -180,7 +180,7 @@ export class AzureChatProvider implements IChatProvider {
// })
const userIdentifier =
actor.user?.id + actor.app?.uid ? `:${actor?.app?.uid}` : '';
`${actor.user?.id}` + actor.app?.uid ? `:${actor?.app?.uid}` : '';
// Resolve any `puter_path` content parts into inline base64 data URLs.
// Chat Completions doesn't support file uploads, so this is the only
@@ -20,7 +20,7 @@
/**
* Integration test for the Azure AI Foundry Responses provider.
*
* Hits the real Azure endpoint with `gpt-5-codex` — a Responses-API-only
* Hits the real Azure endpoint with `gpt-5.3-codex` — a Responses-API-only
* (Codex) model that the Chat Completions endpoint rejects. This verifies
* the completions/responses split actually routes Codex correctly. Codex is
* a reasoning model, so we give it a generous `max_tokens` and low reasoning
@@ -47,7 +47,7 @@ describe.skipIf(skipUnlessEnv(KEY_ENV) || skipUnlessEnv(URL_ENV))(
'AzureResponsesProvider (integration)',
() => {
it(
'returns a non-empty completion from gpt-5-codex',
'returns a non-empty completion from gpt-5.3-codex',
{ timeout: INTEGRATION_TEST_TIMEOUT_MS },
async () => {
const provider = new AzureResponsesProvider(
@@ -65,7 +65,7 @@ describe.skipIf(skipUnlessEnv(KEY_ENV) || skipUnlessEnv(URL_ENV))(
const result = await withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
model: 'gpt-5.3-codex',
messages: [
{ role: 'user', content: 'Say hi in one word.' },
],
@@ -156,15 +156,15 @@ describe('AzureResponsesProvider construction', () => {
// -- Model catalog ---------------------------------------------------
describe('AzureResponsesProvider model catalog', () => {
it('returns gpt-5-codex as the default model', () => {
expect(makeProvider().getDefaultModel()).toBe('gpt-5-codex');
it('returns gpt-5.3-codex as the default model', () => {
expect(makeProvider().getDefaultModel()).toBe('gpt-5.3-codex');
});
it('models() exposes only the responses_api_only slice of the Azure catalog', () => {
const ids = makeProvider()
.models()
.map((m) => m.id);
expect(ids).toContain('gpt-5-codex');
expect(ids).toContain('gpt-5.3-codex');
// Chat-Completions models belong to the sibling provider.
expect(ids).not.toContain('gpt-4o');
});
@@ -173,15 +173,15 @@ describe('AzureResponsesProvider model catalog', () => {
const ids = makeProvider()
.models({ no_restrictions: true })
.map((m) => m.id);
expect(ids).toContain('gpt-5-codex');
expect(ids).toContain('gpt-5.3-codex');
expect(ids).toContain('gpt-4o');
expect(ids).toHaveLength(AZURE_MODELS.length);
});
it('list() flattens responses-only ids and their aliases', () => {
const ids = makeProvider().list();
expect(ids).toContain('gpt-5-codex');
expect(ids).toContain('openai/gpt-5-codex');
expect(ids).toContain('gpt-5.3-codex');
expect(ids).toContain('openai/gpt-5.3-codex');
expect(ids).not.toContain('gpt-4o');
});
});
@@ -194,7 +194,7 @@ describe('AzureResponsesProvider.complete argument validation', () => {
await expect(
withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
model: 'gpt-5.3-codex',
messages: 'hello' as unknown as never,
}),
),
@@ -212,7 +212,7 @@ describe('AzureResponsesProvider.complete request shape', () => {
await withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
model: 'gpt-5.3-codex',
messages: [{ role: 'user', content: 'hello' }],
max_tokens: 256,
temperature: 0.3,
@@ -220,7 +220,7 @@ describe('AzureResponsesProvider.complete request shape', () => {
);
const [args] = responsesCreateMock.mock.calls[0]!;
expect(args.model).toBe('gpt-5-codex');
expect(args.model).toBe('gpt-5.3-codex');
expect(args.input).toEqual([{ role: 'user', content: 'hello' }]);
expect(args.max_output_tokens).toBe(256);
expect(args.temperature).toBe(0.3);
@@ -233,12 +233,12 @@ describe('AzureResponsesProvider.complete request shape', () => {
await withTestActor(() =>
provider.complete({
model: 'openai/gpt-5-codex',
model: 'openai/gpt-5.3-codex',
messages: [{ role: 'user', content: 'hi' }],
}),
);
expect(responsesCreateMock.mock.calls[0]![0].model).toBe('gpt-5-codex');
expect(responsesCreateMock.mock.calls[0]![0].model).toBe('gpt-5.3-codex');
});
it('falls back to the default model for an unknown id', async () => {
@@ -252,7 +252,7 @@ describe('AzureResponsesProvider.complete request shape', () => {
}),
);
expect(responsesCreateMock.mock.calls[0]![0].model).toBe('gpt-5-codex');
expect(responsesCreateMock.mock.calls[0]![0].model).toBe('gpt-5.3-codex');
});
it('flattens chat-style function tools into the Responses shape', async () => {
@@ -261,7 +261,7 @@ describe('AzureResponsesProvider.complete request shape', () => {
await withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
model: 'gpt-5.3-codex',
messages: [{ role: 'user', content: 'hi' }],
tools: [
{
@@ -299,7 +299,7 @@ describe('AzureResponsesProvider.complete request shape', () => {
await withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
model: 'gpt-5.3-codex',
messages: [{ role: 'user', content: 'hi' }],
}),
);
@@ -338,7 +338,7 @@ describe('AzureResponsesProvider.complete request shape', () => {
await withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
model: 'gpt-5.3-codex',
messages: [{ role: 'user', content: 'hi' }],
tool_choice: 'auto',
parallel_tool_calls: false,
@@ -380,7 +380,7 @@ describe('AzureResponsesProvider.complete request shape', () => {
await withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
model: 'gpt-5.3-codex',
messages: [{ role: 'user', content: 'hi' }],
compaction: { trigger_tokens: 120_000 },
} as never),
@@ -398,7 +398,7 @@ describe('AzureResponsesProvider.complete request shape', () => {
await withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
model: 'gpt-5.3-codex',
messages: [{ role: 'user', content: 'hi' }],
compaction: true,
context_management: raw,
@@ -416,7 +416,7 @@ describe('AzureResponsesProvider.complete request shape', () => {
responsesCreateMock.mockResolvedValueOnce(okResponse);
await withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
model: 'gpt-5.3-codex',
messages: [{ role: 'user', content: 'hi' }],
reasoning: { effort: 'high' },
verbosity: 'high',
@@ -460,7 +460,7 @@ describe('AzureResponsesProvider usage accounting', () => {
const result = await withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
model: 'gpt-5.3-codex',
messages: [{ role: 'user', content: 'hi' }],
}),
);
@@ -475,11 +475,11 @@ describe('AzureResponsesProvider usage accounting', () => {
cached_tokens: 10,
});
const codex = AZURE_MODELS.find((m) => m.id === 'gpt-5-codex')!;
const codex = AZURE_MODELS.find((m) => m.id === 'gpt-5.3-codex')!;
expect(recordSpy).toHaveBeenCalledTimes(1);
const [usage, actor, prefix, overrides] = recordSpy.mock.calls[0]!;
expect(actor).toBe(SYSTEM_ACTOR);
expect(prefix).toBe('azure-openai:gpt-5-codex');
expect(prefix).toBe('azure-openai:gpt-5.3-codex');
expect(usage).toEqual({
prompt_tokens: 90,
completion_tokens: 50,
@@ -502,7 +502,7 @@ describe('AzureResponsesProvider usage accounting', () => {
await withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
model: 'gpt-5.3-codex',
messages: [{ role: 'user', content: 'hi' }],
}),
);
@@ -539,7 +539,7 @@ describe('AzureResponsesProvider.complete streaming', () => {
const result = await withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
model: 'gpt-5.3-codex',
messages: [{ role: 'user', content: 'say hi' }],
stream: true,
}),
@@ -563,7 +563,7 @@ describe('AzureResponsesProvider.complete streaming', () => {
completion_tokens: 2,
cached_tokens: 1,
});
expect(recordSpy.mock.calls[0]![2]).toBe('azure-openai:gpt-5-codex');
expect(recordSpy.mock.calls[0]![2]).toBe('azure-openai:gpt-5.3-codex');
});
});
@@ -617,7 +617,7 @@ describe('AzureResponsesProvider.complete error mapping', () => {
await expect(
withTestActor(() =>
provider.complete({
model: 'gpt-5-codex',
model: 'gpt-5.3-codex',
messages: [{ role: 'user', content: 'boom' }],
}),
),
@@ -49,7 +49,7 @@ export class AzureResponsesProvider implements IChatProvider {
/** @type {import('openai').OpenAI} */
#openAi: OpenAI;
#defaultModel = 'gpt-5-codex';
#defaultModel = 'gpt-5.3-codex';
#meteringService: MeteringService;
@@ -170,30 +170,6 @@ export const AZURE_MODELS: IChatModel[] = [
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5-codex (same list price as gpt-5).
puterId: 'azure:openai/gpt-5-codex',
id: 'gpt-5-codex',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2024-09-30',
release_date: '2025-08-07',
aliases: ['openai/gpt-5-codex'],
// Codex models are Responses-API only on Azure, same as OpenAI.
responses_api_only: true,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 125,
cached_tokens: 13,
completion_tokens: 1000,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5-nano.
puterId: 'azure:openai/gpt-5-nano',
@@ -260,29 +236,6 @@ export const AZURE_MODELS: IChatModel[] = [
context: 128_000,
max_tokens: 16384,
},
{
// Costs mirror openai gpt-5.1-codex-mini.
puterId: 'azure:openai/gpt-5.1-codex-mini',
id: 'gpt-5.1-codex-mini',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2024-09-30',
release_date: '2025-11-13',
aliases: ['openai/gpt-5.1-codex-mini'],
responses_api_only: true,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 25,
cached_tokens: 3,
completion_tokens: 200,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5.1.
puterId: 'azure:openai/gpt-5.1',
@@ -305,29 +258,6 @@ export const AZURE_MODELS: IChatModel[] = [
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5.1-codex.
puterId: 'azure:openai/gpt-5.1-codex',
id: 'gpt-5.1-codex',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2024-09-30',
release_date: '2025-11-13',
aliases: ['openai/gpt-5.1-codex'],
responses_api_only: true,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 125,
cached_tokens: 13,
completion_tokens: 1000,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5.2.
puterId: 'azure:openai/gpt-5.2',
@@ -350,29 +280,6 @@ export const AZURE_MODELS: IChatModel[] = [
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5.2-codex.
puterId: 'azure:openai/gpt-5.2-codex',
id: 'gpt-5.2-codex',
modalities: { input: ['text', 'image', 'pdf'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-08-31',
release_date: '2025-12-11',
aliases: ['openai/gpt-5.2-codex'],
responses_api_only: true,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 175,
cached_tokens: 18,
completion_tokens: 1400,
},
context: 128_000,
max_tokens: 128000,
},
{
// Costs mirror openai gpt-5.4-nano.
puterId: 'azure:openai/gpt-5.4-nano',
@@ -0,0 +1,63 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Integration test for the BytePlus ModelArk provider.
*
* Uses `seed-1-6-flash-250715` with `thinking: disabled` passed through
* `custom`. Ark's seed models default to deep reasoning and route those
* tokens to a `reasoning_content` field, leaving `content` empty under
* tight budgets. Disabling thinking forces a plain text response so the
* usual `message.content` assertion works. Skipped when
* `PUTER_TEST_AI_BYTEPLUS_API_KEY` is unset.
*/
import { describe, expect, it } from 'vitest';
import {
INTEGRATION_TEST_TIMEOUT_MS,
makeMeteringStub,
optionalEnv,
skipUnlessEnv,
withTestActor,
} from '../../../integrationTestUtil.js';
import { BytePlusProvider } from './BytePlusProvider.js';
const ENV_VAR = 'PUTER_TEST_AI_BYTEPLUS_API_KEY';
describe.skipIf(skipUnlessEnv(ENV_VAR))('BytePlusProvider (integration)', () => {
it('returns a non-empty completion from seed-1-6-flash-250715', { timeout: INTEGRATION_TEST_TIMEOUT_MS }, async () => {
const provider = new BytePlusProvider(
{ apiKey: optionalEnv(ENV_VAR)! },
makeMeteringStub(),
);
const result = await withTestActor(() =>
provider.complete({
model: 'seed-1-6-flash-250715',
messages: [{ role: 'user', content: 'Say hi in one word.' }],
max_tokens: 16,
custom: { thinking: { type: 'disabled' } },
}),
);
const text = (result as { message?: { content?: string } }).message
?.content;
expect(typeof text === 'string' && text.length > 0).toBe(true);
});
});
@@ -0,0 +1,815 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Offline unit tests for BytePlusProvider.
*
* Boots a real PuterServer (in-memory sqlite + dynamo + s3 + mock
* redis) and constructs BytePlusProvider directly against the live
* wired `MeteringService` so the recording side is exercised
* end-to-end. The OpenAI SDK is mocked at the module boundary —
* ModelArk is OpenAI-compatible so the provider talks to it through
* the same client — so the provider never reaches the network. The
* companion integration test (BytePlusProvider.integration.test.ts)
* exercises the real ModelArk endpoint.
*/
import { Writable } from 'node:stream';
import {
afterAll,
afterEach,
beforeAll,
beforeEach,
describe,
expect,
it,
vi,
type MockInstance,
} from 'vitest';
import { SYSTEM_ACTOR } from '../../../../core/actor.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import { PuterServer } from '../../../../server.js';
import { setupTestServer } from '../../../../testUtil.js';
import { withTestActor } from '../../../integrationTestUtil.js';
import { AIChatStream } from '../../utils/Streaming.js';
import { BytePlusProvider } from './BytePlusProvider.js';
import { BYTEPLUS_MODELS } from './models.js';
// -- OpenAI SDK mock -------------------------------------------------
//
// `vi.hoisted` lets us share spies between the (hoisted) factory and
// the test body so each test can stub `chat.completions.create` with
// the response shape it cares about. ModelArk uses the OpenAI wire
// shape so the provider talks to it via the OpenAI SDK.
const { createMock, openAICtor } = vi.hoisted(() => {
const createMock = vi.fn();
const openAICtor = vi.fn();
return { createMock, openAICtor };
});
vi.mock('openai', () => {
const OpenAICtor = vi.fn().mockImplementation(function (
this: Record<string, unknown>,
opts: unknown,
) {
openAICtor(opts);
this.chat = { completions: { create: createMock } };
});
// Some providers (e.g. OllamaChatProvider) import the default export
// and access `.OpenAI` on it, so expose the same constructor under
// both shapes — the test server boots every provider, not just
// BytePlus.
return { OpenAI: OpenAICtor, default: { OpenAI: OpenAICtor } };
});
// -- Test harness ----------------------------------------------------
let server: PuterServer;
let recordSpy: MockInstance<MeteringService['utilRecordUsageObject']>;
beforeAll(async () => {
server = await setupTestServer();
});
afterAll(async () => {
await server?.shutdown();
});
const makeProvider = (
config: { apiKey?: string; apiBaseUrl?: string } = {},
) => {
const provider = new BytePlusProvider(
{
apiKey: config.apiKey ?? 'test-key',
...(config.apiBaseUrl ? { apiBaseUrl: config.apiBaseUrl } : {}),
},
server.services.metering,
);
return { provider };
};
const asAsyncIterable = <T>(items: T[]): AsyncIterable<T> => ({
async *[Symbol.asyncIterator]() {
for (const item of items) {
yield item;
}
},
});
const makeCapturingChatStream = () => {
const chunks: string[] = [];
const sink = new Writable({
write(chunk, _enc, cb) {
chunks.push(chunk.toString('utf8'));
cb();
},
});
const chatStream = new AIChatStream({ stream: sink });
return {
chatStream,
events: () =>
chunks
.join('')
.split('\n')
.filter(Boolean)
.map((line) => JSON.parse(line)),
};
};
beforeEach(() => {
createMock.mockReset();
openAICtor.mockReset();
// Spy on the live MeteringService — we don't replace the impl
// (that would skip the recording side we want covered) but we
// capture the calls the provider makes so per-test assertions
// can verify metering shape.
recordSpy = vi.spyOn(server.services.metering, 'utilRecordUsageObject');
});
afterEach(() => {
vi.restoreAllMocks();
});
// -- Construction ----------------------------------------------------
describe('BytePlusProvider construction', () => {
it('points the OpenAI SDK at the ModelArk base URL with the configured key', () => {
makeProvider();
expect(openAICtor).toHaveBeenCalledTimes(1);
expect(openAICtor).toHaveBeenCalledWith({
apiKey: 'test-key',
baseURL: 'https://ark.ap-southeast.bytepluses.com/api/v3',
});
});
it('honours a custom apiBaseUrl override (e.g. the eu-west region)', () => {
makeProvider({
apiBaseUrl: 'https://ark.eu-west.bytepluses.com/api/v3',
});
expect(openAICtor).toHaveBeenCalledWith({
apiKey: 'test-key',
baseURL: 'https://ark.eu-west.bytepluses.com/api/v3',
});
});
});
// -- Model catalog ---------------------------------------------------
describe('BytePlusProvider model catalog', () => {
it('returns seed-2-0-lite-260428 as the default', () => {
const { provider } = makeProvider();
expect(provider.getDefaultModel()).toBe('seed-2-0-lite-260428');
});
it('exposes the static BYTEPLUS_MODELS list verbatim from models()', () => {
const { provider } = makeProvider();
expect(provider.models()).toBe(BYTEPLUS_MODELS);
});
it('list() flattens canonical ids and aliases', () => {
const { provider } = makeProvider();
const names = provider.list();
for (const m of BYTEPLUS_MODELS) {
expect(names).toContain(m.id);
for (const a of m.aliases ?? []) {
expect(names).toContain(a);
}
}
// Sanity: undated series aliases resolve alongside canonical ids.
expect(names).toContain('seed-1-6');
expect(names).toContain('byteplus/seed-1-6');
expect(names).toContain('byteplus/seed-1-6-250915');
});
it('never claims the bare deepseek-v4 names owned by the DeepSeek provider', () => {
const { provider } = makeProvider();
const names = provider.list();
expect(names).not.toContain('deepseek-v4-pro');
expect(names).not.toContain('deepseek-v4-flash');
expect(names).toContain('byteplus/deepseek-v4-pro');
expect(names).toContain('byteplus/deepseek-v4-flash');
});
});
// -- Request shape ---------------------------------------------------
describe('BytePlusProvider.complete request shape', () => {
const baseCompletion = {
choices: [
{
message: { content: 'hi', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 1, completion_tokens: 1 },
};
it('forwards model, messages, and bare-bones request without optional knobs', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [{ role: 'user', content: 'hello' }],
}),
);
const [args] = createMock.mock.calls[0]!;
expect(args.model).toBe('seed-1-6-250915');
expect(args.messages).toEqual([{ role: 'user', content: 'hello' }]);
// Optional generation knobs should be absent unless supplied.
expect('max_tokens' in args).toBe(false);
expect('temperature' in args).toBe(false);
expect('top_p' in args).toBe(false);
expect('tools' in args).toBe(false);
expect('tool_choice' in args).toBe(false);
expect('thinking' in args).toBe(false);
});
it('forwards max_tokens, temperature, top_p, tools, and tool_choice when supplied', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
const tools = [
{
type: 'function',
function: {
name: 'lookup',
description: 'find a thing',
parameters: {
type: 'object',
properties: { q: { type: 'string' } },
required: ['q'],
},
},
},
];
await withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [{ role: 'user', content: 'hi' }],
max_tokens: 256,
temperature: 0.4,
top_p: 0.9,
tools,
tool_choice: 'auto',
}),
);
const [args] = createMock.mock.calls[0]!;
expect(args.max_tokens).toBe(256);
expect(args.temperature).toBe(0.4);
expect(args.top_p).toBe(0.9);
expect(args.tools).toBe(tools);
expect(args.tool_choice).toBe('auto');
});
it('forwards Ark-specific custom params (thinking, stop, response_format)', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [{ role: 'user', content: 'hi' }],
custom: {
thinking: { type: 'disabled' },
stop: ['\n\n'],
response_format: { type: 'json_object' },
},
}),
);
const [args] = createMock.mock.calls[0]!;
expect(args.thinking).toEqual({ type: 'disabled' });
expect(args.stop).toEqual(['\n\n']);
expect(args.response_format).toEqual({ type: 'json_object' });
});
it('strips Anthropic-style cache_control from messages before sending', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [
{
role: 'user',
content: 'hi',
cache_control: { type: 'ephemeral' },
} as unknown as { role: string; content: string },
],
}),
);
const [args] = createMock.mock.calls[0]!;
expect('cache_control' in args.messages[0]).toBe(false);
});
it('only sets stream_options.include_usage when streaming', async () => {
const { provider } = makeProvider();
// Non-stream path.
createMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [{ role: 'user', content: 'hi' }],
stream: false,
}),
);
const [nonStreamArgs] = createMock.mock.calls[0]!;
expect(nonStreamArgs.stream).toBe(false);
expect('stream_options' in nonStreamArgs).toBe(false);
// Stream path.
createMock.mockReturnValueOnce(asAsyncIterable([]));
await withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [{ role: 'user', content: 'hi' }],
stream: true,
}),
);
const [streamArgs] = createMock.mock.calls[1]!;
expect(streamArgs.stream).toBe(true);
expect(streamArgs.stream_options).toEqual({ include_usage: true });
});
});
// -- Model resolution ------------------------------------------------
describe('BytePlusProvider model resolution', () => {
const baseCompletion = {
choices: [
{
message: { content: 'ok', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 1, completion_tokens: 1 },
};
it('resolves an exact canonical id', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'glm-4-7-251222',
messages: [{ role: 'user', content: 'hi' }],
}),
);
expect(createMock.mock.calls[0]![0].model).toBe('glm-4-7-251222');
expect(recordSpy).toHaveBeenCalledWith(
expect.any(Object),
expect.anything(),
'byteplus:glm-4-7-251222',
expect.any(Object),
);
});
it('resolves an undated series alias to the newest snapshot (alias rewriting)', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'seed-1-6',
messages: [{ role: 'user', content: 'hi' }],
}),
);
// The wire model should be the canonical dated id, not the alias.
expect(createMock.mock.calls[0]![0].model).toBe('seed-1-6-250915');
expect(recordSpy).toHaveBeenCalledWith(
expect.any(Object),
expect.anything(),
'byteplus:seed-1-6-250915',
expect.any(Object),
);
});
it('falls back to the default model when given an unknown id', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce(baseCompletion);
await withTestActor(() =>
provider.complete({
model: 'totally-not-a-real-model',
messages: [{ role: 'user', content: 'hi' }],
}),
);
expect(createMock.mock.calls[0]![0].model).toBe(
'seed-2-0-lite-260428',
);
});
});
// -- Non-stream completion + reasoning_content normalisation ---------
describe('BytePlusProvider.complete non-stream output', () => {
it('returns the first choice and runs the metered usage calculator', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'hi there', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: {
prompt_tokens: 100,
completion_tokens: 50,
prompt_tokens_details: { cached_tokens: 10 },
},
});
const result = await withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [{ role: 'user', content: 'hi' }],
}),
);
expect(result).toMatchObject({
message: { content: 'hi there', role: 'assistant' },
finish_reason: 'stop',
});
expect((result as { usage: unknown }).usage).toEqual({
prompt_tokens: 100,
completion_tokens: 50,
cached_tokens: 10,
});
// Cost overrides scale per-token usage by the per-token cents from
// the model's costs table, so derive expectations from
// BYTEPLUS_MODELS directly to avoid hardcoded float-precision drift.
const seed16 = BYTEPLUS_MODELS.find(
(m) => m.id === 'seed-1-6-250915',
)!;
expect(recordSpy).toHaveBeenCalledTimes(1);
const [usage, actor, prefix, overrides] = recordSpy.mock.calls[0]!;
expect(usage).toEqual({
prompt_tokens: 100,
completion_tokens: 50,
cached_tokens: 10,
});
expect(actor).toBe(SYSTEM_ACTOR);
expect(prefix).toBe('byteplus:seed-1-6-250915');
expect(overrides.prompt_tokens).toBeCloseTo(
100 * Number(seed16.costs.prompt_tokens),
5,
);
expect(overrides.completion_tokens).toBeCloseTo(
50 * Number(seed16.costs.completion_tokens),
5,
);
expect(overrides.cached_tokens).toBeCloseTo(
10 * Number(seed16.costs.cached_tokens ?? 0),
5,
);
});
it('preserves OpenAI-shaped tool_calls on the assistant response', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: {
role: 'assistant',
content: null,
tool_calls: [
{
id: 'call_1',
type: 'function',
function: {
name: 'lookup',
arguments: '{"q":"puter"}',
},
},
],
},
finish_reason: 'tool_calls',
},
],
usage: { prompt_tokens: 1, completion_tokens: 1 },
});
const result = (await withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [{ role: 'user', content: 'do a tool call' }],
tools: [
{
type: 'function',
function: { name: 'lookup', parameters: {} },
},
],
}),
)) as { message: { tool_calls?: unknown[] }; finish_reason: string };
expect(result.finish_reason).toBe('tool_calls');
expect(result.message.tool_calls).toEqual([
{
id: 'call_1',
type: 'function',
function: {
name: 'lookup',
arguments: '{"q":"puter"}',
},
},
]);
});
it('renames Ark `reasoning_content` to `reasoning` on the message', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: {
role: 'assistant',
content: 'final answer',
reasoning_content: 'thinking out loud',
},
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 1, completion_tokens: 1 },
});
const result = (await withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [{ role: 'user', content: 'hi' }],
}),
)) as { message: Record<string, unknown> };
expect(result.message.reasoning).toBe('thinking out loud');
expect('reasoning_content' in result.message).toBe(false);
});
it('does not overwrite an existing `reasoning` field if both are present', async () => {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: {
role: 'assistant',
content: 'final',
reasoning: 'original',
reasoning_content: 'should-be-dropped',
},
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 1, completion_tokens: 1 },
});
const result = (await withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [{ role: 'user', content: 'hi' }],
}),
)) as { message: Record<string, unknown> };
expect(result.message.reasoning).toBe('original');
expect('reasoning_content' in result.message).toBe(false);
});
});
// -- Streaming deltas ------------------------------------------------
describe('BytePlusProvider.complete streaming', () => {
it('streams text deltas through to text events and meters final usage', async () => {
const { provider } = makeProvider();
createMock.mockReturnValueOnce(
asAsyncIterable([
{ choices: [{ delta: { content: 'hel' } }] },
{ choices: [{ delta: { content: 'lo' } }] },
{
choices: [{ delta: {} }],
usage: {
prompt_tokens: 4,
completion_tokens: 2,
prompt_tokens_details: { cached_tokens: 1 },
},
},
]),
);
const result = await withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [{ role: 'user', content: 'say hi' }],
stream: true,
}),
);
expect((result as { stream: boolean }).stream).toBe(true);
const harness = makeCapturingChatStream();
await (
result as {
init_chat_stream: (p: { chatStream: unknown }) => Promise<void>;
}
).init_chat_stream({ chatStream: harness.chatStream });
const events = harness.events();
const textEvents = events.filter((e) => e.type === 'text');
expect(textEvents.map((e) => e.text)).toEqual(['hel', 'lo']);
const usageEvent = events.find((e) => e.type === 'usage');
expect(usageEvent?.usage).toEqual({
prompt_tokens: 4,
completion_tokens: 2,
cached_tokens: 1,
});
const seed16 = BYTEPLUS_MODELS.find(
(m) => m.id === 'seed-1-6-250915',
)!;
expect(recordSpy).toHaveBeenCalledTimes(1);
const [, , prefix, overrides] = recordSpy.mock.calls[0]!;
expect(prefix).toBe('byteplus:seed-1-6-250915');
expect(overrides.prompt_tokens).toBeCloseTo(
4 * Number(seed16.costs.prompt_tokens),
5,
);
expect(overrides.completion_tokens).toBeCloseTo(
2 * Number(seed16.costs.completion_tokens),
5,
);
expect(overrides.cached_tokens).toBeCloseTo(
1 * Number(seed16.costs.cached_tokens ?? 0),
5,
);
});
it('streams reasoning_content deltas as reasoning events', async () => {
const { provider } = makeProvider();
createMock.mockReturnValueOnce(
asAsyncIterable([
{
choices: [
{ delta: { reasoning_content: 'pondering...' } },
],
},
{ choices: [{ delta: { content: 'answer' } }] },
{
choices: [{ delta: {} }],
usage: { prompt_tokens: 1, completion_tokens: 1 },
},
]),
);
const result = await withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [{ role: 'user', content: 'think hard' }],
stream: true,
}),
);
const harness = makeCapturingChatStream();
await (
result as {
init_chat_stream: (p: { chatStream: unknown }) => Promise<void>;
}
).init_chat_stream({ chatStream: harness.chatStream });
const events = harness.events();
const reasoningEvents = events.filter((e) => e.type === 'reasoning');
expect(reasoningEvents.map((e) => e.reasoning)).toEqual([
'pondering...',
]);
const textEvents = events.filter((e) => e.type === 'text');
expect(textEvents.map((e) => e.text)).toEqual(['answer']);
});
it('builds a tool_use block from streamed function-call deltas', async () => {
const { provider } = makeProvider();
createMock.mockReturnValueOnce(
asAsyncIterable([
{
choices: [
{
delta: {
tool_calls: [
{
index: 0,
id: 'call_1',
function: {
name: 'lookup',
arguments: '{"q":',
},
},
],
},
},
],
},
{
choices: [
{
delta: {
tool_calls: [
{
index: 0,
function: { arguments: '"puter"}' },
},
],
},
},
],
},
{
choices: [{ delta: {} }],
usage: { prompt_tokens: 1, completion_tokens: 1 },
},
]),
);
const result = await withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [{ role: 'user', content: 'do tool call' }],
tools: [
{
type: 'function',
function: { name: 'lookup', parameters: {} },
},
],
stream: true,
}),
);
const harness = makeCapturingChatStream();
await (
result as {
init_chat_stream: (p: { chatStream: unknown }) => Promise<void>;
}
).init_chat_stream({ chatStream: harness.chatStream });
const events = harness.events();
const toolEvent = events.find((e) => e.type === 'tool_use');
expect(toolEvent).toBeDefined();
expect(toolEvent?.id).toBe('call_1');
expect(toolEvent?.name).toBe('lookup');
expect(toolEvent?.input).toEqual({ q: 'puter' });
});
});
// -- Error mapping ---------------------------------------------------
describe('BytePlusProvider.complete error mapping', () => {
it('rethrows errors raised by the OpenAI client unchanged', async () => {
const { provider } = makeProvider();
const apiError = new Error('ModelArk exploded');
createMock.mockRejectedValueOnce(apiError);
await expect(
withTestActor(() =>
provider.complete({
model: 'seed-1-6-250915',
messages: [{ role: 'user', content: 'boom' }],
}),
),
).rejects.toBe(apiError);
// No metering should be recorded on a failed call.
expect(recordSpy).not.toHaveBeenCalled();
});
});
// -- Moderation ------------------------------------------------------
describe('BytePlusProvider.checkModeration', () => {
it('throws — BytePlus provider does not implement moderation', () => {
const { provider } = makeProvider();
expect(() => provider.checkModeration('anything')).toThrow(
/not implemented/i,
);
});
});
@@ -0,0 +1,180 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { OpenAI } from 'openai';
import { ChatCompletionCreateParams } from 'openai/resources/index.js';
import { Context } from '../../../../core/context.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import type { IChatProvider, ICompleteArguments } from '../../types.js';
import * as OpenAIUtil from '../../utils/OpenAIUtil.js';
import { BYTEPLUS_MODELS } from './models.js';
type BytePlusConfig = {
apiKey: string;
apiBaseUrl?: string;
};
type BytePlusCustomParams = {
response_format?: unknown;
stop?: string[];
// Ark-specific toggle for deep reasoning; the seed models reason by
// default and route those tokens to `reasoning_content`.
thinking?: {
type?: 'enabled' | 'disabled' | 'auto';
};
};
const asRecord = (value: unknown): Record<string, unknown> =>
value && typeof value === 'object' && !Array.isArray(value)
? (value as Record<string, unknown>)
: {};
/**
* BytePlus ModelArk provider — an OpenAI-compatible endpoint serving
* ByteDance's Seed models plus hosted third-party models (GLM, DeepSeek,
* GPT-OSS). https://docs.byteplus.com/en/docs/ModelArk/1330626
*/
export class BytePlusProvider implements IChatProvider {
#openai: OpenAI;
#meteringService: MeteringService;
#defaultModel = 'seed-2-0-lite-260428';
constructor(config: BytePlusConfig, meteringService: MeteringService) {
this.#openai = new OpenAI({
apiKey: config.apiKey,
baseURL:
config.apiBaseUrl ??
'https://ark.ap-southeast.bytepluses.com/api/v3',
});
this.#meteringService = meteringService;
}
getDefaultModel() {
return this.#defaultModel;
}
models() {
return BYTEPLUS_MODELS;
}
list() {
const modelIds: string[] = [];
for (const model of this.models()) {
modelIds.push(model.id);
if (model.aliases) {
modelIds.push(...model.aliases);
}
}
return modelIds;
}
async complete(
params: ICompleteArguments,
): ReturnType<IChatProvider['complete']> {
const {
custom,
max_tokens,
stream,
temperature,
tools,
tool_choice,
top_p,
} = params;
let { messages, model } = params;
const actor = Context.get('actor');
const availableModels = this.models();
const modelUsed =
availableModels.find((m) =>
[m.id, ...(m.aliases || [])].includes(model),
) || availableModels.find((m) => m.id === this.getDefaultModel())!;
messages = await OpenAIUtil.process_input_messages(messages);
messages = messages.map((message) => {
delete message.cache_control;
return message;
});
const customParams = asRecord(custom) as BytePlusCustomParams;
const completionParams: ChatCompletionCreateParams = {
messages,
model: modelUsed.id,
...(tools ? { tools } : {}),
...(tool_choice !== undefined ? { tool_choice } : {}),
...(max_tokens !== undefined ? { max_tokens } : {}),
...(temperature !== undefined ? { temperature } : {}),
...(top_p !== undefined ? { top_p } : {}),
...(customParams.response_format
? { response_format: customParams.response_format }
: {}),
...(customParams.stop ? { stop: customParams.stop } : {}),
...(customParams.thinking
? { thinking: customParams.thinking }
: {}),
stream: !!stream,
...(stream
? {
stream_options: { include_usage: true },
}
: {}),
} as ChatCompletionCreateParams;
const completion =
await this.#openai.chat.completions.create(completionParams);
const result = await OpenAIUtil.handle_completion_output({
usage_calculator: ({ usage }) => {
const trackedUsage = usage
? OpenAIUtil.extractMeteredUsage(usage)
: {
prompt_tokens: 0,
completion_tokens: 0,
cached_tokens: 0,
};
const costsOverride = Object.fromEntries(
Object.entries(trackedUsage).map(([key, value]) => {
return [key, value * Number(modelUsed.costs[key] ?? 0)];
}),
);
this.#meteringService.utilRecordUsageObject(
trackedUsage,
actor!,
`byteplus:${modelUsed.id}`,
costsOverride,
);
return trackedUsage;
},
stream,
completion,
});
// Ark's deep-reasoning models return `reasoning_content` (DeepSeek
// wire convention); expose it under `reasoning` like other providers.
OpenAIUtil.normalizeReasoningContent(result);
return result;
}
checkModeration(
_text: string,
): ReturnType<IChatProvider['checkModeration']> {
throw new Error('Method not implemented.');
}
}
@@ -0,0 +1,196 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { IChatModel } from '../../types.js';
import { usdPerMToken } from '../../utils/pricing.js';
const K = 1_000;
const bytePlusModel = (
id: string,
name: string,
context: number,
maxTokens: number,
costs: IChatModel['costs'],
opts: {
input?: string[];
extraAliases?: string[];
openWeights?: boolean;
} = {},
): IChatModel => ({
puterId: `byteplus:byteplus/${id}`,
id,
name,
aliases: [`byteplus/${id}`, ...(opts.extraAliases ?? [])],
modalities: { input: opts.input ?? ['text'], output: ['text'] },
open_weights: opts.openWeights ?? false,
tool_call: true,
context,
max_tokens: maxTokens,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs,
});
// An undated series alias (e.g. `seed-1-6`) points at the newest snapshot of
// that series, mirroring how other providers alias their rolling names.
const series = (alias: string) => [alias, `byteplus/${alias}`];
const VISION = ['text', 'image', 'video'];
// Hardcoded from https://docs.byteplus.com/en/docs/ModelArk/1330310 (catalog)
// and https://docs.byteplus.com/en/docs/ModelArk/1544106 (pricing).
//
// Some models bill a higher rate for prompts above 128K tokens; the costs
// below are the base tier. seed-2-0-lite/mini-260428 also accept audio input
// at a much higher per-token rate ModelArk folds into prompt_tokens, so audio
// is deliberately left out of `modalities` to keep metering honest.
export const BYTEPLUS_MODELS: IChatModel[] = [
bytePlusModel(
'dola-seed-2-1-turbo-260628',
'Dola Seed 2.1 Turbo',
256 * K,
256 * K,
usdPerMToken(0.5, 2.5, 0.1),
{ input: VISION, extraAliases: series('dola-seed-2-1-turbo') },
),
bytePlusModel(
'seed-2-0-lite-260428',
'Seed 2.0 Lite',
256 * K,
128 * K,
usdPerMToken(0.25, 2, 0.05),
{ input: VISION, extraAliases: series('seed-2-0-lite') },
),
bytePlusModel(
'seed-2-0-lite-260228',
'Seed 2.0 Lite',
256 * K,
128 * K,
usdPerMToken(0.25, 2, 0.05),
{ input: VISION },
),
bytePlusModel(
'seed-2-0-mini-260428',
'Seed 2.0 Mini',
256 * K,
128 * K,
usdPerMToken(0.1, 0.4, 0.02),
{ input: VISION, extraAliases: series('seed-2-0-mini') },
),
bytePlusModel(
'seed-2-0-mini-260215',
'Seed 2.0 Mini',
256 * K,
128 * K,
usdPerMToken(0.1, 0.4, 0.02),
{ input: VISION },
),
bytePlusModel(
'seed-2-0-pro-260328',
'Seed 2.0 Pro',
256 * K,
128 * K,
usdPerMToken(0.5, 3, 0.1),
{ input: VISION, extraAliases: series('seed-2-0-pro') },
),
bytePlusModel(
'seed-2-0-code-preview-260328',
'Seed 2.0 Code Preview',
256 * K,
128 * K,
usdPerMToken(0.5, 3, 0.1),
{ input: VISION, extraAliases: series('seed-2-0-code-preview') },
),
bytePlusModel(
'seed-1-8-251228',
'Seed 1.8',
256 * K,
64 * K,
usdPerMToken(0.25, 2, 0.05),
{ input: VISION, extraAliases: series('seed-1-8') },
),
bytePlusModel(
'seed-1-6-250915',
'Seed 1.6',
256 * K,
32 * K,
usdPerMToken(0.25, 2, 0.05),
{ input: VISION, extraAliases: series('seed-1-6') },
),
bytePlusModel(
'seed-1-6-flash-250715',
'Seed 1.6 Flash',
256 * K,
32 * K,
usdPerMToken(0.075, 0.3, 0.015),
{ input: VISION, extraAliases: series('seed-1-6-flash') },
),
bytePlusModel(
'glm-5-2-260617',
'GLM-5.2',
1_024 * K,
128 * K,
usdPerMToken(1.4, 4.4, 0.26),
{ extraAliases: series('glm-5-2') },
),
bytePlusModel(
'glm-4-7-251222',
'GLM-4.7',
200 * K,
128 * K,
usdPerMToken(0.6, 2.2, 0.11),
{ extraAliases: series('glm-4-7') },
),
// The bare `deepseek-v4-*` names belong to the first-party DeepSeek
// provider, so these only carry the byteplus-prefixed aliases.
bytePlusModel(
'deepseek-v4-pro-260425',
'DeepSeek V4 Pro',
1_024 * K,
384 * K,
usdPerMToken(1.74, 3.48, 0.145),
{ extraAliases: ['byteplus/deepseek-v4-pro'] },
),
bytePlusModel(
'deepseek-v4-flash-260425',
'DeepSeek V4 Flash',
1_024 * K,
384 * K,
usdPerMToken(0.14, 0.28, 0.028),
{ extraAliases: ['byteplus/deepseek-v4-flash'] },
),
bytePlusModel(
'deepseek-v3-2-251201',
'DeepSeek V3.2',
128 * K,
32 * K,
usdPerMToken(0.28, 0.42, 0.056),
{ extraAliases: series('deepseek-v3-2') },
),
bytePlusModel(
'gpt-oss-120b-250805',
'GPT-OSS 120B',
128 * K,
64 * K,
usdPerMToken(0.1, 0.5),
{ extraAliases: series('gpt-oss-120b'), openWeights: true },
),
];
@@ -536,7 +536,10 @@ export class ClaudeProvider implements IChatProvider {
}
}
}
chatStream.end(usageSum);
// Metered before `end`: handing usage to `end` is what tells
// the driver this completion has been charged for, so anything
// that throws between the two would leave it charged to
// nobody.
const costsOverrideFromModel =
this.#buildCostsOverrideFromModel(usageSum, modelUsed);
this.#meteringService.utilRecordUsageObject(
@@ -545,6 +548,7 @@ export class ClaudeProvider implements IChatProvider {
`claude:${modelUsed.id}`,
costsOverrideFromModel,
);
chatStream.end(usageSum);
};
return {
@@ -20,7 +20,7 @@
/**
* Integration test for the Gemini chat provider.
*
* Hits the real Google Gemini API with `gemini-2.0-flash-lite` (one
* Hits the real Google Gemini API with `gemini-3.7-flash` (one
* of the cheapest variants). Skipped when `PUTER_TEST_AI_GEMINI_API_KEY`
* is unset.
*/
@@ -40,14 +40,14 @@ const ENV_VAR = 'PUTER_TEST_AI_GEMINI_API_KEY';
describe.skipIf(skipUnlessEnv(ENV_VAR))(
'GeminiChatProvider (integration)',
() => {
it('returns a non-empty completion from gemini-2.0-flash-lite', { timeout: INTEGRATION_TEST_TIMEOUT_MS }, async () => {
it('returns a non-empty completion from gemini-3.7-flash', { timeout: INTEGRATION_TEST_TIMEOUT_MS }, async () => {
const provider = new GeminiChatProvider(makeMeteringStub(), {
apiKey: optionalEnv(ENV_VAR)!,
});
const result = await withTestActor(() =>
provider.complete({
model: 'gemini-2.0-flash-lite',
model: 'gemini-3.7-flash',
messages: [
{ role: 'user', content: 'Say hi in one word.' },
],
@@ -400,46 +400,51 @@ describe('GeminiChatProvider.complete non-stream output', () => {
});
it('bills cached tokens at the input rate when the model prices no cache read', async () => {
// gemini-2.0-flash-lite's catalogue entry has no cached_tokens rate.
// Cached tokens are subtracted out of prompt_tokens, so pricing them
// at zero bills them nowhere.
// Every shipped entry currently prices cache reads, so drop the rate
// off one for this call. Cached tokens are subtracted out of
// prompt_tokens, so pricing them at zero bills them nowhere.
const lite = GEMINI_MODELS.find(
(m) => m.id === 'gemini-2.0-flash-lite',
(m) => m.id === 'gemini-3.1-flash-lite',
)!;
expect(lite.costs.cached_tokens).toBeUndefined();
const cachedRate = lite.costs.cached_tokens;
delete lite.costs.cached_tokens;
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'cached', role: 'assistant' },
finish_reason: 'stop',
try {
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'cached', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: {
prompt_tokens: 3000,
completion_tokens: 40,
prompt_tokens_details: { cached_tokens: 2900 },
},
],
usage: {
prompt_tokens: 3000,
completion_tokens: 40,
prompt_tokens_details: { cached_tokens: 2900 },
},
});
});
await withTestActor(() =>
provider.complete({
model: 'gemini-2.0-flash-lite',
messages: [{ role: 'user', content: 'hi' }],
}),
);
await withTestActor(() =>
provider.complete({
model: 'gemini-3.1-flash-lite',
messages: [{ role: 'user', content: 'hi' }],
}),
);
const [, , , overrides] = recordSpy.mock.calls[0]!;
const inputRate = Number(lite.costs.prompt_tokens);
expect(overrides).toMatchObject({
prompt_tokens: (3000 - 2900) * inputRate,
completion_tokens: 40 * Number(lite.costs.completion_tokens),
cached_tokens: 2900 * inputRate,
});
expect(
(overrides as Record<string, number>).cached_tokens,
).toBeGreaterThan(0);
const [, , , overrides] = recordSpy.mock.calls[0]!;
const inputRate = Number(lite.costs.prompt_tokens);
expect(overrides).toMatchObject({
prompt_tokens: (3000 - 2900) * inputRate,
completion_tokens: 40 * Number(lite.costs.completion_tokens),
cached_tokens: 2900 * inputRate,
});
expect(
(overrides as Record<string, number>).cached_tokens,
).toBeGreaterThan(0);
} finally {
lite.costs.cached_tokens = cachedRate;
}
});
it('zeroes cached_tokens when prompt_tokens_details is missing', async () => {
@@ -643,9 +648,9 @@ describe('GeminiChatProvider.complete grounding request metering', () => {
// generation; without its own rate the fee fell through to the input
// token rate, which is several orders of magnitude below list.
const lite = GEMINI_MODELS.find(
(m) => m.id === 'gemini-2.0-flash-lite',
(m) => m.id === 'gemini-3.1-flash-lite',
)!;
expect(lite.costs.grounding_requests).toBe(3_500_000);
expect(lite.costs.grounding_requests).toBe(1_400_000);
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
@@ -666,7 +671,7 @@ describe('GeminiChatProvider.complete grounding request metering', () => {
await withTestActor(() =>
provider.complete({
model: 'gemini-2.0-flash-lite',
model: 'gemini-3.1-flash-lite',
messages: [{ role: 'user', content: 'search for foo' }],
}),
);
@@ -50,57 +50,85 @@ export const GEMINI_MODELS: IChatModel[] = [
},
},
{
puterId: 'google:google/gemini-2.0-flash',
id: 'gemini-2.0-flash',
puterId: 'google:google/gemini-3.5-flash-lite',
id: 'gemini-3.5-flash-lite',
modalities: {
input: ['text', 'image', 'audio', 'video', 'pdf'],
output: ['text'],
},
open_weights: false,
tool_call: true,
knowledge: '2024-06',
release_date: '2024-12-11',
name: 'Gemini 2.0 Flash',
aliases: ['google/gemini-2.0-flash'],
context: 131072,
knowledge: '2025-01',
release_date: '2026-07-21',
name: 'Gemini 3.5 Flash-Lite',
aliases: ['google/gemini-3.5-flash-lite'],
context: 1_048_576,
max_tokens: 65_536,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 10,
completion_tokens: 40,
prompt_tokens: 30,
completion_tokens: 250,
thinking_tokens: 250,
cached_tokens: 3,
// Gemini 2.x grounding is $35 / 1,000 requests
grounding_requests: 3_500_000,
grounding_requests: 1_400_000,
},
max_tokens: 8192,
},
{
puterId: 'google:google/gemini-2.0-flash-lite',
id: 'gemini-2.0-flash-lite',
puterId: 'google:google/gemini-3.6-flash',
id: 'gemini-3.6-flash',
modalities: {
input: ['text', 'image', 'audio', 'video', 'pdf'],
output: ['text'],
},
open_weights: false,
tool_call: true,
knowledge: '2024-06',
release_date: '2024-12-11',
name: 'Gemini 2.0 Flash-Lite',
aliases: ['google/gemini-2.0-flash-lite'],
knowledge: '2025-01',
release_date: '2026-07-21',
name: 'Gemini 3.6 Flash',
aliases: ['google/gemini-3.6-flash'],
context: 1_048_576,
max_tokens: 65_536,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 8,
completion_tokens: 30,
// Gemini 2.x grounding is $35 / 1,000 requests
grounding_requests: 3_500_000,
prompt_tokens: 150,
completion_tokens: 750,
thinking_tokens: 750,
cached_tokens: 15,
grounding_requests: 1_400_000,
},
},
{
puterId: 'google:google/gemini-3.7-flash',
id: 'gemini-3.7-flash',
modalities: {
input: ['text', 'image', 'audio', 'video', 'pdf'],
output: ['text'],
},
open_weights: false,
tool_call: true,
knowledge: '2025-01',
release_date: '2026-08-13',
name: 'Gemini 3.7 Flash',
aliases: ['google/gemini-3.7-flash'],
context: 1_048_576,
max_tokens: 65_536,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 75,
completion_tokens: 375,
thinking_tokens: 375,
cached_tokens: 8,
grounding_requests: 1_400_000,
},
max_tokens: 8192,
},
{
puterId: 'google:google/gemini-2.5-flash',
@@ -273,4 +301,32 @@ export const GEMINI_MODELS: IChatModel[] = [
},
max_tokens: 65536,
},
{
puterId: 'google:google/gemini-3.7-flash',
id: 'gemini-3.7-flash',
modalities: {
input: ['text', 'image', 'video', 'audio', 'pdf'],
output: ['text'],
},
open_weights: false,
tool_call: true,
knowledge: '2026-03',
release_date: '2026-08-13',
name: 'Gemini 3.7 Flash',
aliases: ['google/gemini-3.7-flash'],
context: 1_048_576,
max_tokens: 65_536,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 75,
completion_tokens: 375,
thinking_tokens: 375,
cached_tokens: 7.5,
// Gemini 3.x grounding is $14 / 1,000 requests
grounding_requests: 1_400_000,
},
},
];
@@ -305,16 +305,16 @@ describe('GroqAIProvider model resolution', () => {
await withTestActor(() =>
provider.complete({
model: 'gemma2-9b-it',
model: 'llama-3.3-70b-versatile',
messages: [{ role: 'user', content: 'hi' }],
}),
);
expect(createMock.mock.calls[0]![0].model).toBe('gemma2-9b-it');
expect(createMock.mock.calls[0]![0].model).toBe('llama-3.3-70b-versatile');
expect(recordSpy).toHaveBeenCalledWith(
expect.any(Object),
expect.anything(),
'groq:gemma2-9b-it',
'groq:llama-3.3-70b-versatile',
expect.any(Object),
);
});
@@ -19,116 +19,17 @@
import type { IChatModel } from '../../types.js';
// Hardcoded from https://models.dev/api.json
// Hardcoded from https://models.dev/api.json (Groq chat/text models)
export const GROQ_MODELS: IChatModel[] = [
{
id: 'gemma2-9b-it',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: true,
knowledge: '2024-06',
release_date: '2024-06-27',
name: 'Gemma 2 9B 8k',
context: 8192,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 20,
completion_tokens: 20,
cached_tokens: 0,
},
max_tokens: 8192,
},
{
id: 'gemma-7b-it',
// Not present in models.dev/api.json (as of 2026-02-11)
name: 'Gemma 7B 8k Instruct',
context: 8192,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 7,
completion_tokens: 7,
cached_tokens: 0,
},
max_tokens: 8192,
},
{
id: 'llama3-groq-70b-8192-tool-use-preview',
// Not present in models.dev/api.json (as of 2026-02-11)
name: 'Llama 3 Groq 70B Tool Use Preview 8k',
context: 8192,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 89,
completion_tokens: 89,
cached_tokens: 0,
},
max_tokens: 8192,
},
{
id: 'llama3-groq-8b-8192-tool-use-preview',
// Not present in models.dev/api.json (as of 2026-02-11)
name: 'Llama 3 Groq 8B Tool Use Preview 8k',
context: 8192,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 19,
completion_tokens: 19,
cached_tokens: 0,
},
max_tokens: 8192,
},
{
id: 'llama-3.1-70b-versatile',
// Not present in models.dev/api.json (as of 2026-02-11)
name: 'Llama 3.1 70B Versatile 128k',
context: 128000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 59,
completion_tokens: 79,
cached_tokens: 0,
},
max_tokens: 128000,
},
{
id: 'llama-3.1-70b-specdec',
// Not present in models.dev/api.json (as of 2026-02-11)
name: 'Llama 3.1 8B Instant 128k',
context: 128000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 59,
completion_tokens: 99,
cached_tokens: 0,
},
max_tokens: 128000,
},
{
puterId: 'groq:llama-3.1-8b-instant',
id: 'llama-3.1-8b-instant',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: true,
knowledge: '2023-12',
release_date: '2024-07-23',
name: 'Llama 3.1 8B Instant 128k',
name: 'Llama 3.1 8B Instant',
aliases: ['llama-3.1-8b-instant'],
context: 131072,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
@@ -142,28 +43,182 @@ export const GROQ_MODELS: IChatModel[] = [
max_tokens: 131072,
},
{
id: 'meta-llama/llama-guard-4-12b',
modalities: { input: ['text', 'image'], output: ['text'] },
puterId: 'groq:llama-3.3-70b-versatile',
id: 'llama-3.3-70b-versatile',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: false,
release_date: '2025-04-05',
name: 'Llama Guard 4 12B',
tool_call: true,
release_date: '2024-12-06',
name: 'Llama 3.3 70B Versatile',
aliases: ['llama-3.3-70b-versatile'],
context: 131072,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 20,
completion_tokens: 20,
prompt_tokens: 59,
completion_tokens: 79,
cached_tokens: 0,
},
max_tokens: 1024,
max_tokens: 32768,
},
{
puterId: 'groq:openai/gpt-oss-120b',
id: 'openai/gpt-oss-120b',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: true,
release_date: '2025-08-05',
name: 'GPT OSS 120B',
aliases: ['openai/gpt-oss-120b'],
context: 131072,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 15,
completion_tokens: 60,
cached_tokens: 8,
},
max_tokens: 65536,
},
{
puterId: 'groq:openai/gpt-oss-20b',
id: 'openai/gpt-oss-20b',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: true,
release_date: '2025-08-05',
name: 'GPT OSS 20B',
aliases: ['openai/gpt-oss-20b'],
context: 131072,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 8,
completion_tokens: 30,
cached_tokens: 4,
},
max_tokens: 65536,
},
{
puterId: 'groq:openai/gpt-oss-safeguard-20b',
id: 'openai/gpt-oss-safeguard-20b',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: true,
release_date: '2025-10-29',
name: 'GPT OSS Safeguard 20B',
aliases: ['openai/gpt-oss-safeguard-20b'],
context: 131072,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 8,
completion_tokens: 30,
cached_tokens: 0,
},
max_tokens: 65536,
},
{
puterId: 'groq:groq/compound',
id: 'groq/compound',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: false,
release_date: '2025-09-04',
name: 'Groq Compound',
aliases: ['groq/compound'],
context: 131072,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 0,
completion_tokens: 0,
cached_tokens: 0,
},
max_tokens: 8192,
},
{
puterId: 'groq:groq/compound-mini',
id: 'groq/compound-mini',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: false,
release_date: '2025-09-04',
name: 'Groq Compound Mini',
aliases: ['groq/compound-mini'],
context: 131072,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 0,
completion_tokens: 0,
cached_tokens: 0,
},
max_tokens: 8192,
},
{
puterId: 'groq:qwen/qwen3.6-27b',
id: 'qwen/qwen3.6-27b',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: true,
tool_call: true,
release_date: '2026-04-22',
name: 'Qwen3.6 27B',
aliases: ['qwen/qwen3.6-27b'],
context: 131072,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 60,
completion_tokens: 300,
cached_tokens: 30,
},
max_tokens: 16384,
},
{
puterId: 'groq:meta-llama/llama-prompt-guard-2-22m',
id: 'meta-llama/llama-prompt-guard-2-22m',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: false,
release_date: '2025-05-29',
name: 'Llama Prompt Guard 2 22M',
aliases: ['meta-llama/llama-prompt-guard-2-22m'],
context: 512,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 3,
completion_tokens: 3,
cached_tokens: 0,
},
max_tokens: 512,
},
{
puterId: 'groq:meta-llama/llama-prompt-guard-2-86m',
id: 'meta-llama/llama-prompt-guard-2-86m',
// Not present in models.dev/api.json (as of 2026-02-11)
name: 'Prompt Guard 2 86M',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: false,
release_date: '2025-05-29',
name: 'Llama Prompt Guard 2 86M',
aliases: ['meta-llama/llama-prompt-guard-2-86m'],
context: 512,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
@@ -177,133 +232,36 @@ export const GROQ_MODELS: IChatModel[] = [
max_tokens: 512,
},
{
id: 'llama-3.2-1b-preview',
// Not present in models.dev/api.json (as of 2026-02-11)
name: 'Llama 3.2 1B (Preview) 8k',
context: 128000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 4,
completion_tokens: 4,
cached_tokens: 0,
},
max_tokens: 128000,
},
{
id: 'llama-3.2-3b-preview',
// Not present in models.dev/api.json (as of 2026-02-11)
name: 'Llama 3.2 3B (Preview) 8k',
context: 128000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 6,
completion_tokens: 6,
cached_tokens: 0,
},
max_tokens: 128000,
},
{
id: 'llama-3.2-11b-vision-preview',
// Not present in models.dev/api.json (as of 2026-02-11)
name: 'Llama 3.2 11B Vision 8k (Preview)',
context: 8000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 18,
completion_tokens: 18,
cached_tokens: 0,
},
max_tokens: 8000,
},
{
id: 'llama-3.2-90b-vision-preview',
// Not present in models.dev/api.json (as of 2026-02-11)
name: 'Llama 3.2 90B Vision 8k (Preview)',
context: 8000,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 90,
completion_tokens: 90,
cached_tokens: 0,
},
max_tokens: 8000,
},
{
id: 'llama3-70b-8192',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: true,
knowledge: '2023-03',
release_date: '2024-04-18',
name: 'Llama 3 70B 8k',
context: 8192,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 59,
completion_tokens: 79,
cached_tokens: 0,
},
max_tokens: 8192,
},
{
id: 'llama3-8b-8192',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: true,
knowledge: '2023-03',
release_date: '2024-04-18',
name: 'Llama 3 8B 8k',
context: 8192,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 5,
completion_tokens: 8,
cached_tokens: 0,
},
max_tokens: 8192,
},
{
id: 'mixtral-8x7b-32768',
// Not present in models.dev/api.json (as of 2026-02-11)
name: 'Mixtral 8x7B Instruct 32k',
context: 32768,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 24,
completion_tokens: 24,
cached_tokens: 0,
},
max_tokens: 32768,
},
{
id: 'llama-guard-3-8b',
puterId: 'groq:allam-2-7b',
id: 'allam-2-7b',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: false,
release_date: '2024-07-23',
name: 'Llama Guard 3 8B 8k',
context: 8192,
release_date: '2025-01-23',
name: 'ALLaM 2 7B',
aliases: ['allam-2-7b'],
context: 4096,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 0,
completion_tokens: 0,
cached_tokens: 0,
},
max_tokens: 4096,
},
{
puterId: 'groq:meta-llama/llama-guard-4-12b',
id: 'meta-llama/llama-guard-4-12b',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: true,
tool_call: false,
release_date: '2025-04-05',
name: 'Llama Guard 4 12B',
aliases: ['meta-llama/llama-guard-4-12b'],
context: 131072,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
@@ -313,6 +271,6 @@ export const GROQ_MODELS: IChatModel[] = [
completion_tokens: 20,
cached_tokens: 0,
},
max_tokens: 8192,
max_tokens: 1024,
},
];
@@ -0,0 +1,77 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Integration test for the Meta Model API (Muse Spark) provider.
*
* Muse Spark always reasons, and those tokens come out of the same output
* budget, so a tight `max_tokens` comes back as `content: null` with
* `finish_reason: 'length'` — hence the generous cap on a one-word prompt.
* Skipped when `PUTER_TEST_AI_META_API_KEY` is unset.
*/
import { describe, expect, it } from 'vitest';
import type { FSService } from '../../../../services/fs/FSService.js';
import type { FSEntryStore } from '../../../../stores/fs/FSEntryStore.js';
import type { S3ObjectStore } from '../../../../stores/fs/S3ObjectStore.js';
import {
INTEGRATION_TEST_TIMEOUT_MS,
makeMeteringStub,
optionalEnv,
skipUnlessEnv,
withTestActor,
} from '../../../integrationTestUtil.js';
import { MetaProvider } from './MetaProvider.js';
const ENV_VAR = 'PUTER_TEST_AI_META_API_KEY';
// Only `puter_path` content parts reach the filesystem, and this test sends
// plain text, so the stores stay untouched.
const UNUSED_STORES = {} as { fsEntry: FSEntryStore; s3Object: S3ObjectStore };
const UNUSED_FS_SERVICE = {} as FSService;
describe.skipIf(skipUnlessEnv(ENV_VAR))('MetaProvider (integration)', () => {
it(
'returns a non-empty completion from muse-spark-1.2',
{ timeout: INTEGRATION_TEST_TIMEOUT_MS },
async () => {
const provider = new MetaProvider(
makeMeteringStub(),
UNUSED_STORES,
UNUSED_FS_SERVICE,
{ apiKey: optionalEnv(ENV_VAR)! },
);
const result = await withTestActor(() =>
provider.complete({
model: 'muse-spark-1.2',
messages: [
{ role: 'user', content: 'Say hi in one word.' },
],
max_tokens: 2048,
reasoning_effort: 'low',
}),
);
const text = (result as { message?: { content?: string } }).message
?.content;
expect(typeof text === 'string' && text.length > 0).toBe(true);
},
);
});
@@ -0,0 +1,744 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Offline unit tests for MetaProvider.
*
* Boots a real PuterServer (in-memory sqlite + dynamo + s3 + mock redis) and
* constructs MetaProvider against the live wired `MeteringService` so the
* recording side is exercised end-to-end. The OpenAI SDK is mocked at the
* module boundary — Meta's Model API is OpenAI-compatible, so the provider
* reaches it through the same client — meaning nothing leaves the process. The
* companion integration test (MetaProvider.integration.test.ts) exercises the
* real api.meta.ai endpoint.
*/
import { Writable } from 'node:stream';
import {
afterAll,
afterEach,
beforeAll,
beforeEach,
describe,
expect,
it,
vi,
type MockInstance,
} from 'vitest';
import type { Actor } from '../../../../core/actor.js';
import { SYSTEM_ACTOR } from '../../../../core/actor.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import { PuterServer } from '../../../../server.js';
import { setupTestServer } from '../../../../testUtil.js';
import { withTestActor } from '../../../integrationTestUtil.js';
import { AIChatStream } from '../../utils/Streaming.js';
import { MetaProvider } from './MetaProvider.js';
import { META_MODELS } from './models.js';
// -- OpenAI SDK mock ----------------------------------------------
//
// `vi.hoisted` shares spies between the hoisted factory and the test body so
// each test can stub `chat.completions.create` with the response shape it
// cares about.
const { createMock, openAICtor } = vi.hoisted(() => {
const createMock = vi.fn();
const openAICtor = vi.fn();
return { createMock, openAICtor };
});
vi.mock('openai', () => {
const OpenAICtor = vi.fn().mockImplementation(function (
this: Record<string, unknown>,
opts: unknown,
) {
openAICtor(opts);
this.chat = { completions: { create: createMock } };
});
// The test server boots every provider, and some (e.g. OllamaChatProvider)
// import the default export and read `.OpenAI` off it — expose both shapes.
return { OpenAI: OpenAICtor, default: { OpenAI: OpenAICtor } };
});
// -- Test harness -------------------------------------------------
let server: PuterServer;
let recordSpy: MockInstance<MeteringService['utilRecordUsageObject']>;
beforeAll(async () => {
server = await setupTestServer();
});
afterAll(async () => {
await server?.shutdown();
});
const makeProvider = (config: { apiKey?: string; apiBaseUrl?: string } = {}) =>
new MetaProvider(
server.services.metering,
{
fsEntry: server.stores.fsEntry,
s3Object: server.stores.s3Object,
},
server.services.fs,
{
apiKey: config.apiKey ?? 'test-key',
...(config.apiBaseUrl ? { apiBaseUrl: config.apiBaseUrl } : {}),
},
);
const asAsyncIterable = <T>(items: T[]): AsyncIterable<T> => ({
async *[Symbol.asyncIterator]() {
for (const item of items) {
yield item;
}
},
});
const makeCapturingChatStream = () => {
const chunks: string[] = [];
const sink = new Writable({
write(chunk, _enc, cb) {
chunks.push(chunk.toString('utf8'));
cb();
},
});
const chatStream = new AIChatStream({ stream: sink });
return {
chatStream,
events: () =>
chunks
.join('')
.split('\n')
.filter(Boolean)
.map((line) => JSON.parse(line)),
};
};
const OK_COMPLETION = {
choices: [
{
message: { content: 'hi', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 1, completion_tokens: 1 },
};
const complete = (
provider: MetaProvider,
params: Record<string, unknown> = {},
actor?: Actor,
) =>
withTestActor(
() =>
provider.complete({
model: 'muse-spark-1.2',
messages: [{ role: 'user', content: 'hi' }],
...params,
} as never),
actor,
);
beforeEach(() => {
createMock.mockReset();
openAICtor.mockReset();
// Spy on the live MeteringService without replacing the impl, so the
// recording side still runs while per-test assertions see the calls.
recordSpy = vi.spyOn(server.services.metering, 'utilRecordUsageObject');
});
afterEach(() => {
vi.restoreAllMocks();
});
// -- Construction -------------------------------------------------
describe('MetaProvider construction', () => {
it('points the OpenAI SDK at the Meta Model API with the configured key', () => {
makeProvider();
expect(openAICtor).toHaveBeenCalledTimes(1);
expect(openAICtor).toHaveBeenCalledWith({
apiKey: 'test-key',
baseURL: 'https://api.meta.ai/v1',
});
});
it('honours a custom apiBaseUrl override', () => {
makeProvider({ apiBaseUrl: 'https://staging.meta.test/v1' });
expect(openAICtor).toHaveBeenCalledWith({
apiKey: 'test-key',
baseURL: 'https://staging.meta.test/v1',
});
});
});
// -- Model catalog ------------------------------------------------
describe('MetaProvider model catalog', () => {
it('returns muse-spark-1.2 as the default', () => {
expect(makeProvider().getDefaultModel()).toBe('muse-spark-1.2');
});
it('exposes the static META_MODELS list verbatim from models()', () => {
expect(makeProvider().models()).toBe(META_MODELS);
});
it('list() flattens canonical ids and aliases', () => {
const names = makeProvider().list();
for (const m of META_MODELS) {
expect(names).toContain(m.id);
for (const a of m.aliases ?? []) {
expect(names).toContain(a);
}
}
expect(names).toContain('meta/muse-spark-1.2');
});
it('leaves the contributor tier out of the catalog entirely', () => {
// It is the cheapest input rate Meta sells, so any name pointing at it
// would win bucket routing — and Meta trains on what it serves. It is
// also gated behind a separate enrolment, so a standard-tier key gets
// `model_not_found` for it.
const names = new Set(makeProvider().list());
for (const name of names) {
expect(name).not.toContain('contributor');
}
});
});
// -- Request shape ------------------------------------------------
describe('MetaProvider.complete request shape', () => {
it('sends model and messages without optional knobs', async () => {
createMock.mockResolvedValueOnce(OK_COMPLETION);
await complete(makeProvider());
const [args] = createMock.mock.calls[0]!;
expect(args.model).toBe('muse-spark-1.2');
expect(args.messages).toEqual([{ role: 'user', content: 'hi' }]);
for (const key of [
'max_completion_tokens',
'temperature',
'top_p',
'tools',
'tool_choice',
'reasoning_effort',
'prompt_cache_key',
'prompt_cache_retention',
'response_format',
'seed',
]) {
expect(key in args).toBe(false);
}
});
it('caps output with max_completion_tokens, not the legacy max_tokens', async () => {
createMock.mockResolvedValueOnce(OK_COMPLETION);
await complete(makeProvider(), { max_tokens: 512 });
const [args] = createMock.mock.calls[0]!;
expect(args.max_completion_tokens).toBe(512);
expect('max_tokens' in args).toBe(false);
});
it('forwards temperature, top_p, tools, and tool_choice when supplied', async () => {
createMock.mockResolvedValueOnce(OK_COMPLETION);
const tools = [
{
type: 'function',
function: {
name: 'lookup',
parameters: {
type: 'object',
properties: { q: { type: 'string' } },
},
},
},
];
await complete(makeProvider(), {
temperature: 0.4,
top_p: 0.9,
tools,
tool_choice: 'auto',
});
const [args] = createMock.mock.calls[0]!;
expect(args.temperature).toBe(0.4);
expect(args.top_p).toBe(0.9);
expect(args.tools).toBe(tools);
expect(args.tool_choice).toBe('auto');
});
it('forwards reasoning_effort, including the Meta-only tiers', async () => {
createMock.mockResolvedValueOnce(OK_COMPLETION);
await complete(makeProvider(), { reasoning_effort: 'xhigh' });
expect(createMock.mock.calls[0]![0].reasoning_effort).toBe('xhigh');
});
it('falls back to reasoning.effort when reasoning_effort is absent', async () => {
createMock.mockResolvedValueOnce(OK_COMPLETION);
await complete(makeProvider(), { reasoning: { effort: 'low' } });
expect(createMock.mock.calls[0]![0].reasoning_effort).toBe('low');
});
it('drops reasoning_effort: none, which Muse Spark rejects with a 400', async () => {
createMock.mockResolvedValueOnce(OK_COMPLETION);
await complete(makeProvider(), { reasoning_effort: 'none' });
expect('reasoning_effort' in createMock.mock.calls[0]![0]).toBe(false);
});
it("translates the hyphenated in-memory cache retention to Meta's enum", async () => {
const provider = makeProvider();
createMock.mockResolvedValueOnce(OK_COMPLETION);
await complete(provider, {
prompt_cache_key: 'cache-abc',
prompt_cache_retention: 'in-memory',
});
// Meta rejects the hyphenated spelling outright: "unknown variant
// `in-memory`, expected `in_memory` or `24h`".
const [args] = createMock.mock.calls[0]!;
expect(args.prompt_cache_key).toBe('cache-abc');
expect(args.prompt_cache_retention).toBe('in_memory');
createMock.mockResolvedValueOnce(OK_COMPLETION);
await complete(provider, { prompt_cache_retention: '24h' });
expect(createMock.mock.calls[1]![0].prompt_cache_retention).toBe('24h');
});
it('forwards Muse-Spark-specific custom params', async () => {
createMock.mockResolvedValueOnce(OK_COMPLETION);
await complete(makeProvider(), {
custom: {
seed: 7,
response_format: { type: 'json_object' },
frequency_penalty: 0.5,
presence_penalty: -0.5,
},
});
const [args] = createMock.mock.calls[0]!;
expect(args.seed).toBe(7);
expect(args.response_format).toEqual({ type: 'json_object' });
expect(args.frequency_penalty).toBe(0.5);
expect(args.presence_penalty).toBe(-0.5);
});
it('strips Anthropic-style cache_control from messages before sending', async () => {
createMock.mockResolvedValueOnce(OK_COMPLETION);
await complete(makeProvider(), {
messages: [
{
role: 'user',
content: 'hi',
cache_control: { type: 'ephemeral' },
},
],
});
expect(
'cache_control' in createMock.mock.calls[0]![0].messages[0],
).toBe(false);
});
it('derives safety_identifier from the actor and truncates it to 64 chars', async () => {
createMock.mockResolvedValueOnce(OK_COMPLETION);
const userActor: Actor = {
user: { id: 42, uuid: 'u42', username: 'alice' },
app: { id: 7, uid: 'a'.repeat(80) },
};
await complete(makeProvider(), {}, userActor);
const identifier = createMock.mock.calls[0]![0].safety_identifier;
expect(identifier.startsWith('puter-42-a')).toBe(true);
expect(identifier.length).toBe(64);
});
it('prefers an explicit custom.safety_identifier over the actor-derived one', async () => {
createMock.mockResolvedValueOnce(OK_COMPLETION);
const userActor: Actor = { user: { id: 42, uuid: 'u42' } };
await complete(
makeProvider(),
{ custom: { safety_identifier: 'caller-supplied' } },
userActor,
);
expect(createMock.mock.calls[0]![0].safety_identifier).toBe(
'caller-supplied',
);
});
it('omits safety_identifier for the system actor (no user.id)', async () => {
createMock.mockResolvedValueOnce(OK_COMPLETION);
await complete(makeProvider());
expect('safety_identifier' in createMock.mock.calls[0]![0]).toBe(false);
});
it('only sets stream_options.include_usage when streaming', async () => {
const provider = makeProvider();
createMock.mockResolvedValueOnce(OK_COMPLETION);
await complete(provider, { stream: false });
const [nonStreamArgs] = createMock.mock.calls[0]!;
expect(nonStreamArgs.stream).toBe(false);
expect('stream_options' in nonStreamArgs).toBe(false);
createMock.mockReturnValueOnce(asAsyncIterable([]));
await complete(provider, { stream: true });
const [streamArgs] = createMock.mock.calls[1]!;
expect(streamArgs.stream).toBe(true);
expect(streamArgs.stream_options).toEqual({ include_usage: true });
});
});
// -- Model resolution ---------------------------------------------
describe('MetaProvider model resolution', () => {
it('resolves an alias to its canonical id', async () => {
createMock.mockResolvedValueOnce(OK_COMPLETION);
await complete(makeProvider(), { model: 'meta/muse-spark-1.1' });
expect(createMock.mock.calls[0]![0].model).toBe('muse-spark-1.1');
expect(recordSpy).toHaveBeenCalledWith(
expect.any(Object),
expect.anything(),
'meta:muse-spark-1.1',
expect.any(Object),
);
});
it('falls back to the default model when given an unknown id', async () => {
createMock.mockResolvedValueOnce(OK_COMPLETION);
await complete(makeProvider(), { model: 'totally-not-a-real-model' });
expect(createMock.mock.calls[0]![0].model).toBe('muse-spark-1.2');
expect(recordSpy).toHaveBeenCalledWith(
expect.any(Object),
expect.anything(),
'meta:muse-spark-1.2',
expect.any(Object),
);
});
});
// -- Non-stream output --------------------------------------------
describe('MetaProvider.complete non-stream output', () => {
it('bills cache reads separately from the remaining prompt tokens', async () => {
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'hi there', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: {
prompt_tokens: 100,
completion_tokens: 50,
prompt_tokens_details: { cached_tokens: 40 },
},
});
const result = await complete(makeProvider());
expect(result).toMatchObject({
message: { content: 'hi there', role: 'assistant' },
finish_reason: 'stop',
});
// Meta reports cache reads inside prompt_tokens; only the uncached
// remainder is charged at the input rate.
expect((result as { usage: unknown }).usage).toEqual({
prompt_tokens: 60,
completion_tokens: 50,
cached_tokens: 40,
});
const model = META_MODELS.find((m) => m.id === 'muse-spark-1.2')!;
expect(recordSpy).toHaveBeenCalledTimes(1);
const [usage, actor, prefix, overrides] = recordSpy.mock.calls[0]!;
expect(usage).toEqual({
prompt_tokens: 60,
completion_tokens: 50,
cached_tokens: 40,
});
expect(actor).toBe(SYSTEM_ACTOR);
expect(prefix).toBe('meta:muse-spark-1.2');
expect(overrides!.prompt_tokens).toBeCloseTo(
60 * Number(model.costs.prompt_tokens),
5,
);
expect(overrides!.completion_tokens).toBeCloseTo(
50 * Number(model.costs.completion_tokens),
5,
);
expect(overrides!.cached_tokens).toBeCloseTo(
40 * Number(model.costs.cached_tokens),
5,
);
});
it('zeroes cached_tokens when prompt_tokens_details is missing', async () => {
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'ok', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 7, completion_tokens: 3 },
});
await complete(makeProvider());
const [usage, , , overrides] = recordSpy.mock.calls[0]!;
expect(usage).toEqual({
prompt_tokens: 7,
completion_tokens: 3,
cached_tokens: 0,
});
expect(overrides).toMatchObject({ cached_tokens: 0 });
});
it('preserves OpenAI-shaped tool_calls on the assistant response', async () => {
createMock.mockResolvedValueOnce({
choices: [
{
message: {
role: 'assistant',
content: null,
tool_calls: [
{
id: 'call_1',
type: 'function',
function: {
name: 'lookup',
arguments: '{"q":"puter"}',
},
},
],
},
finish_reason: 'tool_calls',
},
],
usage: { prompt_tokens: 1, completion_tokens: 1 },
});
const result = (await complete(makeProvider(), {
tools: [
{
type: 'function',
function: { name: 'lookup', parameters: {} },
},
],
})) as { message: { tool_calls?: unknown[] }; finish_reason: string };
expect(result.finish_reason).toBe('tool_calls');
expect(result.message.tool_calls).toEqual([
{
id: 'call_1',
type: 'function',
function: { name: 'lookup', arguments: '{"q":"puter"}' },
},
]);
});
it('leaves reasoning token counts out of the metered usage', async () => {
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'done', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: {
prompt_tokens: 8,
completion_tokens: 100,
completion_tokens_details: { reasoning_tokens: 90 },
},
});
await complete(makeProvider());
// Muse Spark always reasons, and counts those tokens inside
// `completion_tokens` — billing them again would double-charge.
const [usage] = recordSpy.mock.calls[0]!;
expect(usage).toEqual({
prompt_tokens: 8,
completion_tokens: 100,
cached_tokens: 0,
});
});
});
// -- Streaming ----------------------------------------------------
describe('MetaProvider.complete streaming', () => {
it('streams text deltas and meters the final usage chunk', async () => {
createMock.mockReturnValueOnce(
asAsyncIterable([
{ choices: [{ delta: { content: 'hel' } }] },
{ choices: [{ delta: { content: 'lo' } }] },
{
choices: [{ delta: {} }],
usage: {
prompt_tokens: 10,
completion_tokens: 2,
prompt_tokens_details: { cached_tokens: 4 },
},
},
]),
);
const result = await complete(makeProvider(), { stream: true });
expect((result as { stream: boolean }).stream).toBe(true);
const harness = makeCapturingChatStream();
await (
result as {
init_chat_stream: (p: { chatStream: unknown }) => Promise<void>;
}
).init_chat_stream({ chatStream: harness.chatStream });
const events = harness.events();
expect(
events.filter((e) => e.type === 'text').map((e) => e.text),
).toEqual(['hel', 'lo']);
expect(events.find((e) => e.type === 'usage')?.usage).toEqual({
prompt_tokens: 6,
completion_tokens: 2,
cached_tokens: 4,
});
const model = META_MODELS.find((m) => m.id === 'muse-spark-1.2')!;
expect(recordSpy).toHaveBeenCalledTimes(1);
const [, , prefix, overrides] = recordSpy.mock.calls[0]!;
expect(prefix).toBe('meta:muse-spark-1.2');
expect(overrides!.prompt_tokens).toBeCloseTo(
6 * Number(model.costs.prompt_tokens),
5,
);
expect(overrides!.cached_tokens).toBeCloseTo(
4 * Number(model.costs.cached_tokens),
5,
);
});
it('builds a tool_use event from streamed function-call deltas', async () => {
createMock.mockReturnValueOnce(
asAsyncIterable([
{
choices: [
{
delta: {
tool_calls: [
{
index: 0,
id: 'call_1',
function: {
name: 'lookup',
arguments: '{"q":',
},
},
],
},
},
],
},
{
choices: [
{
delta: {
tool_calls: [
{
index: 0,
function: { arguments: '"puter"}' },
},
],
},
},
],
},
{
choices: [{ delta: {} }],
usage: { prompt_tokens: 1, completion_tokens: 1 },
},
]),
);
const result = await complete(makeProvider(), {
stream: true,
tools: [
{
type: 'function',
function: { name: 'lookup', parameters: {} },
},
],
});
const harness = makeCapturingChatStream();
await (
result as {
init_chat_stream: (p: { chatStream: unknown }) => Promise<void>;
}
).init_chat_stream({ chatStream: harness.chatStream });
const toolEvent = harness.events().find((e) => e.type === 'tool_use');
expect(toolEvent).toBeDefined();
expect(toolEvent?.id).toBe('call_1');
expect(toolEvent?.name).toBe('lookup');
expect(toolEvent?.input).toEqual({ q: 'puter' });
});
});
// -- Error mapping ------------------------------------------------
describe('MetaProvider.complete error mapping', () => {
it('rethrows errors raised by the OpenAI client unchanged', async () => {
const apiError = new Error('Model API exploded');
createMock.mockRejectedValueOnce(apiError);
await expect(complete(makeProvider())).rejects.toBe(apiError);
// A failed call must not be metered.
expect(recordSpy).not.toHaveBeenCalled();
});
it('rejects a missing messages payload with a 400', async () => {
await expect(
withTestActor(() =>
makeProvider().complete({
model: 'muse-spark-1.2',
} as never),
),
).rejects.toMatchObject({ statusCode: 400 });
expect(createMock).not.toHaveBeenCalled();
});
});
// -- Moderation ---------------------------------------------------
describe('MetaProvider.checkModeration', () => {
it('throws — the Meta provider does not implement moderation', () => {
expect(() => makeProvider().checkModeration('anything')).toThrow(
/not implemented/i,
);
});
});
@@ -0,0 +1,260 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { OpenAI } from 'openai';
import { ChatCompletionCreateParams } from 'openai/resources/index.js';
import { Context } from '../../../../core/context.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import type { FSService } from '../../../../services/fs/FSService.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import type { FSEntryStore } from '../../../../stores/fs/FSEntryStore.js';
import type { S3ObjectStore } from '../../../../stores/fs/S3ObjectStore.js';
import type { IChatProvider, ICompleteArguments } from '../../types.js';
import * as OpenAIUtil from '../../utils/OpenAIUtil.js';
import { buildCostsOverride } from '../../utils/pricing.js';
import { processPuterPathUploads } from '../openai/fileUpload.js';
import { META_MODELS, MUSE_SPARK_DEFAULT_MODEL } from './models.js';
const DEFAULT_API_BASE_URL = 'https://api.meta.ai/v1';
// `safety_identifier` is capped at 64 characters by the Model API.
const SAFETY_IDENTIFIER_MAX_LENGTH = 64;
type MetaConfig = {
apiBaseUrl?: string;
apiKey: string;
};
/**
* Chat Completions params Muse Spark accepts that Puter has no first-class
* argument for. Passed through `custom`.
*/
type MetaCustomParams = {
frequency_penalty?: number;
presence_penalty?: number;
response_format?: unknown;
safety_identifier?: string;
seed?: number;
};
const asRecord = (value: unknown): Record<string, unknown> =>
value && typeof value === 'object' && !Array.isArray(value)
? (value as Record<string, unknown>)
: {};
/**
* Meta's Model API — the Muse Spark family, served OpenAI-compatible from
* `https://api.meta.ai/v1`.
*
* Only the Chat Completions protocol is used here; Meta also fronts the same
* models behind Responses- and Anthropic-Messages-shaped endpoints.
*/
export class MetaProvider implements IChatProvider {
#openai: OpenAI;
#meteringService: MeteringService;
#stores: { fsEntry: FSEntryStore; s3Object: S3ObjectStore };
#fsService: FSService;
constructor(
meteringService: MeteringService,
stores: { fsEntry: FSEntryStore; s3Object: S3ObjectStore },
fsService: FSService,
config: MetaConfig,
) {
this.#openai = new OpenAI({
apiKey: config.apiKey,
baseURL: config.apiBaseUrl ?? DEFAULT_API_BASE_URL,
});
this.#meteringService = meteringService;
this.#stores = stores;
this.#fsService = fsService;
}
getDefaultModel() {
return MUSE_SPARK_DEFAULT_MODEL;
}
models() {
return META_MODELS;
}
list() {
const modelIds: string[] = [];
for (const model of this.models()) {
modelIds.push(model.id);
if (model.aliases) {
modelIds.push(...model.aliases);
}
}
return modelIds;
}
async complete(
params: ICompleteArguments,
): ReturnType<IChatProvider['complete']> {
const {
custom,
max_tokens,
prompt_cache_key,
prompt_cache_retention,
reasoning,
reasoning_effort,
stream,
temperature,
tool_choice,
tools,
top_p,
} = params;
let { messages, model } = params;
if (!Array.isArray(messages)) {
throw new HttpError(400, '`messages` must be an array', {
legacyCode: 'bad_request',
});
}
const actor = Context.get('actor');
const availableModels = this.models();
const modelUsed =
availableModels.find((m) =>
[m.id, ...(m.aliases || [])].includes(model),
) || availableModels.find((m) => m.id === this.getDefaultModel())!;
// Muse Spark reads images, video, PDFs and audio, but Chat Completions
// takes them inline only — resolve `puter_path` parts to data URLs.
await processPuterPathUploads(
messages,
this.#stores,
this.#fsService,
actor,
);
messages = await OpenAIUtil.process_input_messages(messages);
messages = messages.map((message) => {
// Anthropic-shaped cache hints don't belong on this wire; Meta
// caches via `prompt_cache_key` / `prompt_cache_retention`.
delete message.cache_control;
return message;
});
const customParams = asRecord(custom) as MetaCustomParams;
// Reasoning is always on for Muse Spark — `reasoning_effort: 'none'`
// is a 400 — so a request to switch it off is dropped, not forwarded.
const requestedEffort = (reasoning_effort ?? reasoning?.effort) as
string | undefined;
const effort =
requestedEffort && requestedEffort !== 'none'
? requestedEffort
: undefined;
// Puter spells the in-memory retention with a hyphen; Meta's enum
// uses an underscore.
const cacheRetention =
prompt_cache_retention === 'in-memory'
? 'in_memory'
: prompt_cache_retention;
const safetyIdentifier =
customParams.safety_identifier ??
(actor?.user?.id
? `puter-${actor.user.id}${actor.app?.uid ? `-${actor.app.uid}` : ''}`.slice(
0,
SAFETY_IDENTIFIER_MAX_LENGTH,
)
: undefined);
const completionParams = {
messages,
model: modelUsed.id,
...(tools ? { tools } : {}),
...(tool_choice !== undefined ? { tool_choice } : {}),
// Reasoning tokens come out of this same budget, so a tight cap
// returns `content: null` with `finish_reason: 'length'`.
...(max_tokens !== undefined
? { max_completion_tokens: max_tokens }
: {}),
...(temperature !== undefined ? { temperature } : {}),
...(top_p !== undefined ? { top_p } : {}),
...(effort ? { reasoning_effort: effort } : {}),
...(prompt_cache_key !== undefined ? { prompt_cache_key } : {}),
...(cacheRetention !== undefined
? { prompt_cache_retention: cacheRetention }
: {}),
...(safetyIdentifier
? { safety_identifier: safetyIdentifier }
: {}),
...(customParams.response_format
? { response_format: customParams.response_format }
: {}),
...(customParams.frequency_penalty !== undefined
? { frequency_penalty: customParams.frequency_penalty }
: {}),
...(customParams.presence_penalty !== undefined
? { presence_penalty: customParams.presence_penalty }
: {}),
...(customParams.seed !== undefined
? { seed: customParams.seed }
: {}),
stream: !!stream,
...(stream ? { stream_options: { include_usage: true } } : {}),
} as ChatCompletionCreateParams;
const completion =
await this.#openai.chat.completions.create(completionParams);
return OpenAIUtil.handle_completion_output({
usage_calculator: ({ usage }) => {
const cachedTokens =
usage?.prompt_tokens_details?.cached_tokens ?? 0;
// Meta reports cache reads as a subset of `prompt_tokens`, so
// the remainder is what the input rate applies to. Reasoning
// tokens (`completion_tokens_details.reasoning_tokens`) are
// likewise already inside `completion_tokens` — metering them
// again would bill the same tokens twice.
const trackedUsage = {
prompt_tokens: (usage?.prompt_tokens ?? 0) - cachedTokens,
completion_tokens: usage?.completion_tokens ?? 0,
cached_tokens: cachedTokens,
};
const costsOverride = buildCostsOverride(
trackedUsage,
modelUsed,
);
this.#meteringService.utilRecordUsageObject(
trackedUsage,
actor!,
`meta:${modelUsed.id}`,
costsOverride,
);
return trackedUsage;
},
stream,
completion,
});
}
checkModeration(
_text: string,
): ReturnType<IChatProvider['checkModeration']> {
throw new Error('Method not implemented.');
}
}
@@ -0,0 +1,78 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { IChatModel } from '../../types.js';
import { usdPerMToken } from '../../utils/pricing.js';
export const MUSE_SPARK_DEFAULT_MODEL = 'muse-spark-1.2';
const museSpark = (model: {
id: string;
name: string;
context: number;
maxTokens: number;
releaseDate: string;
inputModalities: string[];
costs: IChatModel['costs'];
}): IChatModel => ({
puterId: `meta:meta/${model.id}`,
id: model.id,
name: model.name,
aliases: [`meta/${model.id}`],
modalities: { input: model.inputModalities, output: ['text'] },
open_weights: false,
tool_call: true,
release_date: model.releaseDate,
context: model.context,
max_tokens: model.maxTokens,
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: model.costs,
});
// Hardcoded from https://models.dev/api.json and
// https://dev.meta.ai/docs/pricing-rate-limits.
//
// Meta also sells `muse-spark-1.2-contributor`: the same checkpoint at a tenth
// of the price in exchange for permission to train on the prompts and
// completions it serves. It stays out of this catalog — it needs a separate
// enrolment (a standard-tier key gets `model_not_found`), and its input rate
// would make it the cheapest route in the bucket, so listing it would quietly
// route callers' prompts into Meta's training set.
export const META_MODELS: IChatModel[] = [
museSpark({
id: 'muse-spark-1.2',
name: 'Muse Spark 1.2',
context: 1_048_576,
maxTokens: 131_072,
releaseDate: '2026-08-05',
inputModalities: ['text', 'image', 'video', 'audio', 'pdf'],
costs: usdPerMToken(1.25, 4.25, 0.15),
}),
museSpark({
id: 'muse-spark-1.1',
name: 'Muse Spark 1.1',
context: 1_048_576,
maxTokens: 32_000,
releaseDate: '2026-04-08',
inputModalities: ['text', 'image', 'video', 'pdf'],
costs: usdPerMToken(1.25, 4.25, 0.15),
}),
];
@@ -22,8 +22,8 @@ import type { IChatModel } from '../../types.js';
// Hardcoded from https://models.dev/api.json and https://docs.mistral.ai/models/overview
export const MISTRAL_MODELS: IChatModel[] = [
{
puterId: 'mistralai:mistralai/mistral-medium-3-5',
id: 'mistral-medium-3-5',
puterId: 'mistralai:mistralai/mistral-medium-2604',
id: 'mistral-medium-2604',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: true,
tool_call: true,
@@ -31,8 +31,10 @@ export const MISTRAL_MODELS: IChatModel[] = [
release_date: '2026-04-29',
name: 'Mistral Medium 3.5',
aliases: [
'mistral-medium-3-5',
'mistral-medium-3',
'mistral-medium-2604',
'mistral-medium-latest',
'mistral-medium',
'mistralai/mistral-medium-3-5',
'mistralai/mistral-medium-2604',
],
@@ -50,34 +52,6 @@ export const MISTRAL_MODELS: IChatModel[] = [
completion_tokens: 750,
},
},
{
puterId: 'mistralai:mistralai/mistral-medium-2508',
id: 'mistral-medium-2508',
modalities: { input: ['text', 'image'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-05',
release_date: '2025-08-12',
name: 'Mistral Medium 3.1',
aliases: [
'mistralai/mistral-medium-2508',
'mistral-medium-2508',
'mistral-medium-latest',
'mistralai/mistral-medium-latest',
],
context: 262_144,
max_tokens: 262_144,
description: 'Frontier-class multimodal model.',
provider: 'mistral',
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 40,
completion_tokens: 200,
},
},
{
puterId: 'mistralai:mistralai/mistral-large-2512',
id: 'mistral-large-2512',
@@ -161,68 +135,19 @@ export const MISTRAL_MODELS: IChatModel[] = [
},
},
{
puterId: 'mistralai:mistralai/devstral-2512',
id: 'devstral-2512',
puterId: 'mistralai:mistralai/magistral-small-latest',
id: 'magistral-small-latest',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: true,
knowledge: '2025-12',
release_date: '2025-12-09',
name: 'Devstral 2',
knowledge: '2025-06',
release_date: '2025-09-01',
name: 'Magistral Small',
aliases: [
'devstral-latest',
'mistralai/devstral-latest',
'mistralai/devstral-2512',
'devstral-medium-latest',
'mistralai/devstral-medium-latest',
'magistral-small-2509',
'mistralai/magistral-small-2509',
'mistralai/magistral-small-latest',
],
context: 262_144,
max_tokens: 262_144,
description:
'Frontier open-source code agents model for software engineering.',
provider: 'mistral',
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 40,
completion_tokens: 200,
},
},
{
puterId: 'mistralai:mistralai/magistral-medium-2509',
id: 'magistral-medium-2509',
modalities: { input: ['text'], output: ['text'] },
open_weights: false,
tool_call: true,
knowledge: '2025-06',
release_date: '2025-09-01',
name: 'Magistral Medium 1.2',
aliases: ['magistral-medium-latest', 'mistralai/magistral-medium-2509'],
context: 131_072,
max_tokens: 131_072,
description: 'Frontier-class multimodal reasoning model.',
provider: 'mistral',
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 200,
completion_tokens: 500,
},
},
{
puterId: 'mistralai:mistralai/magistral-small-2509',
id: 'magistral-small-2509',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: true,
knowledge: '2025-06',
release_date: '2025-09-01',
name: 'Magistral Small 1.2',
aliases: ['magistral-small-latest', 'mistralai/magistral-small-2509'],
context: 131_072,
max_tokens: 131_072,
description: 'Efficient open-weight reasoning model.',
@@ -307,36 +232,6 @@ export const MISTRAL_MODELS: IChatModel[] = [
completion_tokens: 4,
},
},
{
puterId: 'mistralai:mistralai/open-mistral-nemo-2407',
id: 'open-mistral-nemo-2407',
modalities: { input: ['text'], output: ['text'] },
open_weights: true,
tool_call: true,
knowledge: '2024-07',
release_date: '2024-07-25',
name: 'Mistral Nemo 12B',
aliases: [
'open-mistral-nemo',
'mistral-nemo',
'mistral-nemo-latest',
'mistralai/open-mistral-nemo',
'mistralai/open-mistral-nemo-2407',
'mistralai/mistral-nemo',
],
context: 128_000,
max_tokens: 128_000,
description: 'Best multilingual open source model.',
provider: 'mistral',
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 15,
completion_tokens: 15,
},
},
{
puterId: 'mistralai:mistralai/voxtral-small-2507',
id: 'voxtral-small-2507',
@@ -68,6 +68,51 @@ export const MOONSHOT_MODELS: IChatModel[] = [
knowledge: '2025-01',
},
// -- Kimi K2.7 Code ---------------------------------------------
{
puterId: 'moonshotai:moonshotai/kimi-k2.7-code',
id: 'kimi-k2.7-code',
name: 'Kimi K2.7 Code',
aliases: ['moonshotai/kimi-k2.7-code', 'moonshot/kimi-k2.7-code'],
modalities: { input: ['text', 'image', 'video'], output: ['text'] },
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 95,
completion_tokens: 400,
cached_tokens: 19,
},
context: 262_144,
max_tokens: 262_144,
tool_call: true,
release_date: '2026-06-12',
},
{
puterId: 'moonshotai:moonshotai/kimi-k2.7-code-highspeed',
id: 'kimi-k2.7-code-highspeed',
name: 'Kimi K2.7 Code Highspeed',
aliases: [
'moonshotai/kimi-k2.7-code-highspeed',
'moonshot/kimi-k2.7-code-highspeed',
],
modalities: { input: ['text', 'image', 'video'], output: ['text'] },
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
costs: {
tokens: 1_000_000,
prompt_tokens: 190,
completion_tokens: 800,
cached_tokens: 38,
},
context: 262_144,
max_tokens: 262_144,
tool_call: true,
release_date: '2026-06-12',
},
// -- Kimi K2.5 --------------------------------------------------
{
puterId: 'moonshotai:moonshotai/kimi-k2.5',
@@ -95,8 +95,7 @@ export class NeuralwattProvider implements IChatProvider {
async models(): Promise<IChatModel[]> {
let apiModels = kv.get(KV_MODELS_KEY) as
| NeuralwattApiModel[]
| undefined;
NeuralwattApiModel[] | undefined;
if (!apiModels) {
try {
const resp = await axios.request({
@@ -127,16 +126,15 @@ export class NeuralwattProvider implements IChatProvider {
}
/**
* Cached account accounting method (`energy` | `token`) from
* `GET /v1/quota`. Used only to annotate returned usage — billing
* always prefers `cost.request_cost_usd` on the completion.
* Cached account accounting method (`energy` | `token`) from `GET
* /v1/quota`. Used only to annotate returned usage — billing always prefers
* `cost.request_cost_usd` on the completion.
*/
async getAccountingMethod(): Promise<
NeuralwattAccountingMethod | undefined
> {
let method = kv.get(KV_QUOTA_KEY) as
| NeuralwattAccountingMethod
| undefined;
NeuralwattAccountingMethod | undefined;
if (method === 'energy' || method === 'token') return method;
try {
@@ -187,9 +185,7 @@ export class NeuralwattProvider implements IChatProvider {
if (!modelUsed) {
if (hasImages) {
modelUsed = availableModels.find((m) =>
modelSupportsVision(m),
);
modelUsed = availableModels.find((m) => modelSupportsVision(m));
}
modelUsed =
modelUsed ||
@@ -221,8 +217,7 @@ export class NeuralwattProvider implements IChatProvider {
messages = await OpenAIUtil.process_input_messages(messages);
const requestedReasoningEffort =
reasoning_effort ?? reasoning?.effort;
const requestedReasoningEffort = reasoning_effort ?? reasoning?.effort;
const supportsReasoningEffort = modelUsed.reasoning_effort === true;
const completionParams = {
@@ -353,8 +348,7 @@ export class NeuralwattProvider implements IChatProvider {
...chunk.usage,
...(typeof chunk.cost?.request_cost_usd === 'number'
? {
request_cost_usd:
chunk.cost.request_cost_usd,
request_cost_usd: chunk.cost.request_cost_usd,
}
: {}),
...(chunk.energy
Loaded 100 of 396 files, more files were not shown because too many files have changed in this diff. Show more