Compare commits

...
Author SHA1 Message Date
Nariman Jelveh 8bcfd0c620 fix(dashboard): temporary accounts can delete themselves again
The server reports oidc_only for every passwordless account, temporary
ones included, so the shared user-protected helper sent a temporary
account into the linked-provider sign-in, which has no URL for it, and the
deletion never went out. Temporary accounts post directly, as the route
admits them without credentials.
2026-10-22 16:59:08 -07:00
Nariman Jelveh df2c5978b1 fix(dashboard): phone dialog inputs stay 16px instead of jumping on focus
On phones style.css's .device-phone input rule outranked the dialog's
16px size at rest, so fields drew at 17px and shrank to 16px the moment
they took focus.
2026-10-22 16:56:57 -07:00
Nariman Jelveh 831fd421a0 fix(dashboard): dialog inputs keep a 1px border when focused
style.css's global input :focus rule still set a 2px border width, so a
dialog field's text shifted by a pixel each time it took focus. The focus
ring is the box-shadow; the border stays 1px.
2026-10-22 16:56:26 -07:00
Nariman Jelveh b33286b106 fix(dashboard): form dialog labels drop the global label shadow
style.css's bare `label` rule gave the field labels a white text-shadow,
which ghosted them on the dark theme, and an extra 3px under each label.
2026-10-22 16:55:01 -07:00
Nariman Jelveh 8e7c64ad55 fix(dashboard): requesting an email change no longer sets the cached email
The server only stages the new address until the emailed link is
followed, but the dialog wrote it into window.user.email straight away.
That cached a wrong address for the session and kept refresh_user_data from
updating the Account tab once the change was confirmed, since the two
already matched.
2026-10-22 16:53:35 -07:00
Nariman Jelveh bc59376884 fix(dashboard): screen readers hear a form dialog's success message
The done state arrives as a new status region that already holds its text,
which screen readers don't announce, so after a change succeeded they only
heard "Done, button". The Done button that takes focus is now described by
the message.
2026-10-22 16:52:16 -07:00
Nariman Jelveh c327ff6d69 fix(dashboard): password show/hide reads as one action, not a pressed toggle
The button flipped its label between Show and Hide password and also set
aria-pressed, so assistive tech announced "Hide password, pressed" while
the password was showing. It keeps the action label and drops aria-pressed.
2026-10-22 16:51:17 -07:00
Nariman Jelveh 992e24b943 fix(gui): a blocked re-sign-in popup fails instead of waiting forever
openRevalidatePopup polled a popup that was never opened, so when the
browser blocked it the account dialog stayed busy indefinitely. It now
rejects straight away and the dialog says the sign-in didn't finish.
2026-10-22 16:50:38 -07:00
Nariman Jelveh 987521ffd4 fix(dashboard): a form error keeps the field it names in view
The status line scrolled the dialog body to the bottom, which on a short
screen (a phone with its keyboard up) left the field being focused and
marked invalid scrolled off the top. The body now scrolls back up just
far enough to show that field.
2026-10-22 16:49:35 -07:00
Nariman Jelveh 06401516bf fix(dashboard): a failed form submit keeps focus in the dialog
Locking the fields disabled the focused control, which drops focus to the
page, so after an error Enter no longer retried and screen readers lost
their place. The control that had focus gets it back once the fields
unlock, unless the handler moved focus elsewhere in the dialog.
2026-10-22 16:48:51 -07:00
Nariman Jelveh 15cbb041dc fix(dashboard): a double-click that opens a dialog no longer dismisses it
The second press of the double-click lands on the freshly mounted backdrop
and closed the dialog straight away. Backdrop presses with a click count
above one no longer count toward dismissal.
2026-10-22 16:46:55 -07:00
Nariman Jelveh 337006d99c feat(dashboard): account dialogs as dashboard modals instead of windows
Edit profile, change username, change password, change email, create API
token, and delete account in the Account tab (and change password in the
Security tab) now open as dashboard overlay cards, like the share,
properties, and confirm dialogs, instead of desktop UIWindows. They are
centered cards on desktop and tablet and bottom sheets on phones.

- dashboardOverlay.js: the overlay shell UIDashboardDialog already had
  (reveal, Esc/backdrop/close dismissal, focus trap, focus return), pulled
  out so the new form dialog shares it. It also pins the overlay to the
  visual viewport so iOS's keyboard can't cover a sheet's fields, and
  leaves Esc to a modal opened on top of it.
- UIDashboardFormDialog.js: labelled fields, password show/hide, an
  aria-live status line that marks and focuses the offending field, a busy
  submit button, and a done state.
- postUserProtected.js: the cookie-retry / provider re-sign-in / reauth
  flow the four account windows each carried, in one place.
- Deleting an account is one dialog now (warning plus confirmation) rather
  than two stacked windows.
- Touch: 44px controls, 16px inputs (no iOS focus zoom), no autofocus
  that pops the keyboard mid-animation, form buttons side by side on
  phones, safe-area padding dropped while the keyboard is up.
- The replaced windows had no other callers and are removed with their
  tests, which are ported to the new modules.
2026-10-22 16:32:32 -07:00
Daniel Salazar 80a9cd85ee fix(ai): chat requests run to their upstream usage report after the caller disconnects (#4302)
- Providers send upstream calls with a driver-owned `upstreamSignal`
  instead of the caller's `abortSignal`. A disconnect detaches the chat
  stream: output stops, the provider keeps reading until the upstream
  reports usage, and that usage is metered.
- A detached stream gets SDK_TIMEOUT_MS to finish. After that the driver
  aborts the upstream, and the stream is metered like any other stream
  that ends without usage.
- The disconnect check also runs right before dispatch.
- The OpenAI stream handlers report usage that arrived before an abort.
- Docs: stopping a stream stops the output, and a request that reached
  the model is billed in full. Detached streams get 10 more minutes.
2026-10-22 16:08:05 -07:00
Nariman Jelveh 63370fcfa5 fix(dashboard): mobile and tablet review — layout, touch, safe areas (#4303)
* fix(dashboard): give narrow Files listings the two-row layout

The two-row rows (name, then size and date) were keyed to a phone-width
viewport, so between 481px and ~930px, where the sidebar and folder column
take their share, the fixed size and modified columns squeezed names to a
few characters. The layout now switches on the listing's own width with a
container query, which phones still meet and wide listings never do.

* fix(dashboard): wrap the Files toolbar onto two rows in narrow listings

On tablets the nav buttons, crumbs and actions sat on one row wider than
the listing, so the sort, view, new-folder and upload buttons ran off the
right edge. Narrow listings now use the phone's two-row toolbar (crumbs
above, buttons below); only the menu-button clearance stays phone-only.

* fix(dashboard): keep a deep folder path on one row with the current folder in view

A long path wrapped the crumbs over several lines on top of the column
headers and pushed the toolbar actions off-screen; where the crumbs already
scrolled (phones), the row opened at the root, hiding the current folder.
The crumbs now scroll within their row and are scrolled to their end on
render and when the tab is shown.

* fix(dashboard): end clipped file names in the list with an ellipsis

With the name column auto-sized, which is always the case on phones and
tablets, a long name was cut off mid-letter with no sign it continued.

* fix(dashboard): keep the Files selection bar under the drawer and sheets

The bar's visible state set z-index 99999999999999999 (clamped to the
maximum), so in phone select mode it drew over the open navigation drawer,
covering the account button, and over the share sheet it opens. Its base
z-index already clears the rows and the footer.

* fix(dashboard): let the last Files rows scroll clear of the phone selection bar

In phone select mode the list's bottom padding only cleared the footer, so
the last row stayed behind the floating action bar even fully scrolled.

* fix(dashboard): keep the phone Files footer and selection bar above the home indicator

The installed app runs edge to edge, so the footer's item count sat under
the iPhone home indicator. The footer now grows by the bottom safe-area
inset, and the list padding and floating action bar move up with it, as
the Apps tab and the dashboard's bottom sheets already do.

* fix(dashboard): clear the home indicator in the item properties sheet

On phones the sheet sits flush with the bottom edge, and unlike the share
sheet it mirrors, its body didn't pad for the bottom safe-area inset, so
the last details scrolled under the iPhone home indicator.

* fix(dashboard): keep the app feedback sheet's buttons above the home indicator

As a phone bottom sheet it sits flush with the bottom edge, so Cancel and
Send ended under the iPhone home indicator.

* fix(dashboard): size the touch context menu as a card on tablets

The touch menu took 90% of the viewport, which on an iPad meant a ~920px
strip with its labels far from the row they belong to. It now stops at
400px, which every phone stays under. Its height is also measured after
the width is set, so the clamp to the viewport uses the real height.

* fix(dashboard): let the Apps modals scroll in a short viewport

The Add-an-app and uninstall cards had no height cap and their overlay
centers without scrolling, so on a landscape phone with the keyboard up
(the viewport resizes for it) the request form's title and buttons sat
off-screen with no way to reach them.

* fix(dashboard): align Account and Security with the other tabs on phones

Below 768px their wrapper added a 16px gutter inside the content area's
own, so both tabs sat 32px in while Home, Apps and Usage (and the menu
button above them) sit at 16px. The extra padding stays for Apps, whose
content area has none.

* fix(dashboard): keep Home's recent apps inside their card on tablets

Each recent app was a fixed 200px inside 1fr columns, whose minimum is
that width, so in the half-width card on a landscape iPad the second
column ran past the card's edge and its name was cut off with the
ellipsis out of view. The rows now shrink with their column.

* fix(dashboard): keep every Files folder reachable in a short viewport

Trash was fixed to the bottom of the viewport, so in a landscape phone or
a short window it sat on top of Public and Videos, while Shared ran below
the column with no way to scroll to it. The folder column now scrolls,
and Trash sits at its foot in the flow: in the same spot when there is
room, after Shared when there isn't.

* fix(dashboard): don't leave Account and Security buttons highlighted after a tap

Their hover background wasn't limited to hover-capable pointers like the
rest of the dashboard's hover styles, so on touch devices a tapped button
(Change Password, Manage Sessions, ...) stayed shaded after its dialog
closed.

* fix(dashboard): give the Files row menu button a full-size touch target

On phones the row's '⋯' is 40x23, right beside a row whose tap opens the
item, so a slightly-off tap opened the file or folder instead of its
menu. In the two-row layout it now has a 44px square hit area on touch
screens.

* fix(dashboard): enlarge the Files back, forward and up touch targets

They were 28px squares on touch screens. On coarse pointers they now take
36px with a matching negative margin, so the toolbar layout is unchanged.

* fix(dashboard): size Home's recent apps card to its apps on phones

Its 310px minimum height matches the square welcome card beside it on
wider screens. In the single-column layout the welcome card drops its
square shape but the minimum stayed, leaving a block of empty card under
a short list of recent apps.

* fix(dashboard): stop clipping app names' descenders on phones and small tablets

Below 768px the tile height budgeted 52px icon + 10px margin + 12px for a
label line that is 15.6px tall, so the bottoms of g, p and y were cut off
in every app name. The tile now has the full line (78px) and the row gap
gives back the 4px, so tiles sit exactly where they did.

* fix(dashboard): show Usage amounts in full on phones

Every Usage table cell ellipsized, numbers included, so on a phone a
token count like 12,345,678 read as "12,345...". Below 480px the units
and cost columns now keep their full width and the resource name is the
one that truncates. Wider layouts are unchanged.

* fix(dashboard): stack Home's recent apps when their card is narrow

In the two-column Home on a portrait tablet (iPad Air and Pro 11 at
820-834px, or 769px windows) the Apps card is about 200px wide, and two
columns of recent apps cut every name to a letter or two. The list now
switches to one column on the card's own width; 1024px and wider keep two.

* fix(dashboard): fit two grid tiles a row in narrow Files listings

Grid view's 170px column minimum left one 330px tile per row on 360px
phones (and in a tablet's narrow listing), so barely two files fit on
screen. Narrow listings now use two equal columns, and the tile's icon
box shrinks with the tile instead of keeping its fixed 150px.

* fix(dashboard): align the Shared and Trash rows' menu button on phones

Item rows get position: relative from jQuery UI draggable, but the Shared
and Trash rows above Home's folders aren't draggable, so their '⋯' was
positioned against the whole list and sat 3px off from every other row's.
2026-10-22 15:39:47 -07:00
Daniel Salazar 631b264c1c test(ai): every AI driver holds and meters at the AI cost factor (#4304)
Image, video, TTS, speech-to-text and voice-changer driver tests run one
request at factor 1 and again with every `ai.cost.factor.*` hook set to
2, and check that both the credit hold and the recorded cost double.
Chat and OCR already had this coverage. A driver that hands a provider
the raw metering service instead of its factored view now fails its
test.

`withAiCostFactorHook` in integrationTestUtil keeps the hook in place
for a callback, since providers record their usage unawaited.
2026-10-22 15:31:15 -07:00
Filip KujundžićandClaude Opus 5.5 809c303117 fix(ai): treat a gateway 4xx with no upstream error body as a route failure (#4301)
Infron (a new-api gateway) answers "bad response status code N" with the
upstream's status when the upstream replied without a JSON error, e.g. a
model the upstream no longer serves. That 4xx was classified as the
caller's bad request, so the request skipped fallback, the dead route
stayed in rotation, and the caller got upstream_bad_request. It now counts
as an upstream failure: fallback runs, the route is marked unhealthy, and
an exhausted chain reports upstream_provider_unavailable. A gateway error
that carries the vendor's own message still stays request-level.

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-22 15:14:28 -07:00
VonypetandCursor afad53d3f4 feat(gui): add Filipino and restore the language menu (#4263)
The desktop built the language list and never placed it on the user menu, so a saved locale could not be chosen. Register Filipino and show that list.

Co-authored-by: Cursor <[email protected]>
2026-10-22 15:13:43 -07:00
Daniel Salazar 8b8dd19eae fix(teams): refuse to boot when teams_enabled has nothing registering Teams (#4300)
* fix(teams): refuse to boot when teams_enabled has nothing registering Teams

Core's seat checks (team 2FA, seat password reset, email and username
changes, self-delete, the signup placeholder claim) read the team store and
skip when it is absent. With `teams_enabled` on and no extension registering
the store and service, the server now fails at boot instead of serving
without those checks.

- whoami no longer string-matches the extension proxy's "missing property"
  error: with the flag on, the store is always there.
- Tests that turned the flag on without Teams drop it; the puter.js harness
  sets it only under the `teams` capability.
- Drop a stale services/index.ts comment.

* refactor(gui): drop the Teams console styles, which ship with the console now
2026-10-22 15:11:36 -07:00
Juan Fernando Castro 1fdde02f6a feat(metering): free and no-allowance plan traits, provisional resolver answers (#4284)
* feat(metering): let registered plans declare free and no-allowance traits

A policy passed to registerPolicy can now set `free: true`, so
isFreeSubscription and everything that reads it count the plan as free.
It can also set `noAllowance: true`: the plan grants no monthly
allowance, so only purchased credit can be spent, and the cached
credit check refuses an actor that has none. A non-positive allowance
without the trait still means the plan is not metered.

getActorSubscriptionStatus returns the plan along with whether it was
paid for and whether the answer is provisional (a lookup behind it
failed, or the actor lacks a field resolvers key on), so callers can
tell a fallback from a resolved plan. getActorSubscription is unchanged.

* feat(metering): provisional resolver answers, and tighter no-allowance accounting

A subscription resolver can answer { id, provisional: true }. The answer
wins like any id but is cached only for the fallback window.

On a no-allowance plan, the month record now records allowanceUsed = 0,
so spend that purchased credit paid for is not counted again against an
allowance the actor gets later in the month. An actor that never had
budget raises no credit-state event. setMonthAllowance refuses plans
whose allowance an override never applies to.

* fix(metering): keep resolved plans clean when a default fails, and announce moves onto no budget

A failed or provisional default resolver now marks the answer provisional
only when the default is what applied; an explicit plan that won is
cached for the full window. A provisional answer with no id still marks
the result provisional.

A plan or credit change now re-arms an actor's credit alert instead of
forgetting it, so an actor moving onto a plan with no budget hears
'exhausted' once. An actor never seen with budget still hears nothing.

* chore(metering): let a provisional answer carry no id, and fix two resolver docs

* chore: one-line comments
2026-10-22 15:47:01 -04:00
Juan Fernando Castro 1627650abc refactor: make teams an optional feature (#4248)
* refactor: let core run without a teams registration

Teams becomes optional: `services.team` and `stores.team` may be absent, and
core stops assuming either is there.

Seventeen call sites across share, signup, account deletion and the homepage
now degrade instead of throwing: a team recipient that cannot be resolved is
"no such recipient" rather than a fault, a group share folds in nothing, and
account deletion stops asking what a seat was charged for. This generalises
the pattern `gates.ts` and `SocketService` already used.

Teams' own files take a private handle to their store and service, so the
feature's code stays free of optionality that only core needs.

No behaviour changes where Teams is registered.

* refactor: put a contract between core and teams

`core/teams.ts` holds the shapes core reads and the calls it makes, and the
registries declare the pair as provided rather than built in. The service,
store and controller are registered by whatever runs the feature.

Typing them properly turned up eight AuthController call sites the previous
commit could not see: the controller reads its stores from the built-in
registry, so an absent `team` fell to an index signature and typed as
`unknown` instead of erroring. They now go through the same optional path as
the rest.

The feature's own tests move with it. The cases in core suites that build a
team to exercise a core rule still need relocating, so those suites are red
here.

* test: stub core's seat rules, rehome the rest with the feature

A core suite can no longer build a team, so the rules that refuse a seat get
`testFixtures/seatStub.ts` instead: it answers the contract, which is all
those rules read, and puts the registry back when the test ends.

Seventeen cases move that way. Four could not: group shares and a seat's 2FA
enrolment over HTTP need the real feature and the `/teams` route, so they go
with it, as does the whole of `GroupGrants.test.ts`.

Two whoami cases asserted a seat's team through `extension.import`, which
resolves the registry rather than the stores object. They now assert the
absent case, and the behaviour they covered is asserted against
`buildUserDetails`, which takes its stores as an argument.

Suite: 12252 passed, 38 skipped.

* refactor: take the teams console out of the built-in tabs

`TabTeams`, `teamsConsole` and `teamActionIcons` are no longer built in. The
dashboard already invites a tab through its `dashboard-will-open` event, so
whatever provides Teams contributes the console the same way.

Its 113 strings go with it and register into the shipped English dictionary
at load, so `i18n('teams_*')` resolves as it did. Only `en.js` ever carried
them, so no translation is lost.

Three modules stay, because things other than the console use them:
`teamBadge` (the dashboard shell), `orgSeat` (Home, Account, Security) and
`shareTeams` (the share picker). They already render nothing when the
account has no team. `teams_account_of` stays with `teamBadge` for the same
reason, and the `share_*` and `plan_team_*` strings stay with their callers.

The GUI builds clean with no console at all, and builds it in when the tab
is contributed.

* docs: seats past the owner's current limit get no allowance
2026-10-22 15:32:39 -04:00
Daniel Salazar 4968a69d45 fix(s3): keep credential-chain credentials an hour from expiry when signing (#4299)
Presigned URLs die with the credentials that signed them, and the SDK only refreshes temporary credentials five minutes before expiry, so a URL signed late in a credential's life could expire long before its requested lifetime (up to an hour). The chain provider now reports expiry an hour early and force-refreshes credentials that are already inside that window.
2026-10-22 00:47:48 -07:00
Juan Fernando Castro aa47dcd3e5 feat(workers): internal teardown that takes the script down before the row, and destroy reports Cloudflare refusals (PUT-2219) (#4297) 2026-10-21 18:50:54 -07:00
Daniel Salazar c8424f6775 fix(fetch): raise HTTP/2 receive windows for plain fetch (#4296)
Node 26's bundled undici negotiates HTTP/2 by default with a 256 KB
per-stream receive window, which capped large downloads at about
window / RTT (8.6 MB/s vs 46 MB/s over HTTP/1.1 at 23 ms). The guard swaps
in an Agent of the same class with 4 MB stream / 16 MB connection windows.
2026-10-21 13:54:31 -07:00
Daniel Salazar b3c85c86e9 feat(puter.js): open the upgrade page with what was refused (#4294)
The web upgrade dialog's button opens the dashboard with the refusal
reason, the app the call ran as, and the refused method as query params,
so the dashboard can open its upgrade flow with that context.
2026-10-21 13:09:02 -07:00
Daniel Salazar d22d419c4d chore: move to Node 26 (#4295)
* chore: move to Node 26

Docker image and CI run on Node 26 (bookworm-slim, same Debian base as before).

Node 26's bundled undici reads the global dispatcher from the
`undici.globalDispatcher.2` slot, so the guard's 10-minute timeout wrap on
`.1` no longer reached plain fetch() and provider calls fell back to the
5-minute headers timeout. The guard now wraps every present slot and tracks
wrapped dispatchers in a set, since compose() can return a Proxy that writes
a marker property through to the original dispatcher.

* test(puter-js): disable Node's web storage in test workers

Node 26 defines its own localStorage global, undefined without a storage
file, and vitest's jsdom environment doesn't replace globals that already
exist, so jsdom tests saw localStorage as undefined.
2026-10-21 13:04:23 -07:00
Daniel Salazar 0760530d0c feat(gui): edit display name and bio from the Account tab (#4293) 2026-10-21 12:57:15 -07:00
Juan Fernando CastroandDaniel Salazar e18a5c2b54 fix(apps): refuse claimed subdomains as app index_url hosts (PUT-2110) (#4267)
index_url may not point at a system host: the origin, the api host, or a
subdomain an extension has claimed. App create and update now apply the
same refusal to `<claimed>.<domain>` (any case, any port) that they apply
to the origin and api hosts.

The host-to-claim match is a new exported `subdomainLabelOf` in
hostRedirects, also used by the claim dispatch middleware, so both
decide the same way.

Co-authored-by: Daniel Salazar <[email protected]>
2026-10-21 13:19:31 -04:00
Kailasnath TandDaniel Salazar 75385501a7 fix(fs): support offset and byte_count in read operations (#4266)
* fix(fs): support offset and byte_count in read operations

Allow partial content reads via offset and byte_count parameters in puter.fs.read and backend FS controllers.

- In puter.js read operation, check options.offset and options.byte_count against null/undefined so offset: 0 is preserved in query params.
- In handle_resp, accept any 2xx status code so 206 Partial Content responses resolve successfully.
- In backend FS controllers (LegacyFSController and FSController), derive the HTTP byte range from query offset/byte_count parameters when an explicit Range header is not present.
- Add unit and integration tests across controllers and SDK test suites.

* fix(fs): strict range params, 416 past EOF, drop format churn

- resolveReadRange accepts only decimal digit strings and safe integers
- readContent maps S3 InvalidRange to 416 instead of a 500
- revert unrelated union-type reformatting, restore truncated test comment

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-10-20 21:20:51 -07:00
Daniel Salazar 5b0205f2bf fix(hosting): apps publish with write again (#4291)
Reverts the hosting part of #4290. manage stays a user-to-user grant;
an app with write on its user's directory can publish and keep serving it.
2026-10-20 17:36:36 -07:00
Daniel Salazar 3bce3b056e misc: hardening (#4290) 2026-10-20 17:12:13 -07:00
dependabot[bot] f70240784f chore(deps): bump follow-redirects from 1.16.0 to 1.16.1 (#4287)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.16.0 to 1.16.1.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.16.0...v1.16.1)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-version: 1.16.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 16:53:12 -07:00
dependabot[bot] 356e7b2b94 chore(deps): bump follow-redirects in /src/puter-js (#4288)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.16.0 to 1.16.1.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.16.0...v1.16.1)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-version: 1.16.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 16:29:27 -07:00
jelveh f02b4a6b73 feat: tell apps when a launched item is a folder; Open with AI above Open With
A folder opened with an app (Open With, Open with AI) reached puter.js as
a file: the GUI never forwarded the signature's fsentry_is_dir, so the
launched FSItem always had isDirectory false. The GUI now appends
puter.item.is_dir, and puter.js reads it in onLaunchedWithItems and the
onItemsOpened launch replay. Both now share one URL-to-FSItem builder.

"Open with AI" moves directly under Open, above Open With, in the
desktop and dashboard file menus.
2026-10-20 16:16:20 -07:00
dependabot[bot] 0768978111 chore(deps-dev): bump vitest from 4.1.11 to 4.1.12 (#4275)
Bumps [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) from 4.1.11 to 4.1.12.
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.12/packages/vitest)

---
updated-dependencies:
- dependency-name: vitest
  dependency-version: 4.1.12
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 15:47:43 -07:00
Nariman Jelveh 66093e78f3 feat(gui): "Open with AI" in the dashboard Files menu, opening like Open With (#4289)
"Open with AI" now launches the `ai` app with the item the same way
Open With launches any app: launch_app with file_path/file_uid, so the
app gets a signed launch item via puter.ui.onLaunchedWithItems. It
replaces the custom `ai:openFsEntries` postMessage, which went to an
existing AI window.

- The helper moved from UIItem into helpers/openWithAI.js, shared by the
  desktop item menu and the dashboard Files menu (generate_file_context_menu).
- A shortcut opens its target.
- Dropped from multi-select menus, since a launch opens a single item.
- Renamed open_in_ai -> open_with_ai ("Open with AI") and removed the
  unused ai_app_unavailable string.
2026-10-20 15:33:12 -07:00
dependabot[bot] ed531e3992 chore(deps): bump undici from 7.30.0 to 7.30.1 (#4280)
Bumps [undici](https://github.com/nodejs/undici) from 7.30.0 to 7.30.1.
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v7.30.0...v7.30.1)

---
updated-dependencies:
- dependency-name: undici
  dependency-version: 7.30.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 15:27:17 -07:00
dependabot[bot] 53e27f3d41 chore(deps): bump mysql2 from 3.24.5 to 3.24.6 (#4279)
Bumps [mysql2](https://github.com/sidorares/node-mysql2) from 3.24.5 to 3.24.6.
- [Release notes](https://github.com/sidorares/node-mysql2/releases)
- [Changelog](https://github.com/sidorares/node-mysql2/blob/master/Changelog.md)
- [Commits](https://github.com/sidorares/node-mysql2/compare/v3.24.5...v3.24.6)

---
updated-dependencies:
- dependency-name: mysql2
  dependency-version: 3.24.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 15:25:23 -07:00
dependabot[bot] deac121675 chore(deps-dev): bump eslint-plugin-prettier from 5.5.6 to 5.5.7 (#4278)
Bumps [eslint-plugin-prettier](https://github.com/prettier/eslint-plugin-prettier) from 5.5.6 to 5.5.7.
- [Release notes](https://github.com/prettier/eslint-plugin-prettier/releases)
- [Changelog](https://github.com/prettier/eslint-plugin-prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/eslint-plugin-prettier/compare/v5.5.6...v5.5.7)

---
updated-dependencies:
- dependency-name: eslint-plugin-prettier
  dependency-version: 5.5.7
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 15:22:09 -07:00
Nariman Jelveh 53e6e3179b refactor(gui): remove the AI side panel and its desktop button (#4286)
The desktop no longer launches the `ai` app at boot. This removes the
`is_panel` window mode, which existed only for that launch, and the
toolbar/taskbar/window/popover offsets tied to it. It also removes the
desktop sparkles button that showed the panel, and the
`silent_on_failure` launch option, whose only caller was the boot launch.
"Open in AI" in the item context menu still launches the app as a
normal window.
2026-10-20 15:21:21 -07:00
dependabot[bot] 73624bf1d0 chore(deps): bump smtp-server from 3.19.21 to 3.19.23 (#4276)
Bumps [smtp-server](https://github.com/nodemailer/smtp-server) from 3.19.21 to 3.19.23.
- [Release notes](https://github.com/nodemailer/smtp-server/releases)
- [Changelog](https://github.com/nodemailer/smtp-server/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodemailer/smtp-server/compare/v3.19.21...v3.19.23)

---
updated-dependencies:
- dependency-name: smtp-server
  dependency-version: 3.19.23
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 15:19:00 -07:00
Nariman Jelveh 1a9e43f5e5 fix(dashboard): adversarial review — bugs, a11y, perf, i18n (#4285)
* fix(dashboard): size the Notifications sidebar row like its neighbors

The bell button's font: inherit overrode the sidebar item's font-size
(16px instead of 14px, 15px in the drawer), and its icon wrapper stayed
18px while the drawer grows nav icons to 20px.

* test(dashboard): scope the directory-note assertion to the directory row

The 2FA row reuses .teams-directory-note, so the unscoped selector read
both notes and the test failed.

* fix(dashboard): only treat a picked file as a new avatar while the avatar picker is open

The Account tab listened for every file_opened on the dashboard window,
so choosing an image for a web link's Change Icon in the Files tab opened
the profile-picture crop modal, and saving it replaced the avatar.

* fix(dashboard): let dialogs over the Files tab keep their keyboard

The Files keydown handler ran under an open confirm, so pressing Enter on
the focused Delete button of the permanent-delete prompt opened the
trashed item instead (stacking a second alert), and arrows or Delete
acted on the rows behind the dialog.

* fix(dashboard): keep Apps grid keys away from an open context menu

Choosing a tile's menu item with Enter (e.g. Open in new tab) also
launched the focused app in place, and arrows moved tile focus while
they walked the menu.

* fix(dashboard): Escape on a menu or modal inside an app folder keeps the folder open

The menu's and the uninstall modal's Escape handlers removed them before
the folder's listener ran, so it saw nothing above it and closed the
folder with the same keystroke.

* fix(dashboard): a tap on the drawer or sheet scrim only dismisses it

Outside-press handlers closed the mobile drawer and the notifications
sheet on touchstart/pointerdown, which took away the scrim's hit area
before the click, so the tap also landed on the app tile or file row
underneath.

* fix(dashboard): let the sidebar nav scroll on short viewports

The nav clipped its overflow, so in a landscape phone's drawer (or a
short desktop window) Usage, Account, and Security fell below the fold
with no way to reach them.

* fix(dashboard): keep the mobile drawer full width when the desktop sidebar is collapsed

The remembered collapse outranked the drawer's width rule, so below the
drawer breakpoint it opened as an unlabeled 50px icon strip, and the
toggle that would undo it is hidden there.

* perf(dashboard): stop refetching account data each time focus leaves an app

Home's window focus listener fired whenever focus came back from an app's
iframe, so every click from an app to the dashboard re-pulled whoami,
subscription, storage, and usage (5 requests), on any tab. It now only
refreshes after the page itself lost focus to another window.

* fix(dashboard): show a seat's team name unescaped in the plan card

i18n escapes its arguments for HTML, but the note is set with .text(), so
a team called "Bob's & Co" read "Managed by Bob&apos;s &amp; Co".

* fix(dashboard): stop double-escaping app names in uninstall alerts

UIAlert escapes its message, so an app titled "Tom & Jerry" read
"Tom &amp; Jerry" in the uninstall-failed and sessions-not-ended alerts.

* fix(dashboard): roll back an uninstall whose revoke failed

puter.perms.revokeApp resolves with error set instead of rejecting, so a
failed revoke still removed the tile, marked the app removed for later
sessions, and ended its sign-in, while the app kept its permissions and
the rollback and error alert never ran.

* fix(dashboard): read a team account's suspension when its name matches an Object member

The per-account state map was a plain object, so a username such as
constructor or toString was already "in" it, its audit entries were
skipped, and a suspended account showed as active with no Delete.

* fix(dashboard): update the profile card and sidebar name after a username change

Neither carried the .username class that the rename refreshes, so both
kept showing the old name until reload.

* fix(dashboard): ask how to restore a trashed item whose name is taken

Restore moved without overwrite or dedupe, so when the original location
already had an item of that name the move 409'd and nothing happened:
an uncaught error from the item menu, a console line from the others.
All three restore entry points now share one path that offers the
Replace / Keep Both prompt the paste flow uses, alerts on other errors,
and updates the footer and empty notice once the rows are gone.

* fix(dashboard): refresh a renamed row's name, icon, and position in the Files tab

The Files tab passed rename_file a callback it never accepted, so a
renamed row kept the desktop's 20-character truncation even in a wide
column, kept its old type icon, and stayed out of sort order. rename_file
now takes an optional onDone (carried through undo), which the Files tab
uses to fit the name to its column, refresh the icon and type, and move
the row into place. A cancelled rename no longer truncates the name
either.

* fix(dashboard): drop Publish and Properties from the Shared view's menu

The Shared view is a query, not a folder, so both acted on the virtual
shared path: Properties showed an unknown error and Publish opened a
dialog for a path that doesn't exist.

* fix(dashboard): close the Files image preview on Escape and when leaving the folder

Escape cleared the selection but left the Space preview open with
nothing selected, the preview stayed over a different folder after
navigating, and a preview whose URL arrived after it was dismissed or
replaced still opened.

* fix(dashboard): don't move Files history for a navigation that's dropped

renderDirectory ignores a navigation while another folder is loading,
but every caller had already pushed or stepped history, so after a click
during a load the listing and history disagreed and the next Back or
Forward seemed to do nothing. History now only moves when the render
will run; back/forward steps share one helper.

* fix(dashboard): reopening the current folder adds no Files history entry

Clicking the folder already on screen (e.g. Home in the sidebar) pushed
a duplicate entry, so the next Back reloaded the same folder.

* fix(dashboard): show installed apps when the recommended list fails to load

get-launch-apps sat in the same Promise.all as the installed apps and
its body was parsed without a status check, so a network error or a
proxy's HTML error page replaced the whole grid with "Failed to load
apps". A failed list now falls back to the last one that loaded.

* fix(dashboard): fit the app uninstall dialog on narrow phones

The card is mounted outside the dashboard's border-box rule, so its
320px min-width plus padding came to 370px and was clipped on 360px
screens; a long unbroken app title also overflowed it. Desktop sizing is
unchanged.

* fix(dashboard): keep a new team account's one-time password on screen until dismissed

The credential box lived only in the DOM inside the panel that every
refresh repaints, so revisiting the Teams tab or any later action wiped
a password that can't be retrieved again. It is now kept in state, for
the team it belongs to, and drawn with the panel.

* fix(dashboard): reload the Teams tab after a change made while it was loading

refresh() handed back any load already in flight, so an action or team
switch made during a revisit's background load awaited data read before
the change: a toggled setting snapped back, and a team switch could
paint one team's roster next to another's record. A change during a load
now queues one more load, the stale one skips its paint, and revisiting
the tab doesn't start a second load while init's is still running.

* fix(dashboard): send one re-enable request for a double click in Teams

Re-enable has no confirm dialog to absorb a second click, and nothing
held the action while it ran, so a double click enabled the account
twice (two audit entries). A suspend or re-enable already in flight for
an account now ignores further clicks.

* fix(dashboard): revert the avatar and say so when saving a new photo fails

The crop's Save showed the new picture everywhere and revealed Remove
without waiting for the upload, and update_profile swallows failures,
so a failed save looked saved until reload. The save is now awaited: a
failure puts the previous picture back with an error hint, and Remove
waits for the upload so it can't race it.

* fix(dashboard): hide the Usage tab's plan button for team seats

A seat's plan belongs to its team, so Home already hides its upgrade
link; the Usage tab still offered Upgrade, which only reaches a refusal.

* fix(dashboard): don't drop Home's forced usage refresh during a running load

loadUsageData handed any caller the load already in flight before looking
at force, so a subscription change (or a return from the billing portal)
during a routine load was answered with data read before the change, and
the plan card stayed stale until the next refresh. A forced call now runs
once more after the running load.

* fix(dashboard): activate Apps when the URL names an unknown tab

An unknown hash rendered Apps as the active tab but skipped its
activation, so the grid kept the generic content padding and Apps'
onActivate never ran.

* fix(dashboard): make the sidebar user menu reachable from the keyboard

The user button was a div with no tabindex, so Log Out, account
switching, and Contact Us were out of reach without a pointer. It is
now a button (rendering unchanged), with aria-haspopup.

* fix(dashboard): name the sidebar's collapse and drawer buttons for assistive tech

The collapse toggle held only two unlabeled images, so screen readers
announced an unnamed button; it now carries a translated label and
aria-expanded that follow its state. The drawer's open/close labels were
hardcoded English and now go through i18n.

* fix(dashboard): make Home's recent apps reachable from the keyboard

Each recent app was a click-only div, so keyboard users couldn't open
one. They are now buttons with a focus ring; the icon is decorative
since the title names the button.

* fix(dashboard): name the Security tab's 2FA switch for assistive tech

The checkbox sat in a text-less label, so screen readers announced an
unnamed checkbox. It now carries the translated Two Factor
Authentication label, as the Teams switches do.

* fix(dashboard): sort the Usage table from the keyboard

The sortable headers were click-only th cells. Each now holds a button,
carries aria-sort, and keeps focus across the re-render a sort causes.
The Resource/Units/Cost headers go through i18n.

* fix(dashboard): don't rebuild the Files listing under an open menu

Returning to the browser tab re-rendered the folder even with a context
menu or action sheet open, so its actions then targeted detached rows:
Rename did nothing, Restore faded a row that was already gone. The
refresh now waits while a menu is up, as it does for a rename.

* fix(dashboard): point the Files sidebar's Documents at the current home

The entry read window.documents_path, which globals.js sets once from
the cached user and nothing updates; login and username changes keep
window.docs_path current instead. After a username change, Documents
pointed at the old home and showed "This folder couldn't be opened",
even after reloading.

* fix(dashboard): let only the newest share listing render in the share dialog

Each action ends in a refresh, and refreshes from overlapping actions (or
the opening load) could finish out of order, so an older listing landed
last and undid what the user just did, e.g. the link control snapped
back to restricted after turning the link on. Older refreshes now drop
their result.

* perf(dashboard): load a team's record, roster, and plan in parallel

loadSelected awaited the audit log (every page of it), then the members,
then the plan, on every visit to the Teams tab. They don't depend on
each other, so they are now fetched together.

* fix(dashboard): keep keyboard focus on the Teams control that was used

Every Teams action repaints the whole panel, which dropped focus to
<body>: a keyboard user toggling a switch or suspending an account was
sent back to the top of the page. The repaint now refocuses the same
control, or the same account's new action when the old one is gone.

* fix(dashboard): give the app uninstall prompt dialog semantics and focus

The prompt had no dialog role or label and never took focus, so keyboard
and screen-reader users stayed on the inert grid behind it. It is now a
labelled modal dialog that focuses Cancel on open and returns focus to
the app's tile when dismissed.

* fix(dashboard): keep the add-an-app tile when there are no apps

With no apps at all, the empty state replaced the whole grid, add tile
included, so a new user on an instance with no recommended apps had
nothing to act on. The empty line now sits above a grid that keeps the
tile, as a search with no matches already does.

* fix(dashboard): don't rebuild the Apps grid under an active drag

An uninstall finishing (or a failed one rolling back) re-rendered the
grid even while another tile was being dragged, replacing the dragged
tile with a fresh copy: the copy showed under the ghost and the drop
was lost. Renders asked for mid-drag now wait for the one the drop does;
the folder eject, which rebuilds the grid as part of the drag, still can.

* fix(dashboard): return focus to the drawer toggle when the mobile drawer closes

Closing the drawer (Escape, the close button, a tab pick) left focus on
controls that had just become hidden, so it fell to <body>; closing the
notifications sheet opened from the drawer tried to refocus the bell
inside the closed drawer. Both now land on the drawer toggle.

* fix(dashboard): truncate Files names between graphemes, with a binary search

The middle truncation sliced UTF-16 code units, so at many widths a long
name with emoji showed a half surrogate (rendered as a replacement
character) or a broken joined emoji. It also measured one more character
per step, a measureText call per character of every long name on each
render and column resize. The cut now falls between graphemes and is
found by binary search; output for other names is unchanged.

* fix(dashboard): translate the Home tab's hardcoded English

The tab label, the recent-apps empty state, card subtitles, the plan
badge and trial wording, the upgrade/manage links, and the usage cards'
"Used" / "of" lines were English literals. They now go through
i18n, with placeholders where values are interpolated.

* fix(dashboard): translate the sidebar's remaining English

The Apps, Files, and Usage tab labels, the user button's fallback name,
and the user menu's Developers item were English literals; they now use
i18n keys like the other tabs.

* fix(dashboard): translate the Usage tab's hardcoded English

The subtitle, the "used of" meter lines (now the existing used_of
key), Usage Details, the plan button, Show more/less, and the details
error were English literals.

* fix(dashboard): translate the Account and Security tabs' hardcoded English

Both subtitles, the delete-account description, the sessions hint, the
username fallback, and the avatar picker's window title were English
literals.

* fix(dashboard): translate the Apps tab's hardcoded English

The uninstall prompt and its failure alert, empty and no-match notices,
pager and folder-dot labels, the search placeholder, the edit toggle,
and the tile menu's Open in New Tab / Quit / Uninstall were English
literals.

* fix(dashboard): translate the Files tab's hardcoded English

The folder sidebar, footer counts (now the existing item-count keys),
sort and view menus, upload status and errors, the empty and
can't-open notices, the reserved-name alerts, the move error, and the
loading text were English literals.

* fix(dashboard): translate the action sheet's Back row and match delete in any language

The touch action sheet styled an item red when its label contained the
English word "delete", so Delete lost its danger styling in every other
language; it now compares against the translated Delete labels. Its
submenu Back row and the properties panel's Directory/disassociate
labels were English literals.

* fix(dashboard): make the Files sidebar, crumbs, and nav arrows keyboard-operable

The folder sidebar, the breadcrumbs, and the back/forward/up arrows were
click-only li/span/img elements, out of the tab order. They now take
focus, carry button/link roles (the arrows with labels and
aria-disabled), answer Enter and Space, and show a focus ring. The rows'
Enter/Space shortcuts also no longer swallow those keys from a focused
control: with rows selected, Enter on Upload opened the selection.

* perf(dashboard): read grid name widths before writing any in the Files tab

fitItemNames read each tile's name width right after writing the
previous name, forcing a layout per row. Reading them all first, then
writing, cuts switching 400 files into grid view from about 300ms to
170ms; the rendered names are identical.
2026-10-20 13:44:42 -07:00
242a654166 feat(ai): add Kling for video, image and speech generation (#4274)
* feat(ai): add Kling for video, image and speech generation

One `kling` API key now registers a provider on three drivers:

- txt2vid: Kling 3.0 Turbo, 3.0, 3.0 Omni, O1, 2.6 and 2.5 Turbo,
  with first/last frames, reference images on the omni models, native
  audio, and editing a source video on 3.0 Omni and O1. Billed per
  second at the resolution, audio and video-input rate; an edit is held
  at the longest source and billed for the clip Kling returns.
- txt2img: Kling Image 3.0, 3.0 Omni, O1 and 2.1, billed per image.
- txt2speech: Kling TTS with its English and Chinese voices, billed per
  request.

Kling's business error codes map onto the driver error contract, so an
exhausted resource pack reports as credit exhaustion and a submit-time
content block as `moderation_flagged`. "Risk control" failures now
count as content-filter refusals.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(ai): treat Kling's unknown-model codes as a Puter-side failure

Kling answers 1202/1203 when the method or model isn't one it serves.
Puter picks both, so these meant catalog drift but surfaced as a 400 that
blamed the caller and raised no alarm. They now return 502 upstream_failed
with a warning alarm.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(ai): tighten Kling input checks and error mapping

- 3.0 Turbo takes at most 2,500 prompt characters with a first frame,
  the image-to-video endpoint's limit.
- O1 refuses reference images together with a first and last frame.
- 3.0 Omni accepts a source of up to 15.5 seconds, and an edit is held
  at 16 seconds to cover it.
- A source video Kling can't decode fails as 400 upstream_bad_request
  instead of a Puter-side 502.
- A 1201 about the model counts as catalog drift, like 1202/1203.
- Auth failures no longer pass Kling's console wording to the caller.

Adds integration tests for the image, video and TTS providers, gated
on PUTER_TEST_AI_KLING_API_KEY.

* feat(ai): wait up to 14 minutes for a video job

Video jobs, Kling edits in particular, can take over 10 minutes. The
poll window and the Gemini Omni deadline move to 14 minutes, under the
SDK's 15-minute idle timeout, with holds and signed source links
lengthened to match. The keep-alive default moves to 920 seconds so it
stays above a 15-minute proxy idle timeout.

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: Daniel Salazar <[email protected]>
2026-10-20 12:32:19 -07:00
Daniel Salazar 41e4b582d3 fix(subdomains): emit subdomain.delete for sites deleted with an app (#4273)
Sites and workers tied to an app go with it by FK cascade, on app delete and
on deleting the app owner's account. Only caches were dropped for them, so
the subdomain.delete listeners (CDN purge, worker script teardown, worker
sessions, site cleanup) never ran.
2026-10-20 12:31:40 -07:00
Reynaldi Chernando 2a49689fad fix contradicting jsdocs and docs (#4282) 2026-10-20 18:39:49 +07:00
Juan Fernando CastroandDaniel Salazar b54b76715a feat(fs): finish a home rename that stopped partway (#4255)
* feat(fs): finish a home rename that stopped partway

FSEntryStore.rebaseUserSubtree runs renameUserHome's owner-scoped
cascade on its own (same subtree matching, same rebased-cache
invalidation) and also drops the home row's cached copies at either
path. countUserSubtree counts the owner's rows under a prefix with that
same matching, from the primary. renameUserHome now uses the shared
cascade.

* fix(telemetry): resolve the tracer per span so a test's provider is the one used

withSpan held a ProxyTracer built at module load, which caches the first
provider it delegates to. Test files sharing a worker then sent spans to a
provider an earlier file had already shut down.

* fix(fs): drop cached copies of only the rows a home rebase moves

The owner-scoped rewrite read its invalidation set from everything under the new prefix, which for a finishing rebase is the whole live home; it now reads the rows it is about to move.

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-10-20 00:59:40 -07:00
Juan Fernando Castro d978538c9d feat: let a subdomain claim keep the main pipeline (#4268)
`extension.claimSubdomain(name, { mainPipeline: true })` serves
`<name>.<domain>` the way core's own subdomains are served: through CORS,
body parsing, auth and every route, `subdomain: '*'` ones included. Routes
declared for the name are ordinary routes, with no option restrictions.
Gated claims (`{ authorize, checkRoute? }`) behave as before.

onlyoffice's subdomain now comes from its extension's claim, so core no
longer lists it among the subdomains it serves itself. Without that claim
`onlyoffice.<domain>` gets the unclaimed-subdomain 404 like any other name.
2026-10-20 00:43:54 -07:00
e83e27f306 feat(ai): edit a source video with txt2vid (Gemini Omni, FLUX 3) (#4272)
* feat(ai): edit a source video with txt2vid on Gemini Omni

txt2vid takes an `inputVideo` option (sent as `input_video`): a URL, data
URI or Puter file path to an MP4, QuickTime or WebM video of up to 32 MiB.
Gemini Omni receives it inline next to the prompt, which describes the
edit or extension. Google enforces the 10-second limit.

Models declare `videoInput` in the catalog, and the driver rejects a
source video for any other model instead of letting the provider drop it.
The credit hold adds 16,000 input tokens for a source video, since
countTokens reports far fewer tokens than the reply bills.

loadFileInput takes an optional `timeoutMs` for web URLs, and cancels
the body of a failed response.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(ai): edit a source video with FLUX 3 on Together AI

FLUX 3 takes `input_video` as Together's `media.source_video`. A web URL
goes through for Together to fetch; a data URI or Puter file is read with
the caller's access, capped at 32 MiB, sniffed as a video and sent inline.
Editing is estimated and billed at the 1080p rate of $0.54/s, since FLUX 3
jobs report no cost. Image inputs can't be combined with a source video.

Seedance 2.0 ignores `source_video`, treats `reference_videos` as a loose
reference rather than an edit, and Seedance 2.5 fails on reference videos,
so neither declares `videoInput`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(ai): name FLUX 3 as a video editor in the AI overview

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(ai): match Hailuo 02 to Together's price and document estimate billing

Together quotes $0.56 for a Hailuo 02 clip; the catalog had $0.49.

Together's video jobs no longer report what they cost, so every Together
clip is billed at the catalog's highest published rate. The docs said the
charge was the amount Together reports; they now describe the estimate.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix: enforce video edit durations and propagate source fetch cancellation

* fix(ai): pass media inputs as Puter files and stream source videos

Source videos were read whole into the API process, base64-encoded and
sent twice to Gemini (countTokens and the interaction), which held about
128 MiB per in-flight edit for up to ten minutes.

- Video inputs are a URL, a Puter file path or a data URI of at most
  1 MiB. Together gets a Puter file as a 15-minute presigned S3 URL and
  fetches it itself; Gemini streams it into the request body with an
  exact Content-Length, so it's never held as one string.
- Gemini counts only the prompt and image for a source video, and both
  providers read the source only after the credit hold.
- txt2img and txt2vid image inputs accept Puter file paths, read within
  the caller's access right before the provider call. Existing URL,
  data URI and base64 inputs are unchanged.
- puter.js writes a Blob, File or data URI over 1 MiB to a temporary
  file under .tmp, sends its path, and deletes it once the call settles.
- FLUX 3 edits bill at the 720p editing rate, matching their 1280x704
  output.

* fix(ai): stage inputs after website sign-in, keep images working without a write

- puter.js chose the temporary file path before signing a website visitor
  in, so it fell outside the app's data directory and the upload was
  refused. It now signs in first, then picks the path.
- An image that can't be written (no access, storage full) is sent inline
  as before instead of failing the call. Videos still fail, since a
  large inline video isn't accepted.
- Telling raw JPEG base64 from a path decodes only the first 24
  characters instead of the whole input.

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: 404oops <[email protected]>
Co-authored-by: Daniel Salazar <[email protected]>
2026-10-19 23:33:03 -07:00
Udit AgarwalandDaniel Salazar 54f542ace4 fix(gui): refresh Start menu apps without blocking cached rendering (#4244)
* fix(gui): refresh Start menu apps without blocking cached rendering

* docs: remove desktop Start menu limits from SDK reference

* fix(gui): drop the 1 MiB cap on the launch-apps refresh

Recent entries carry each app's raw icon column, which can still hold a data URL of up to 5 MB, so a large icon tripped the cap on every request, including the one at startup, and left the Start menu with no apps. Replace the cap's test with one for the unlaunchable-app filter.

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-10-19 16:13:55 -07:00
Daniel Salazar 18b96d529c fix: misc scan fixes (#4270) 2026-10-19 12:32:13 -07:00
Kunal Jaiswal f146da16d1 fix(gui): drop deleted apps from the start menu (#4243)
The start menu read a session-cached launch-apps list and only refetched when it was empty, so an app deleted after page load (for example in Dev Center) stayed in Recent and Recommended until a reload. Refetch on every open, and filter out entries the backend marks unlaunchable, so a reclaimed or inaccessible app leaves the list too. Apply the same filter to the dashboard's Recent apps. Adds helpers/launchAppsList.js with filterLaunchableApps and its unit test.
2026-10-19 10:50:36 -07:00
Daniel Salazar dd56d54753 docs(bug-bounty): add eligibility rules (#4269)
Bounties cover user security/privacy and platform security/health only.
Reports must show a direct attack without pre-assumed access, and
exploits that rely on sharer/sharee trust are hardening or lower
severity. Also generalizes the metering non-issue, adds existence-only
oracles, and tightens duplicate, proof and first-contact rules.
2026-10-19 10:49:12 -07:00
Daniel Salazar e5fd30ccf0 fix(ai): validate large base64 by decoded length, not regexps (#4265)
* fix(ai): validate large base64 without anchored regexp loops

Gemini TTS replies over ~4 MB of base64 (~65 s of audio) failed with
"Maximum call stack size exceeded". Once a process holds enough compiled
code, V8 compiles new regexps unoptimized, and `^[...]+={0,2}$` then
pushes a backtrack entry per character until its stack overflows.

TTS now searches for a character outside the alphabet instead. The raw
base64 icon/reference-image check drops its own anchored regexp;
decodeStrictBase64 already rejects the same inputs.

* perf(ai): validate base64 by decoded length instead of regexp passes

Unoptimized regexp scans dominated large payloads: 186 ms for max-length
TTS audio, 880 ms in decodeStrictBase64 for 30 MB of video. Buffer.from
skips characters outside the alphabet, so comparing the decoded length
with the one the input length predicts catches them for free.

decodeStrictBase64 now does that plus rejecting the URL-safe alphabet
(~10 ms for 30 MB), and TTS reuses it instead of its own check.
Non-canonical trailing bits now decode instead of being rejected.

* fix(ai): reject non-ASCII base64 that Node decodes by its low byte

Node reads code units above 0xFF by their low byte, so U+012B decodes
as '+' and keeps the decoded length the input predicts. Require ASCII
(~1-2 ms for 43M chars) before the length check.
2026-10-18 21:56:29 -07:00
Neal Shah 517efda38a automerge emergency fix (#4264) 2026-10-19 00:18:58 -04:00
Neal Shah ea3801fb74 PUT-2058 and PUT-2096 (#4256) 2026-10-18 23:30:01 -04:00
Daniel Salazar 0786cc7db2 fix(webdav): PROPPATCH checks write access and answers with the masked path (#4260)
PROPPATCH now requires write access on the target, like PUT and DELETE,
and its response href goes through the request's share-path masker the
way PROPFIND's do.
2026-10-18 00:33:54 -07:00
Daniel Salazar 592020c899 docs(bug-bounty): list re-claimed abandoned site and worker names as a known non-issue (#4261) 2026-10-18 00:33:49 -07:00
Daniel Salazar 9ff224e22f perf(backend): cut sequential round trips on auth, fs, driver, event and site paths (#4259)
Pipeline
- CORS preflights send Access-Control-Max-Age: browsers stop re-preflighting authed calls every 5 s.
- `req.ua` is parsed on first read instead of on every request.
- JSON `rawBody` reuses the parsed buffer instead of copying it.
- `emitAndWait` returns early when nothing listens: no span or closure per unlistened driver call.
- SQLBatcher sends a lone query on the next tick while fewer than 8 flushes are in flight, and
  waits `maxTimeInQueue` as before otherwise. Lone reads skip the 10/30 ms wait; batching under
  load is unchanged.

Auth and permissions
- Token-to-actor auth starts the user, session and app lookups together and checks them in the
  same order: fewer sequential Redis round trips per authed request.
- get-user-app-token and check-app reuse the resolved app row. getOrCreateApp skips the re-select
  after its own insert and doesn't wait on the cache write for replica rows.
- getOrCreateApp reads the race winner from the primary when its insert loses, so it no longer
  hands out a session that was never written.
- whoami runs the email check and the OIDC/taskbar/team lookups in parallel; key order unchanged.
- OIDC providers are memoized once every discovery succeeds, with one discovery fetch in flight
  per URL, and the Apple key is parsed once.
- gui-token and sync-cookie use `req.actor.user` instead of re-reading the user.
- listSessions batches app lookups. App icon probes run together, and the original-icon probe
  skips the lineage fallback.
- Default-granted permissions are answered from config before the scan cache, and exploded
  options are deduped: one less Redis read per /drivers/call.
- ACL resolves ancestors lazily (owners and full-access tokens skip them) and scans the node
  first, then the ancestors concurrently on a miss.

Filesystem
- Missing-path lineage lookups run 10 at a time: batch-upload path resolution drops from about
  1.3 s to about 40 ms per 100 new files.
- MySQL single-file inserts return the row built from the written values, falling back to the
  primary read: one less primary round trip per new file.
- Batch thumbnails are stored with one UPDATE per chunk (200 rows or 4M chars), with a per-row
  fallback.
- storage.quota.bonus listeners run concurrently next to the usage SUM, and the payload carries
  `userUuid`: quota checks and /df no longer serialize the listener reads.
- readdir cache backfill is one pipeline instead of 3xN SETEX.
- Profile reads skip the lineage fallback; the own profile is read alongside the user lookup.
- Legacy /stat runs is_empty (new `hasChildrenByParentUid`), owner and thumbnail concurrently.
- Downloads and the video proxy pipe through `pipeToResponse`: aborts release the storage stream
  at once with no extra 'close' listeners, and the video proxy no longer crashes on a mid-stream
  upstream error.
- mkdir looks up target and parent together.
- Upload completion reuses the session from the access check (status is still re-read before
  completing), and multipart signing reads it once.
- Moving a shared file to Trash skips two lookups that always returned the entry itself.

Drivers and events
- Subscription resolvers run concurrently and are folded in registration order.
- SubdomainDriver and WorkerDriver overlap independent lookups.
- The events "anyone subscribed?" check is one Redis pipeline when the region is warm.
- Handler publish overlaps the count with the size check.
- Broadcast flushes peers concurrently over a keep-alive agent, so a slow peer no longer delays
  cross-region invalidations to the others.

Sites, share and AI
- Hosted sites resolve the owner first, then the app and root together. Site config loads only
  on 404s. The hosted-app query selects only the columns it uses and moves into AppStore.
- shareFlags asks about the plan only when an item is shared by link only.
- Share listings run the count, teams and plan reads alongside their page query.
- Claude attachment cleanup no longer blocks the response.
- New util `awaitLater` for promises started early and awaited later.
2026-10-17 12:03:35 -07:00
Daniel Salazar 9ad7b99607 fix(puter-js): handle file picker cancel message (#4257) 2026-10-16 23:51:55 -07:00
Daniel Salazar 28f162ee92 fix(metering): fence settles, idempotent monthly claim, split addon consumption (PUT-2001) (#4236)
* fix(metering): pin metering writes to one replica, fence settles, idempotent monthly claim (PUT-2001)

store-kv-v1 is a last-writer-wins global table, and every region settled
its metering buffer into its local replica, so concurrent increments to the
same per-actor item from two regions could drop one side. Writes and
consistent reads of metering:* and policy:actor:* items now go to the
dynamo.home replica.

Buffered settles record each write on the claim with a fence token before
sending it, and the KV store refuses a token the item already holds, so a
re-driven claim applies its write once. Claims are serialized per counter
within a cache so one writer's tokens land in order.

The monthly charge claim is a set-if-absent of a random token, and a failed
claim write keeps its token for the retry, so a write that landed anyway
still charges exactly once.

* fix(metering): drop home-replica pinning; keep addon consumption off the purchase record (PUT-2001)

Pinning every metering write and exact read to one replica put a
cross-region round trip on credit checks near the allowance and on every
settle, and made one region's outage stall metering everywhere. The
per-item clash it guarded against is rare and undercharges at most a
flush window.

The one clash that loses money is a credit purchase and an overage settle
landing on the addons item in different regions at once. Consumption now
goes to its own item and getActorAddons sums it with what the purchase
record already holds.

* perf(metering): read addon consumption only for accounts with purchased credit (PUT-2001)

* perf(metering): settle counters through a sliding pool instead of waves (PUT-2001)

* feat(metering): consistent-read option on getActorAddons (PUT-2001)

Admin credit checks read the purchase record directly to get a consistent
read, which misses consumption now that it lives on its own item. They can
call getActorAddons with consistentRead instead. Also formats the lines this
PR added.
2026-10-16 23:49:23 -07:00
Daniel Salazar fa62b3fb76 move template menu entries into a tested helper (#4258)
Builds the "New > User templates" entries in helpers/templateMenuItem.js
and covers the label with tests, including HTML encoding of both the
extension and the name.
2026-10-16 23:45:04 -07:00
Filip Kujundžićand404oops e950a87f29 Add Gemini Omni video generation with token metering (#4235)
* Add Gemini Omni video generation with token metering

* Keep explicit Gemini video model selection within provider

* Settle reported Gemini video usage before validating output

---------

Co-authored-by: 404oops <[email protected]>
2026-10-16 23:21:48 -07:00
1cb9dd756a Add Gemini transcription with consistent usage settlement (#4234)
* feat(ai): add Gemini unary transcription

* Clarify transcription language results and omit pricing

* style(ai): keep Gemini transcription sources prettier-clean

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(ai-speech2txt): charge Gemini transcriptions that fail after Google answers

Google bills an interaction it processed whether or not it completed, so an
incomplete, failed or malformed reply was free to the caller and paid by
Puter; audio whose transcript overran the output cap could be resubmitted at
no cost. Usage is now recorded before the reply is checked, falling back to
the counted audio tokens when the reported usage can't be read.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Preserve transcript text with optional malformed annotations

* Keep speech provider spacing consistent

---------

Co-authored-by: 404oops <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-16 23:21:47 -07:00
be16bca1d9 Keep Gemini speech instructions separate and validate speech requests (#4233)
* fix(ai): keep Gemini speech instructions separate from transcript

* Clarify Gemini speech instructions

* style(ai-tts): keep Gemini cost comment prettier-clean

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Validate speech requests and settle reported Gemini usage

* Clarify speech settlement for unreadable responses

* Reserve the published Gemini speech token budget

* Clarify speech accounting for bounded responses

* Limit only Gemini TTS text to 3000 characters; charge overrun replies

The 3000-character text cap moves from the TTS driver into the Gemini
provider, so other providers keep their own limits.

The Gemini reply cap rises to 64 MiB, which fits the full 16,384-token
audio budget. A reply that still overruns it or times out reports no
usage, so it is charged the held budget and the error tells the caller
to split the text.

---------

Co-authored-by: 404oops <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: Daniel Salazar <[email protected]>
2026-10-16 23:21:47 -07:00
2305d50e36 Restore supported Gemini chat models and align scheduled pricing (#4232)
* feat(ai): restore available Gemini models and align pricing

* Exclude deprecated Flash Image additions and share scheduled pricing

* Bill Gemini 2.5 Pro at its current list rates

Cache reads are $0.125/M, and prompts over 200k tokens bill every rate at
the long-context tier ($2.50 in, $15 out, $0.25 cached).

---------

Co-authored-by: 404oops <[email protected]>
Co-authored-by: Daniel Salazar <[email protected]>
2026-10-16 23:21:47 -07:00
352427687e Add Gemini Argon with mode-specific limits and native usage metering (#4231)
* feat(ai): add Gemini 4 Argon with native usage metering

* test(ai): isolate Argon HTTP smoke test from optional extensions

* fix(ai): preserve Gemini native completion and refusal reasons

* Refer model specifications and pricing to the directory

* Bound unary Argon requests and reject unsupported content

* Keep Gemini tool metadata out of Claude requests

* Describe provider-specific chat timeout boundaries

* Keep Gemini extra_content off other providers; report Gemini timeouts as 504

Replayed tool calls only carry extra_content for providers that opt in
(Gemini), so OpenAI-compatible and Responses vendors don't see an unknown
field. The native Gemini client now owns its request deadline: the SDK's
own timeout aborted without a reason and read as an internal error. A
deadline timeout returns 504 upstream_timeout and leaves the route in
rotation.

---------

Co-authored-by: 404oops <[email protected]>
Co-authored-by: Daniel Salazar <[email protected]>
2026-10-16 23:21:46 -07:00
145d54cf69 Handle unavailable chat model routes and exclude batch variants (#4230)
* Handle model access denials and filter batch chat variants

* Raise a low alarm when a provider denies model access

A 403 model_not_found still returns 400 upstream_bad_request, but now
alarms at info so a lost provider entitlement gets noticed. HttpError
gains an alarmSeverity option the terminal alarm gate honors.

---------

Co-authored-by: 404oops <[email protected]>
Co-authored-by: Daniel Salazar <[email protected]>
2026-10-16 23:21:46 -07:00
Reynaldi ChernandoandClaude Opus 5.5 471b900462 remove private access routing from docs (#4252)
Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-16 20:14:21 +07:00
Daniel Salazar 2b254b8892 fix: misc bugs (#4251) 2026-10-15 23:49:17 -07:00
Neal Shah 2a15afc570 Resolve PUT-2059 (#4250) 2026-10-16 01:45:56 -04:00
dependabot[bot] f9860b036b chore(deps-dev): bump prettier from 3.9.9 to 3.9.10 (#4240)
Bumps [prettier](https://github.com/prettier/prettier) from 3.9.9 to 3.9.10.
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/3.9.9...3.9.10)

---
updated-dependencies:
- dependency-name: prettier
  dependency-version: 3.9.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 14:46:31 -07:00
dependabot[bot] 76e1f79b5e chore(deps): bump smtp-server from 3.19.17 to 3.19.21 (#4242)
Bumps [smtp-server](https://github.com/nodemailer/smtp-server) from 3.19.17 to 3.19.21.
- [Release notes](https://github.com/nodemailer/smtp-server/releases)
- [Changelog](https://github.com/nodemailer/smtp-server/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodemailer/smtp-server/compare/v3.19.17...v3.19.21)

---
updated-dependencies:
- dependency-name: smtp-server
  dependency-version: 3.19.21
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 14:43:54 -07:00
dependabot[bot] 943a3427c6 chore(deps-dev): bump @types/busboy from 1.5.4 to 1.5.5 (#4241)
Bumps [@types/busboy](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/busboy) from 1.5.4 to 1.5.5.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/busboy)

---
updated-dependencies:
- dependency-name: "@types/busboy"
  dependency-version: 1.5.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 14:38:42 -07:00
dependabot[bot] 38469a3615 chore(deps): bump libphonenumber-js from 1.13.14 to 1.13.15 (#4239)
Bumps [libphonenumber-js](https://gitlab.com/catamphetamine/libphonenumber-js) from 1.13.14 to 1.13.15.
- [Changelog](https://gitlab.com/catamphetamine/libphonenumber-js/blob/master/CHANGELOG.md)
- [Commits](https://gitlab.com/catamphetamine/libphonenumber-js/compare/v1.13.14...v1.13.15)

---
updated-dependencies:
- dependency-name: libphonenumber-js
  dependency-version: 1.13.15
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 14:35:15 -07:00
dependabot[bot] a4d4a5595e chore(deps-dev): bump vite from 8.1.5 to 8.1.6 (#4238)
Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 8.1.5 to 8.1.6.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v8.1.6/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v8.1.6/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-version: 8.1.6
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 14:35:07 -07:00
dependabot[bot] de69269b10 chore(deps): bump uuid from 14.0.2 to 14.0.3 (#4237);'''"????????????????????
Bumps [uuid](https://github.com/uuidjs/uuid) from 14.0.2 to 14.0.3.
- [Release notes](https://github.com/uuidjs/uuid/releases)
- [Changelog](https://github.com/uuidjs/uuid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/uuidjs/uuid/compare/v14.0.2...v14.0.3)

---
updated-dependencies:
- dependency-name: uuid
  dependency-version: 14.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 13:25:49 -07:00
Neal Shah 8faff74793 PUT 2057 and PUT 2068 (#4245) 2026-10-15 02:02:00 -04:00
Neal Shah e5ca303c50 PUT-1903 (#4226) 2026-10-14 20:29:55 -04:00
Juan Fernando Castro c6c0dd5e29 fix: bound how many accounts a team owner can provision (#4223)
* feat: bound provisioning by what an owner may mint, not only what is live

Seat and team caps express how much may exist at one time. This adds the
other half of the rule: how much an owner may create over a window, and the
checks a provisioned account answers to.

- Provisioning puts each seat through `SignupService.runSignupGate`, the same
  extension-owned gate a signup answers, and refuses what it blocks. It fails
  closed, since a check that could not run is not a check that passed.
- A per-owner budget over a rolling window, sized to every seat the owner may
  hold live across every team they may own, times the churn allowed on top.
  It is spent on creation and is not returned.

Keyed on the owner's uuid so a recycled id cannot inherit someone else's
spend, and charged after validation so a taken username costs nothing.

* refactor: tighten the provisioning gate and budget after review

- The gate runs before the seat-cap lock. A listener has no deadline and the
  lease is ten seconds with nothing to renew it, so the lock must not be held
  across one.
- `failOpen: false` rethrows rather than returning false, so a backend outage
  is answered 503 instead of escaping the handler.
- An address goes through `validateEmail` first, as it does on every other
  path that creates an account.
- The gate always runs; a request only enriches what it is told, so one being
  absent is no reason to skip it.
- The gate is told it is looking at a seat, whose address is optional.
- A refusal carries the gate's own code and request code, as a blocked signup
  does, so support can correlate it.
- Restored `provisionAccount`'s docblock, which an insert had displaced.

Tests: the spy restores in `finally`, and the cap overrides are taken after
the fixtures, so one failure cannot cascade through the file.

* fix: show a provisioning refusal as a rule, not a failure

The console special-cases the seat cap and sends everything else to
"Something went wrong", which reads as a fault rather than an answer. The
provisioning budget refuses the same way the cap does, so it is presented
the same way.
2026-10-14 11:54:35 -04:00
Juan Fernando Castro e32bfa839d fix: count one app open per account per app per hour (#4152)
* fix: count one app open per account per app per day

`app_opens` is read two ways at once: `open_count` is a row count and
`user_count` is distinct accounts. A repeated open moves the first
without the first moving the second, and the incentive report prices
opens.

An open is now marked as it is recorded and skipped for the rest of the
day, before both the row and the event ClickHouse ingests, so the two
stores agree. The marker is set only when absent, so the first open of
the day wins a race; an unreachable marker counts the open, since
counting twice beats not counting.

* fix: count one app open per account per app per day, at the count

Dropped the row-suppression approach. `app_opens` is not only a counter:
`getRecentAppOpens` orders the desktop's recent list by `MAX(_id)`, and
`app.opened` is the socket event other tabs reorder on. Suppressing the
repeat row froze both at the first open of the day.

Every open is recorded as before. `open_count` counts each account once
per day instead, which is the figure `user_count` is read against.

`countDistinctUserDays` folds the pair into one number because
`COUNT(DISTINCT a, b)` is MySQL-only, and `dayBucket` covers MySQL's
decimal division. ClickHouse has uniqExact over the tuple.

Note the figure's meaning changes at deploy: a period spanning it mixes
rows with account-days, so a comparison across that point is not one.

* fix: count one app open per account per app per hour

Review feedback: a day is too long a window for the repeat to collapse
into. The period is now an argument rather than a baked-in 86400, held in
one constant on AppStore beside the other store policy numbers, so the SQL
and ClickHouse sites cannot drift and a future change is one line.

The fold `COUNT(DISTINCT user * N + bucket)` is only injective while the
bucket index stays under N. Hourly buckets are 24x the day index and pass
the old 1000000 in 2084, so N moves to 10000000.

Buckets are clock hours, not a rolling window, so two opens a minute apart
either side of the hour still count twice — tested, and the same property
the day version had at UTC midnight.

The detailed-stats fixture gave its repeat open a 60s offset, which under
hourly buckets would have counted twice whenever the suite ran in the first
minute of an hour. It now shares a timestamp with the open it repeats.
2026-10-14 11:54:17 -04:00
Daniel Salazar ff815f730d fix: harden token delivery, session revokes, app identity and fs scoping (#4229)
- relay sign-in popups take the opener origin from the referrer only
- session revokes read the primary, batch per user, and tombstone caches in every region
- wisp relay tokens bound to the minting session; verify rechecks the account
- hosted origins keep app identity with the site owner; deleted app uids recorded in deleted_apps
- subdomain cache tombstoned on delete, update, reclaim and app/account cascades
- email lookups require an exact address match after the SQL lookup
- workers deploy only from files the deployer owns
- share_app_data:false enforced at ACL time, plus copy/move/rename/share/recursive listing
- kv remove returns no value without read; kv:expire op grant usable
- fs errors name the entry instead of the owner's path
- shared-by-me drops rows a delegate no longer has authority over
- private apps can't move index_url onto another user's origin
- GUI: default apps saved and launched by uid; save picker accepts only copy/move; permission dialog waits for focus; Publish as Worker hidden on others' files
2026-10-14 06:56:44 -07:00
Juan Fernando Castro 84aaf005ec fix: give every route behind the user-protected gate a password budget (#4150)
* fix: give every route behind the user-protected gate a password budget

Five of the six routes behind the gate declare a per-user hourly bucket
for the password check. The sixth, delete-own-user, declared none.

The bucket moves into the gate itself, so it covers every route behind
it and a new one cannot be added without one.

It is charged by the outcome rather than by the request: a wrong answer
spends one and a right answer spends nothing, so a caller who knows the
password never meets the limit.

* fix: the review's findings on the password budget

Reading the budget and spending it on the outcome bounded rounds rather
than attempts: everything already in flight read the same budget and was
answered before any of it was spent. The charge lands before the
comparison now, so a burst is bounded by the same number as a sequence.

The key gained a route component. One bucket across all six meant ten
mistyped passwords on one route closed the other five for the hour,
including the one an account owing its team's 2FA has to reach.
2026-10-14 09:52:23 -04:00
Daniel Salazar f6a1e54642 fix(fs): keep S3 read stream errors from crashing the process (#4227)
getObjectStream now attaches an 'error' listener to every body it returns.
A body that fails before its consumer attaches (socket timeout, reset, or a
checksum mismatch on a truncated read) used to emit 'error' with no listener
and exit the process, failing every in-flight request on the task. It now
logs a warning; consumers that attach later still receive the error.
2026-10-13 17:42:28 -07:00
Filip Kujundžić d211a25c6d Merge pull request #4218 from HeyPuter/codex/put-1851-deepseek-peak
Protect direct DeepSeek peak-only pricing
2026-10-14 00:54:11 +02:00
404oops 2e1bf86d2b test(ai): protect direct DeepSeek peak-only pricing 2026-10-13 20:44:00 +02:00
Neal ShahandClaude Opus 5.5 4401b4c4dc Fix worker test (#4156)
* PUT-2012 fix broadcast service drops

* Revert pre-commit lint churn on files merged from main

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix test bug

* Remove other test

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
2026-10-13 14:10:10 -04:00
Neal Shah 62270ff9aa Delete worker sessions when workers are deleted, don't treat sessions from deleted workers as still valid. (#4163) 2026-10-13 12:43:45 -04:00
467 changed files with 48829 additions and 19332 deletions

No files matched your search

@@ -22,7 +22,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
+3 -3
View File
@@ -57,7 +57,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
@@ -88,7 +88,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
@@ -154,7 +154,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
+1 -1
View File
@@ -16,7 +16,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
+3 -3
View File
@@ -57,7 +57,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
@@ -78,7 +78,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
@@ -135,7 +135,7 @@ jobs:
if: needs.changes.outputs.puterjs == 'true'
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
+26 -8
View File
@@ -17,6 +17,21 @@ Out-of-scope:
- Denial of Service (DoS), spam, or volumetric attacks.
- Physical security issues.
## Eligibility
We pay bounties for issues that threaten user security and privacy, or the security and health of the platform. We appreciate every report, but hardening suggestions and correctness bugs are not eligible, even when we fix them. Please open a GitHub issue for those.
A report must show how an attacker can directly perform the action against another account or the system. It is not eligible if it:
- Starts from access the attacker should not already have: a stolen session or token, the victim's password, a root or full-access token, or script execution on `puter.com`.
- Needs the victim to do more than ordinary use (opening a link or an app), such as publishing their own secrets, sending a verification link to an address they don't control, or approving a prompt that accurately states what it grants.
- Gives the attacker nothing they couldn't already get through a supported API.
- Only reproduces in a non-default or development setup (SQLite, a `custom` OIDC provider, an unset outbound proxy, cookies or config planted by hand). Findings in the open-source code must affect `puter.com` or a default self-hosted install.
**Shared state.** Sharing files, KV entries, or cross-app KV access means the sharer trusts the sharee. Exploits that depend on that trust, such as a sharee misusing access the sharer granted, may be treated as hardening or lower severity.
Hardening we ship in a public PR does not make similar code paths vulnerabilities. They are judged by these same rules.
## Known Non-Issues (Please Check Before Submitting)
The following have already been reviewed and determined **not to be vulnerabilities**. Reports that only re-describe one of these are **not eligible for a reward** and will be closed as non-issues — even if they include new code references. Please review this list before submitting:
@@ -27,11 +42,12 @@ The following have already been reviewed and determined **not to be vulnerabilit
- **Apps invoking drivers, creating workers, or using KV.** Applications are intended to do this; worker permissions are scoped to the owning app. This is by design.
- **App metadata or app user-count "leaks".** This information is currently public by design. This covers **private apps** too: private apps are an unreleased, admin-only beta for paid listings in the Puter App Center. Their `index_url` and other metadata are not secret, the flag is not user-settable, and no production app relies on it. Reports that a private app's metadata is reachable through a sibling route (taskbar items, `/get-launch-apps`, `index_url` drift on the hosting gate, etc.) are normalization work, not vulnerabilities.
- **Worker hot-reload driven by a share recipient.** Sharing a worker's source file with write access is how you let a collaborator change that worker's code and have it deploy live. The redeploy running as the file owner, with the owner's worker credentials, is the intended behavior of that grant — the owner chose to hand out write access to executable code. If you do not want that, do not share the source file with write access.
- **Re-claiming a site or worker name its owner deleted.** A hosted site or worker name that its owner deletes is treated as abandoned by that owner. If one of their apps still points at it, whoever claims the name next serves that app's `index_url` and receives what the app is launched with, and a new app on the same origin may share the old one's identity. Guarding app identities against being recycled this way is ongoing hardening, not a security boundary, so new ways to re-claim an abandoned name, or to inherit an app's tokens or data through one, are not eligible.
- **Share-mode downgrades that do not cascade.** Demoting a delegate does not retroactively revoke the grants that delegate already issued downstream. Shares are independent grants once issued, not a live-evaluated chain. Revoking the downstream grant is a separate, supported action.
- **Concurrency / TOCTOU overspend of credits or storage quota.** Firing concurrent requests to exceed an AI credit hold or a storage quota by some margin is known and accepted; the per-account concurrency and rate-limit gates are the control we rely on, and the residual overage is bounded and billed. Report these only if you can demonstrate _unbounded_ overspend.
- **Metering, credit-hold, and quota inaccuracies.** Credit holds and quota checks are up-front estimates, and actual usage is billed afterwards. Overspend that is bounded and still billed is a correctness bug, including concurrent requests that exceed a hold or quota by some margin; the per-account concurrency and rate-limit gates are the control we rely on. Report these only if you can use a service without it being billed, or overspend without bound.
- **Existence-only oracles.** A response that tells "exists but forbidden" apart from "not found" for a path or ID the caller already supplied, without revealing anything else about it, is hardening.
- **Session and credential lifecycle around 2FA enrollment.** Enrolling in 2FA is not a credential-revocation event: existing authenticated sessions stay valid, and outstanding password-reset links stay valid. Likewise, account-management actions behind the `/user-protected/*` gate are password-gated by design, not TOTP-gated. A recovery code that stops being accepted fails _closed_ — it denies access rather than granting it, so it is a bug report, not a vulnerability.
- **Group permissions** (`/auth/revoke-user-group` and siblings). The group-permission surface is unfinished and not exposed anywhere in the product. Findings there are not reachable by any user.
- **Correctness bugs with no security boundary crossed.** For example, an API that reports success while silently leaving data behind. We want these reported — please open a GitHub issue — but they are not bounty-eligible.
- **Missing Subresource Integrity (SRI) on external scripts/styles.** Accepted as hardening, not as a vulnerability, absent a demonstrated compromise of the CDN in question.
- **General "token in a URL" / token-lifetime / token-storage designs** — signed directory URLs exposing children, a write signature implying read, app tokens outliving a web session, or the session token being held in `localStorage` on `puter.com`. These are current intended behaviors. Shorter-lived tokens with a refresh flow are on the roadmap; their absence is not a vulnerability on its own. Show a reproducible attack that does not already presuppose script execution on `puter.com`.
- **Missing PKCE, unverified `id_token` signatures, or other OIDC hardening.** Provider tokens are obtained through a server-to-server authorization-code exchange with the provider's token endpoint over TLS, so the resulting `id_token` / userinfo claims are trusted from that channel rather than from local JWKS signature verification — the callback never accepts a caller-supplied `id_token`. Adding local signature / `aud` / `iss` / `exp` checks is welcome defense-in-depth (please open a GitHub issue/PR), but their absence is not an account-takeover vector on its own.
@@ -52,7 +68,7 @@ To participate, you must:
3. **Respect availability**: Do not perform denial-of-service attacks or automated scans that degrade service.
4. **Follow disclosure policy**: Do not publicly disclose vulnerabilities until we have confirmed and patched the issue.
5. **Act in good faith**: Make every effort to avoid privacy violations, destruction of data, and interruption or degradation of services.
6. **Check the Known Non-Issues list**: Reports matching an item in the "Known Non-Issues" section above are not eligible and will be closed.
6. **Check Eligibility and Known Non-Issues**: Reports that fall outside "Eligibility" or match an item in "Known Non-Issues" above are not eligible and will be closed.
Reports that do not meet these guidelines may not be eligible for a reward.
@@ -63,14 +79,14 @@ Reports **MUST** demonstrate a **working, reproducible exploit with real impact*
- Exact steps to reproduce, the relevant request/response or code path, and the commit or version you tested.
- The **observed** result versus the **expected** result.
- For client-side, UI, or authentication-flow bugs: a short screen recording (≤ 2 minutes) showing the exploit working end-to-end on a real Puter instance.
- For server-side bugs: a runnable proof-of-concept and screenshot or video showing the attack process.
- For server-side bugs: a runnable proof-of-concept and screenshot or video showing the attack process. A unit test or in-process harness alone does not count; show it against a running instance.
Reports based solely on reading the source ("source review only, not tested") or unverified AI/LLM-generated reports are the **lowest triage priority and not eligible for bounty**.
If you used an AI tool to help find an issue, you must personally verify that it actually reproduces before submitting.
If you used an AI tool to help find an issue, you must personally verify that it actually reproduces before submitting. Reports sent by automated agents are not eligible.
Please submit **one issue per report**.
Bundled "audit packs" of many speculative findings will not be elligible for bounty; send each confirmed issue separately.
Additionally please **DO NOT** submit too many issues at once; these will be treated as a bundled report, and will not be elligible for bounty.
Bundled "audit packs" of many speculative findings will not be eligible for bounty; send each confirmed issue separately.
Additionally please **DO NOT** submit too many issues at once; these will be treated as a bundled report, and will not be eligible for bounty.
## Reporting Process
@@ -83,6 +99,8 @@ Include:
- Suggested remediation (if available)
- Paypal id where we can send the bounty (if eligible)
Send the full report in your first email. We don't reply to messages that only ask whether we run a bounty or where to report.
We aim to acknowledge receipt within **72 hours** and provide a resolution timeline.
## Reward Structure
@@ -95,7 +113,7 @@ We offer monetary rewards based on the severity of the vulnerability, as determi
- **Low: \$50 – \$100**
Non-security issues, suggestions, and best practices feedback are always welcome, but may not qualify for a reward.
If multiple researchers report the same issue, the bounty will be awarded to the first eligible report we receive.
If multiple researchers report the same issue, the bounty will be awarded to the first eligible report we receive. Issues we already know about, from internal review, an open ticket, or a fix already in a public commit or PR, count as duplicates.
## Payments Disclaimer
+2 -2
View File
@@ -14,7 +14,7 @@
# config.default.json, so partial overrides work. Absent file = defaults.
# ---- Build stage ----
FROM node:24-slim AS build
FROM node:26-bookworm-slim AS build
WORKDIR /opt/puter
@@ -59,7 +59,7 @@ RUN set -e; \
wait $pjs_pid
# ---- Runtime stage (slim — no build tools) ----
FROM node:24-slim
FROM node:26-bookworm-slim
WORKDIR /opt/puter
+9 -5
View File
@@ -7,9 +7,8 @@
// Each setting lives at exactly one canonical key — there are no fallback
// aliases. Values shown are illustrative, not production secrets.
//
// Keys consumed only by closed-source / hosted-prod extensions (clickhouse,
// cacheUpdateHandler, pages, prodMeteringAndBilling, …) are intentionally
// omitted.
// Keys consumed only by closed-source / hosted-prod extensions are
// intentionally omitted.
// ── Environment / identity ──────────────────────────────────────────
"config_name": "template",
@@ -35,7 +34,7 @@
// Idle keep-alive timeout in ms. Must stay above the idle timeout of any
// proxy in front, or the proxy reuses connections this server has closed
// and its clients see 502s.
"keep_alive_timeout": 620000,
"keep_alive_timeout": 920000,
"protocol": "http",
"domain": "puter.localhost",
// Fully-qualified externally-visible URL. Computed from protocol/domain/
@@ -314,7 +313,8 @@
// The tables ship either way and sit inert; this gates whether `/teams`
// is registered at all, so with it off the paths do not exist rather than
// existing and refusing. It is also the backout: turning it off removes
// the feature without touching data.
// the feature without touching data. Teams ships as an extension; on,
// the server refuses to boot unless one registers it.
"teams_enabled": false,
//
// Live teams one user may own. Soft-deleted ones do not count, so
@@ -451,6 +451,10 @@
"apiKey": "",
"apiBaseUrl": "https://ark.ap-southeast.bytepluses.com/api/v3"
},
// Kling AI. One API key (not the legacy AccessKey/SecretKey pair)
// powers video generation, image generation and TTS; `apiBaseUrl`
// defaults to https://api-singapore.klingai.com.
"kling": { "apiKey": "" },
"zai": { "apiKey": "" },
"alibaba": { "apiKey": "" },
"hoonify": {
+6
View File
@@ -45,6 +45,12 @@ Both are supported ways to define an API. **Prefer a controller when you need fi
- **A specific verified factor:** `requireVerified` (email, and only under `strict_email_verification_required`), `requirePhoneVerified`, `requireCardVerified`. Opt-in, and the factor must actually have been verified, unlike the default-on gate that turns away accounts still carrying a pending verification the abuse harness asked for. `requireAnyVerified: ['phone', 'card']` is the OR of the last two: any listed factor verified at any point passes, as does a paid plan when `card` is listed (a paying account already has a card on file). Otherwise only the factors the deployment can verify are asked for (an SMS provider configured, a card gate an extension reports on); with none verifiable the gate is inert rather than locking the route on a self-hosted install. A denial carries the first verifiable factor's code plus `factors` (the verifiable ones, in the route's order), so a client can lead with one flow and offer the other. `verifiedFactorGate.enabled: false` in config stops every declared gate at once, e.g. during an SMS-provider outage.
- **Metered spend:** an endpoint that spends metered resources on the caller's behalf (moving file content, object-store requests, anything else the account is billed for) declares `requireCredits: true`, which turns an account with no budget left away with a 402 before the handler runs. Endpoints that only describe or delete things deliberately don't: an account that has run out still has to see what it has, clear it, and reach its billing pages. Drivers have no route options, so they call `assertActorHasCredits` themselves ([src/backend/services/metering/enforcement.ts](../src/backend/services/metering/enforcement.ts)) — see `KVStoreDriver`, which does it once for every method.
Provider sign-in requires the account's authenticator or a recovery code when 2FA is enabled, including signup callbacks that resolve an existing account. The five-minute challenge uses the same single-use `/login/otp` and `/login/recovery-code` verification as password sign-in. The GUI carries it in a URL fragment and removes it before initialization. A session and successful popup-return proof are issued only after verification; cancelling retains the popup's signed retry context without marking it authenticated.
Disabling 2FA through `/user-protected/disable-2fa` requires both password or OIDC revalidation and the current six-digit authenticator `code`. Missing, malformed, or incorrect codes leave the secret and recovery codes intact. A team's requirement for 2FA still prevents disabling it.
Email changes through `/user-protected/change-email` require the password or OIDC revalidation cookie, plus a six-digit `code` from the authenticator when 2FA is enabled. Missing or malformed codes return `400 bad_request`; incorrect codes return `401 code_mismatch`. Both include `factor: 'otp'`. Confirmation links are bound to that verified request. A 2FA-enabled account must request a new link if its existing link was issued without second-factor verification.
### 3. puter.js
- Add the method to the matching module in [src/puter-js/src/modules/](../src/puter-js/src/modules/), matching the calling conventions of its siblings (promise-returning; positional shortcut plus options form where that's the local pattern).
+37
View File
@@ -309,4 +309,41 @@ describe('metering extension — handleMeteringAllCosts', () => {
// per request).
expect(costsA).toBe(costsB);
});
it('refreshes a warm catalogue at the UTC price boundary', async () => {
vi.useFakeTimers({ toFake: ['Date'] });
try {
vi.setSystemTime(new Date('2026-11-30T23:59:59Z'));
const before = makeRes();
await handleMeteringAllCosts(makeReq(), before.res);
vi.setSystemTime(new Date('2026-12-01T00:00:00Z'));
const after = makeRes();
await handleMeteringAllCosts(makeReq(), after.res);
expect(
(after.captured.body as { costs: unknown[] }).costs,
).not.toBe((before.captured.body as { costs: unknown[] }).costs);
} finally {
vi.useRealTimers();
}
});
// A scheduled price change has to reach the catalogue without a restart.
it('collects the catalogue again once the cache expires', async () => {
vi.useFakeTimers({ toFake: ['Date'] });
try {
vi.setSystemTime(new Date('2026-12-02T00:00:00Z'));
const a = makeRes();
const b = makeRes();
await handleMeteringAllCosts(makeReq(), a.res);
vi.setSystemTime(Date.now() + 10 * 60 * 1000 + 1);
await handleMeteringAllCosts(makeReq(), b.res);
const costsA = (a.captured.body as { costs: unknown[] }).costs;
const costsB = (b.captured.body as { costs: unknown[] }).costs;
expect(costsB).not.toBe(costsA);
expect(costsB).toEqual(costsA);
} finally {
vi.useRealTimers();
}
});
});
+8 -4
View File
@@ -16,9 +16,11 @@ import type { Request, Response } from 'express';
const services = extension.import('service');
const stores = extension.import('store');
// Cached on first request — the underlying cost catalogues are baked into
// driver/controller source so they only change on deploy.
// Catalogues change on deploy or on a scheduled price change, so the cache
// expires rather than living as long as the process.
const ALL_COSTS_TTL_MS = 10 * 60 * 1000;
let cachedAllCosts: Record<string, unknown>[] | null = null;
let allCostsExpireAt = 0;
function collectAllCosts(): Record<string, unknown>[] {
const all: Record<string, unknown>[] = [];
@@ -187,12 +189,14 @@ export const handleMeteringGlobalUsage = async (
res.json(globalUsage);
};
// First hit walks the registries; subsequent hits serve the in-memory cache.
// A hit walks the registries when the in-memory cache is missing or expired.
export const handleMeteringAllCosts = async (
_req: Request,
res: Response,
): Promise<void> => {
if (!cachedAllCosts) {
if (!cachedAllCosts || Date.now() >= allCostsExpireAt) {
allCostsExpireAt =
(Math.floor(Date.now() / ALL_COSTS_TTL_MS) + 1) * ALL_COSTS_TTL_MS;
const raw = collectAllCosts();
const multiplier = creditMultiplierFrom(extension.config);
// Price the catalogue in the same unit the usage endpoints report:
+88 -25
View File
@@ -1,13 +1,7 @@
import type { Request, Response } from 'express';
import { installSeatStub } from '@heyputer/backend/testFixtures/seatStub';
import { v4 as uuidv4 } from 'uuid';
import {
afterAll,
beforeAll,
describe,
expect,
it,
vi,
} from 'vitest';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { makeActor, type Actor } from '../src/backend/core/actor.ts';
import { runWithContext } from '../src/backend/core/context.ts';
import { configContainer } from '../src/backend/exports.ts';
@@ -49,7 +43,6 @@ beforeAll(async () => {
create_shortcut: true,
payment_bypass: true,
},
teams_enabled: true,
} as never);
});
@@ -83,21 +76,20 @@ describe('whoami extension — handleWhoami', () => {
};
const seatOf = async (teamName: string) => {
const owner = await seedUser();
const seat = await seedSeat();
const team = await server.stores.team.create({
ownerUserId: owner.id as number,
const team = {
uid: `wt-${Math.random().toString(36).slice(2, 9)}`,
name: teamName,
handle: `wt-${Math.random().toString(36).slice(2, 9)}`,
});
await server.stores.team.addMember(team.uid, seat.id as number, {
orgOwned: true,
};
installSeatStub(server as never, [seat.id as number], {
teamUid: team.uid,
teamName: team.name,
});
return { seat, team };
};
it('names the team for a seat', async () => {
const { seat, team } = await seatOf('Acme Corp');
it('names no team where teams is off', async () => {
const { seat } = await seatOf('Acme Corp');
const { res, captured } = makeRes();
await runWithContext(
@@ -105,10 +97,8 @@ describe('whoami extension — handleWhoami', () => {
() => handleWhoami(makeReq(), res),
);
expect((captured.body as { team?: unknown }).team).toEqual({
uid: team.uid,
name: 'Acme Corp',
});
// Off, so the seat is never looked up.
expect((captured.body as { team?: unknown }).team).toBeUndefined();
});
it('says nothing for an account that is not a seat', async () => {
@@ -170,6 +160,81 @@ describe('whoami extension — handleWhoami', () => {
expect(captured.body).toEqual({ error: 'User not found' });
});
describe("an app's view of the email", () => {
const appActorFor = async (user: { uuid: string; id: unknown }) => {
const app = await (
server.stores.app.create as unknown as (
fields: Record<string, unknown>,
opts: { ownerUserId: number },
) => Promise<{ uid: string; id: number }>
)(
{
name: `we-${uuidv4()}`,
title: 'Email reader',
index_url: `https://${uuidv4()}.example/`,
},
{ ownerUserId: user.id as number },
);
return {
app,
actor: makeActor({
user: { uuid: user.uuid, id: user.id as number },
app: { uid: app.uid, id: app.id },
}),
};
};
const whoamiAs = async (actor: Actor) => {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleWhoami(makeReq(), res),
);
return captured;
};
it('is withheld until the user grants it, then shown', async () => {
const user = await seedUser();
const { app, actor } = await appActorFor(user);
const before = (await whoamiAs(actor)).body as Record<
string,
unknown
>;
expect(before).not.toHaveProperty('email');
expect(before).not.toHaveProperty('unconfirmed_email');
await server.services.permission.grantUserAppPermission(
makeActor({ user: { uuid: user.uuid, id: user.id as number } }),
app.uid,
`user:${user.uuid}:email:read`,
);
const after = (await whoamiAs(actor)).body as Record<
string,
unknown
>;
expect(after.email).toBe(user.email);
expect(after.unconfirmed_email).toBe(user.email);
});
it('answers 404 for a missing account even when the check fails', async () => {
const check = vi
.spyOn(server.services.permission, 'check')
.mockRejectedValue(new Error('scan failed'));
try {
const captured = await whoamiAs(
makeActor({
user: { uuid: 'ghost-uuid', id: 99_999_998 },
app: { uid: 'app-ghost', id: 1 },
}),
);
expect(captured.statusCode).toBe(404);
expect(captured.body).toEqual({ error: 'User not found' });
} finally {
check.mockRestore();
}
});
});
it('returns full user details for a user actor', async () => {
const user = await seedUser();
const { res, captured } = makeRes();
@@ -451,9 +516,7 @@ describe('whoami extension — handleWhoami', () => {
}),
]) {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleWhoami(makeReq(), res),
);
await runWithContext({ actor }, () => handleWhoami(makeReq(), res));
const body = captured.body as Record<string, unknown>;
expect(body.phone).toBeUndefined();
+18 -5
View File
@@ -36,6 +36,18 @@ export const handleWhoami = async (
}
const isUser = !actor.effectiveApp;
const canReadEmail = async (uuid: unknown) => {
try {
return await services.permission.check(
actor,
`user:${uuid}:email:read`,
);
} catch {
return false;
}
};
// Started beside the row read: the actor carries the row's uuid.
const emailCheck = isUser ? null : canReadEmail(actor.user.uuid);
const user = await stores.user.getById(actor.user.id);
if (!user) {
res.status(404).json({ error: 'User not found' });
@@ -62,10 +74,10 @@ export const handleWhoami = async (
// Strip sensitive fields for app actors
if (!isUser) {
const canReadEmail = await services.permission
.check(actor, `user:${user.uuid}:email:read`)
.catch(() => false);
if (!canReadEmail) {
const emailReadable = await (user.uuid === actor.user.uuid
? emailCheck
: canReadEmail(user.uuid));
if (!emailReadable) {
delete details.email;
delete details.unconfirmed_email;
}
@@ -109,7 +121,8 @@ export const handleWhoami = async (
}
const subscription = details.subscription as
{ offering?: Record<string, unknown> } | undefined;
| { offering?: Record<string, unknown> }
| undefined;
if (subscription?.offering) {
delete subscription.offering.group;
delete subscription.offering.benefits;
+110 -98
View File
@@ -16,7 +16,7 @@
"@aws-sdk/s3-request-presigner": "^3.1028.0",
"dedent": "^1.5.3",
"javascript-time-ago": "^2.5.11",
"libphonenumber-js": "1.13.14",
"libphonenumber-js": "1.13.15",
"miniflare": "^4.20260617.1",
"open": "^10.1.0"
},
@@ -1503,6 +1503,17 @@
"node": ">=10.0.0"
}
},
"node_modules/@dual-bundle/import-meta-resolve": {
"version": "4.2.1",
"resolved": "https://registry.npmjs.org/@dual-bundle/import-meta-resolve/-/import-meta-resolve-4.2.1.tgz",
"integrity": "sha512-id+7YRUgoUX6CgV0DtuhirQWodeeA7Lf4i2x71JS/vtA5pRb/hIGWlw+G6MeXvsM+MXrz0VAydTGElX1rAfgPg==",
"dev": true,
"license": "MIT",
"funding": {
"type": "github",
"url": "https://github.com/sponsors/JounQin"
}
},
"node_modules/@emnapi/core": {
"version": "1.11.1",
"resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.1.tgz",
@@ -5921,9 +5932,9 @@
}
},
"node_modules/@types/busboy": {
"version": "1.5.4",
"resolved": "https://registry.npmjs.org/@types/busboy/-/busboy-1.5.4.tgz",
"integrity": "sha512-kG7WrUuAKK0NoyxfQHsVE6j1m01s6kMma64E+OZenQABMQyTJop1DumUWcLwAQ2JzpefU7PDYoRDKl8uZosFjw==",
"version": "1.5.5",
"resolved": "https://registry.npmjs.org/@types/busboy/-/busboy-1.5.5.tgz",
"integrity": "sha512-IwqrfVFxWa23c4WeE7uZswssCqGfvnl5Q7b1pdshbT3HaQDa3UIbJx5BAxtcj9nZYX9tVDArM9TUv9SpvORbpQ==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -6526,14 +6537,14 @@
}
},
"node_modules/@vitest/coverage-v8": {
"version": "4.1.11",
"resolved": "https://registry.npmjs.org/@vitest/coverage-v8/-/coverage-v8-4.1.11.tgz",
"integrity": "sha512-8MVGEFnJIcdGjcbfKmeq8z0pZHH0JlVtoVZH9Q/qwUp6wyFnEJUBMrw9DCaj+ra3vShGmhavjalMIhPNxZAUcw==",
"version": "4.1.12",
"resolved": "https://registry.npmjs.org/@vitest/coverage-v8/-/coverage-v8-4.1.12.tgz",
"integrity": "sha512-6S7/iVknKglLAxMFxQGNppBI0iqJS2NgM6FBxavitj7KaDq/vywOWGf6dGu7F2L1Qy0G6S3uTp7aGshHrzK8JA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@bcoe/v8-coverage": "^1.0.2",
"@vitest/utils": "4.1.11",
"@vitest/utils": "4.1.12",
"ast-v8-to-istanbul": "^1.0.0",
"istanbul-lib-coverage": "^3.2.2",
"istanbul-lib-report": "^3.0.1",
@@ -6547,8 +6558,8 @@
"url": "https://opencollective.com/vitest"
},
"peerDependencies": {
"@vitest/browser": "4.1.11",
"vitest": "4.1.11"
"@vitest/browser": "4.1.12",
"vitest": "4.1.12"
},
"peerDependenciesMeta": {
"@vitest/browser": {
@@ -6557,16 +6568,16 @@
}
},
"node_modules/@vitest/expect": {
"version": "4.1.11",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.11.tgz",
"integrity": "sha512-VX2x5vNJXET47KAFzwERI+KRMtTTCSWTfSMKsW7JsUsXV4psq++e3DvZpuTDOpHcxytiDs6p2nhVb2tVDiiUYw==",
"version": "4.1.12",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.12.tgz",
"integrity": "sha512-vFlTt+YEu9WBpNjdGayNmlMVzrWPi1+sv853/7t2FU7hpO5gAFBKHzQmkCq5Xjpr+evegF2994LQT3hF3mRlqQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@standard-schema/spec": "^1.1.0",
"@types/chai": "^5.2.2",
"@vitest/spy": "4.1.11",
"@vitest/utils": "4.1.11",
"@vitest/spy": "4.1.12",
"@vitest/utils": "4.1.12",
"chai": "^6.2.2",
"tinyrainbow": "^3.1.0"
},
@@ -6575,13 +6586,13 @@
}
},
"node_modules/@vitest/mocker": {
"version": "4.1.11",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.11.tgz",
"integrity": "sha512-2XJVD55d1o5AZous5CCGKS74g/riOj9odEt2bQpCVZeblHyHdnMeFl4jl0XjU21stf4mbjUkew2eXQZt65g5CQ==",
"version": "4.1.12",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.12.tgz",
"integrity": "sha512-fKShNpKZzupWZuY0z7egydvL0sqW4INEZBEExMBMAa6izlENEVHwoXgdaTCaBZ5fwlr4xf2o1wXC83n9RpO+NA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/spy": "4.1.11",
"@vitest/spy": "4.1.12",
"estree-walker": "^3.0.3",
"magic-string": "^0.30.21"
},
@@ -6602,9 +6613,9 @@
}
},
"node_modules/@vitest/pretty-format": {
"version": "4.1.11",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.11.tgz",
"integrity": "sha512-yiZzPbGTS9Sr/JpFl8zHrcIkAofNbFV6k21vIgQN/cY/oxZeXhJv5sc/MBJ5jFKWmWs+oJHw0UXLZjmf931+Vw==",
"version": "4.1.12",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.12.tgz",
"integrity": "sha512-XW9O8+Mxx4VpCtkYdMJLZQQwsVryP9wk7cPKDxtXFfwSak6yRh5+2fep/Y0XF/ElBvJHq8bcYdi6aeuai0opZA==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -6615,13 +6626,13 @@
}
},
"node_modules/@vitest/runner": {
"version": "4.1.11",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.11.tgz",
"integrity": "sha512-LztvUgdwMNJMIkj3hQnnxiC2Xy1zNxq928W/xhjCLaNCzqTZOudjwbQf6v9IntZGPw132i2Lq2rgTRZHD3JHNw==",
"version": "4.1.12",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.12.tgz",
"integrity": "sha512-NKlVKCr5QnCe8jijdMnmkpHHU+WTqiLFR6G0pHji9uRjXK9xJl1k1nTw6F3pOOj2PDcsDt2GKQMI7wIsaT491w==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/utils": "4.1.11",
"@vitest/utils": "4.1.12",
"pathe": "^2.0.3"
},
"funding": {
@@ -6629,14 +6640,14 @@
}
},
"node_modules/@vitest/snapshot": {
"version": "4.1.11",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.11.tgz",
"integrity": "sha512-pN7ikn1ON7h8ee4gIAp4AzyK+zBtJPzVbqOgu5LCEh4VaJVbPQcgYQYJIMGQPXVeJJq1fnfazis7a5pFNPahog==",
"version": "4.1.12",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.12.tgz",
"integrity": "sha512-ko1y1Qgb/slzQ9mr+b89Cuqud8WG1TKtsPD8r7ZUBdvVOGt85Af2HwCqHujrhwJq95hs/9RM7fcAPg57o1vlzw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "4.1.11",
"@vitest/utils": "4.1.11",
"@vitest/pretty-format": "4.1.12",
"@vitest/utils": "4.1.12",
"magic-string": "^0.30.21",
"pathe": "^2.0.3"
},
@@ -6645,9 +6656,9 @@
}
},
"node_modules/@vitest/spy": {
"version": "4.1.11",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.11.tgz",
"integrity": "sha512-apNa/prQy2qCeywhnixOHPRCgGNhvg7T4Dapfl1GahLp/R+uhBm5cPyFoNVyqsNd2h1nJxL6BqqdIjiABL60YA==",
"version": "4.1.12",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.12.tgz",
"integrity": "sha512-RGq9x4JNGNTZHEJxvN0oSiJjeSCExm0n2gZKKJZHY9cX5AVA6M7VJILBT3ZJ/tJ/0J9uWT7fETV9/btoi//vSg==",
"dev": true,
"license": "MIT",
"funding": {
@@ -6655,13 +6666,13 @@
}
},
"node_modules/@vitest/utils": {
"version": "4.1.11",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.11.tgz",
"integrity": "sha512-zTCVGpyFsGWBhllOyKlTw/vnr6D9qxsfSDyfbyZmTyjHw5N/VuvzHpHoQjm2ZJzn4RJgx5w4r7V0er69CmLgPQ==",
"version": "4.1.12",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.12.tgz",
"integrity": "sha512-Q3awXrwOfEV+qc3WmO5ONyE0yO7Q1gQ9/oyHxGLZ/pl7N3NO/ZztqQwKLuwMkV+CA1JYSIeO3gkHnRDIo4AVaA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "4.1.11",
"@vitest/pretty-format": "4.1.12",
"convert-source-map": "^2.0.0",
"tinyrainbow": "^3.1.0"
},
@@ -7768,9 +7779,9 @@
"license": "CC-BY-4.0"
},
"node_modules/chai": {
"version": "6.2.2",
"resolved": "https://registry.npmjs.org/chai/-/chai-6.2.2.tgz",
"integrity": "sha512-NUPRluOfOiTKBKvWPtSD4PhFvWCqOi0BGStNWs57X9js7XGTprSmFoz5F0tWhR4WPjNeR9jXqdC7/UpSJTnlRg==",
"version": "6.3.0",
"resolved": "https://registry.npmjs.org/chai/-/chai-6.3.0.tgz",
"integrity": "sha512-XWAtwJ6OHO+tj0EKCs0Y2UamnyOxseZWltU4x2U2wh8g4AigdjwvtUjvLP2tqkA/avxHEtzxNaqGq/YGNwckKg==",
"dev": true,
"license": "MIT",
"engines": {
@@ -9125,14 +9136,14 @@
}
},
"node_modules/eslint-plugin-prettier": {
"version": "5.5.6",
"resolved": "https://registry.npmjs.org/eslint-plugin-prettier/-/eslint-plugin-prettier-5.5.6.tgz",
"integrity": "sha512-ifetmTcxWfz+4qRW3pH/ujdTq2jQIj59AxJMIN26K5avYgU8dxycUETQonWiW+wPrYXA0j3Try0l1CnwVQtDqQ==",
"version": "5.5.7",
"resolved": "https://registry.npmjs.org/eslint-plugin-prettier/-/eslint-plugin-prettier-5.5.7.tgz",
"integrity": "sha512-WjHyn3GrFd+ZR0rzsLiM4zJZ35PZMOE0SUIxzGmnWNG0126yNfJKJzObmmkYDwHJfGlkz3PSlzyG9ez/LJntwg==",
"dev": true,
"license": "MIT",
"dependencies": {
"prettier-linter-helpers": "^1.0.1",
"synckit": "^0.11.13"
"synckit": "^0.12.3"
},
"engines": {
"node": "^14.18.0 || >=16.0.0"
@@ -9881,9 +9892,9 @@
"license": "ISC"
},
"node_modules/follow-redirects": {
"version": "1.16.0",
"resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.0.tgz",
"integrity": "sha512-y5rN/uOsadFT/JfYwhxRS5R7Qce+g3zG97+JrtFZlC9klX/W5hD7iiLzScI4nZqUS7DNUdhPgw4xI8W2LuXlUw==",
"version": "1.16.1",
"resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.1.tgz",
"integrity": "sha512-FNvFGzoMLWmE6Yj9spb/zjd7yiNCHiAW9/Tg9CXrQ8wuu32HtlJOwWO11OJafl5FfY3DxTdQ0vj42zU1kvv5jg==",
"funding": [
{
"type": "individual",
@@ -11523,9 +11534,9 @@
}
},
"node_modules/libphonenumber-js": {
"version": "1.13.14",
"resolved": "https://registry.npmjs.org/libphonenumber-js/-/libphonenumber-js-1.13.14.tgz",
"integrity": "sha512-llihgCcx0BFLksecLP+x1J+6JDE1GsXS1RN/LoPF6qcwpeQcnjj0lcvZxY8AzbEpYwyZWPZW/nDuqkqzm3amiw==",
"version": "1.13.15",
"resolved": "https://registry.npmjs.org/libphonenumber-js/-/libphonenumber-js-1.13.15.tgz",
"integrity": "sha512-AHdhEQeDFEJ4GKDAQ9AwxChy+v2kUILUB3TGQRpkXMkWePjXTU/WQEzWVQoIx/Gz8fE73Wr5SEuZCy2vk8jrMQ==",
"license": "MIT"
},
"node_modules/light-my-request": {
@@ -13363,9 +13374,9 @@
"license": "MIT"
},
"node_modules/mysql2": {
"version": "3.24.5",
"resolved": "https://registry.npmjs.org/mysql2/-/mysql2-3.24.5.tgz",
"integrity": "sha512-X6Ujsr2QSkkLpkQGjxzpKRAPn9nu4axpR63ntBzquFVEvPOArgbUQ1sJjFKI7hnaYtiVZxa17Z7q18KSubW0IQ==",
"version": "3.24.6",
"resolved": "https://registry.npmjs.org/mysql2/-/mysql2-3.24.6.tgz",
"integrity": "sha512-enu73KhbAua2L3dIxjc6qg6ZJ7H3pt1BImZASWNWY5xSsmJ1ZuUPULTDGXgmJbI61vb+nJAG19IZclIq19UidQ==",
"license": "MIT",
"dependencies": {
"aws-ssl-profiles": "^1.1.2",
@@ -14367,9 +14378,9 @@
}
},
"node_modules/prettier": {
"version": "3.9.9",
"resolved": "https://registry.npmjs.org/prettier/-/prettier-3.9.9.tgz",
"integrity": "sha512-Z/CJHIkdujO/OtN7nXUii0Rf3VT5SRuhjBA82Xvu2XhBUgX3nhP67T0LHceBdQLex7OOFGTox+Q5Yg8Jk2Qivg==",
"version": "3.9.10",
"resolved": "https://registry.npmjs.org/prettier/-/prettier-3.9.10.tgz",
"integrity": "sha512-skGHTH1Yb9QvaTC4hWnRTzEYtCpApF0UMBPz+t/0ZPzP8ihefo/btKd+QA7pUE0fEwA1SbS/NmhyEcstdXOIdA==",
"dev": true,
"license": "MIT",
"bin": {
@@ -15578,13 +15589,13 @@
"license": "MIT"
},
"node_modules/smtp-server": {
"version": "3.19.17",
"resolved": "https://registry.npmjs.org/smtp-server/-/smtp-server-3.19.17.tgz",
"integrity": "sha512-/9RwyNTcR8osMQWOvC8NV8l57ddPyYXvuG+wa1Ua4cD/I41z7zCMsGtXLa9vxlrggjRrfejr50X0WS/9NuVWqQ==",
"version": "3.19.23",
"resolved": "https://registry.npmjs.org/smtp-server/-/smtp-server-3.19.23.tgz",
"integrity": "sha512-t6AOBe7RR9Y/9Kox4e5SIm7wS2ZA8P0VuhVEZ5Yiox0WyevX+rMjATzIwEX3A+B6SGwnJ8QvoHtJAhbJYbjy3A==",
"license": "MIT-0",
"dependencies": {
"ipv6-normalize": "1.0.1",
"nodemailer": "10.0.14",
"nodemailer": "10.1.2",
"punycode.js": "2.3.1"
},
"engines": {
@@ -15592,9 +15603,9 @@
}
},
"node_modules/smtp-server/node_modules/nodemailer": {
"version": "10.0.14",
"resolved": "https://registry.npmjs.org/nodemailer/-/nodemailer-10.0.14.tgz",
"integrity": "sha512-eJoLFKg55fesSlzUekmfOv/SBXh4KUYT8oiQc2RLP0tfG0rbNMJBK7wgHJwXTJwqyKlSJMz9UzDZ4v+aHEvo8g==",
"version": "10.1.2",
"resolved": "https://registry.npmjs.org/nodemailer/-/nodemailer-10.1.2.tgz",
"integrity": "sha512-rnQn9Rzh8RAwcp42MVsIvxbsv6YZ2a2/av+uyJ/LPU1+2QSlrRgFsWWZTKenyRUqkIl8urOI0c4+CEcmFdsvQw==",
"license": "MIT-0",
"engines": {
"node": ">=20.0.0"
@@ -16003,12 +16014,13 @@
"license": "MIT"
},
"node_modules/synckit": {
"version": "0.11.13",
"resolved": "https://registry.npmjs.org/synckit/-/synckit-0.11.13.tgz",
"integrity": "sha512-eNRKgb3z66Yp3D2CixVujOUvXLFUTij/zVnV8KRyvFdQwpz7I5DS8UfRkTeLzb64u+dkzDSdelE24izu+zSSUg==",
"version": "0.12.3",
"resolved": "https://registry.npmjs.org/synckit/-/synckit-0.12.3.tgz",
"integrity": "sha512-U+/AXT6LWsxnj2vz9HeUuPWBNgdWxjyIEjEPdbJARC9txmgPzo2VEdqnFo4hHaNyMPtycyVJrSaQYeQv/PoyyA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@dual-bundle/import-meta-resolve": "^4.2.1",
"@pkgr/core": "^0.3.6"
},
"engines": {
@@ -16607,9 +16619,9 @@
"license": "MIT"
},
"node_modules/undici": {
"version": "7.30.0",
"resolved": "https://registry.npmjs.org/undici/-/undici-7.30.0.tgz",
"integrity": "sha512-dkrQXeHSaoamnItlYbmzG0wFYrM0ZwDxCIg0A7aKjTyyhh9svRzCNFEzV+Vm05/yehjCzjDZ31KXfGEjYSztDQ==",
"version": "7.30.1",
"resolved": "https://registry.npmjs.org/undici/-/undici-7.30.1.tgz",
"integrity": "sha512-03KrMbjIFDWQR5RNPrSw3bd1r12JpyFQl9/yxjUbJtqkjmllqC5lFjZh6ujeuJCKr8EOQvRW872Oy/az+nLYxg==",
"license": "MIT",
"engines": {
"node": ">=20.18.1"
@@ -16730,9 +16742,9 @@
"license": "MIT"
},
"node_modules/uuid": {
"version": "14.0.2",
"resolved": "https://registry.npmjs.org/uuid/-/uuid-14.0.2.tgz",
"integrity": "sha512-xZe/16rV4aa+HGSOCiY2YeLT1OybRLrrkL/Rqaq7p7GMVXjFh+6wN4oMYgjFmnSnhY8t6Xpdl2l9qmnHYuMHwQ==",
"version": "14.0.3",
"resolved": "https://registry.npmjs.org/uuid/-/uuid-14.0.3.tgz",
"integrity": "sha512-Sc5u1VVln93y1Xnz4R+CunOZc7R9LUcTmhj7Rc7Eg3HVcf7rVn2FOoa812fe0W8p3jD7xeoGSYbfqCGGH2B6xw==",
"funding": [
"https://github.com/sponsors/broofa",
"https://github.com/sponsors/ctavan"
@@ -16775,9 +16787,9 @@
}
},
"node_modules/vite": {
"version": "8.1.5",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.1.5.tgz",
"integrity": "sha512-7ULLwsCdYx/nRyrpiEwvqb5TFHrMVZyBt+rg/OAXT7rgj/z+DtTDyKFeLAdDkubDVDKD8jOsndmy7m55XcfUsw==",
"version": "8.1.6",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.1.6.tgz",
"integrity": "sha512-aRAHIxx5SW5zc1sMqFwJU/t5qaFfXb5Hj0J7W0VXihu/X99+346tz6080tO1T1xVIJORX/b3zVTXGFqUXh35uw==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -16853,19 +16865,19 @@
}
},
"node_modules/vitest": {
"version": "4.1.11",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.11.tgz",
"integrity": "sha512-fhACrNXUidIbGSBr5FlbuBkO7VWC1ZyLl0DO4CU2DrQoAPxX84Ysxs+HeGQpii5lZWV1Q4gBZTTu49mF+A6Edw==",
"version": "4.1.12",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.12.tgz",
"integrity": "sha512-5eWdz2yb3ajxWF3C8C4TjkXl7FGm3n2E/VlazALrQHzK7nYMSGFnCZto0bbPNl77AwkOZVLk0YnBAejUu0NiWA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/expect": "4.1.11",
"@vitest/mocker": "4.1.11",
"@vitest/pretty-format": "4.1.11",
"@vitest/runner": "4.1.11",
"@vitest/snapshot": "4.1.11",
"@vitest/spy": "4.1.11",
"@vitest/utils": "4.1.11",
"@vitest/expect": "4.1.12",
"@vitest/mocker": "4.1.12",
"@vitest/pretty-format": "4.1.12",
"@vitest/runner": "4.1.12",
"@vitest/snapshot": "4.1.12",
"@vitest/spy": "4.1.12",
"@vitest/utils": "4.1.12",
"es-module-lexer": "^2.0.0",
"expect-type": "^1.3.0",
"magic-string": "^0.30.21",
@@ -16893,12 +16905,12 @@
"@edge-runtime/vm": "*",
"@opentelemetry/api": "^1.9.0",
"@types/node": "^20.0.0 || ^22.0.0 || >=24.0.0",
"@vitest/browser-playwright": "4.1.11",
"@vitest/browser-preview": "4.1.11",
"@vitest/browser-webdriverio": "4.1.11",
"@vitest/coverage-istanbul": "4.1.11",
"@vitest/coverage-v8": "4.1.11",
"@vitest/ui": "4.1.11",
"@vitest/browser-playwright": "4.1.12",
"@vitest/browser-preview": "4.1.12",
"@vitest/browser-webdriverio": "^4.1.11",
"@vitest/coverage-istanbul": "4.1.12",
"@vitest/coverage-v8": "4.1.12",
"@vitest/ui": "4.1.12",
"happy-dom": "*",
"jsdom": "*",
"vite": "^6.0.0 || ^7.0.0 || ^8.0.0"
@@ -17704,7 +17716,7 @@
"lorem-ipsum": "^2.0.8",
"mime-types": "^2.1.35",
"murmurhash": "^2.0.1",
"mysql2": "^3.24.5",
"mysql2": "^3.24.6",
"nodemailer": "^9.1.1",
"openai": "^6.34.0",
"otpauth": "^9.5.2",
@@ -17712,19 +17724,19 @@
"postal-mime": "^3.0.1",
"replicate": "^1.0.0",
"sharp": "^0.35.5",
"smtp-server": "^3.19.17",
"smtp-server": "^3.19.23",
"socket.io": "^4.8.4",
"svg-captcha": "^1.4.0",
"together-ai": "^0.33.0",
"ua-parser-js": "^1.0.41",
"undici": "^7.29.1",
"uuid": "^14.0.2",
"undici": "^7.30.1",
"uuid": "^14.0.3",
"validator": "^13.15.35"
},
"devDependencies": {
"@types/bcrypt": "^6.0.0",
"@types/better-sqlite3": "^7.6.13",
"@types/busboy": "^1.5.4",
"@types/busboy": "^1.5.5",
"@types/express": "^5.0.0",
"@types/jsonwebtoken": "^9.0.10",
"@types/mime-types": "^2.1.4",
@@ -17735,7 +17747,7 @@
"@types/validator": "^13.15.10",
"pgmock": "^1.0.3",
"typescript": "^5.9.3",
"vite": "^8.1.5",
"vite": "^8.1.6",
"vitest": "^4.0.14"
}
},
+1 -1
View File
@@ -77,7 +77,7 @@
"@aws-sdk/s3-request-presigner": "^3.1028.0",
"dedent": "^1.5.3",
"javascript-time-ago": "^2.5.11",
"libphonenumber-js": "1.13.14",
"libphonenumber-js": "1.13.15",
"miniflare": "^4.20260617.1",
"open": "^10.1.0"
},
+135
View File
@@ -0,0 +1,135 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { v4 as uuidv4 } from 'uuid';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import type { PuterServer } from './server.ts';
import { createTestUser, setupTestServer } from './testUtil.ts';
/**
* Teams is optional, so core has to work when neither the service nor the store
* is registered, which is how this test server boots.
*/
describe('core without a teams registration', () => {
let server: PuterServer;
const makeUser = async (): Promise<{ id: number; username: string }> => {
const username = `nt_${Math.random().toString(36).slice(2, 10)}`;
const created = (await server.stores.user.create({
username,
uuid: uuidv4(),
password: null,
email: `${username}@test.local`,
})) as unknown as { id: number };
return { id: created.id, username };
};
beforeAll(async () => {
server = await setupTestServer({
// Without this the allowlist check returns before asking Teams.
teams_allowed_email_domains: ['test.local'],
} as never);
});
afterAll(async () => {
await server?.shutdown();
});
it('has no teams in the registry', () => {
expect(server.services.team).toBeUndefined();
expect(server.stores.team).toBeUndefined();
});
it('refuses to boot when the config asks for teams', async () => {
await expect(
setupTestServer({ teams_enabled: true } as never),
).rejects.toThrow(/teams_enabled/);
});
it('deletes an account without asking what it was charged for', async () => {
const user = await makeUser();
await expect(
server.services.userAccount.cascadeDelete(user.id),
).resolves.not.toThrow();
expect(await server.stores.user.getById(user.id)).toBeFalsy();
});
it('answers the address-claim question signup asks', async () => {
const user = await makeUser();
const row = (await server.stores.user.getById(user.id))!;
// Nothing is a seat, so the holder is returned as-is.
await expect(
server.services.signup.signupEmailHolder(row.email as string, {
releaseSeat: true,
}),
).resolves.toMatchObject({ id: user.id });
});
it('lists what is shared with an account, folding in no team shares', async () => {
const user = await makeUser();
await expect(
server.services.share.listSharedWithMe({
user: { id: user.id },
} as never),
).resolves.toMatchObject({ items: [] });
});
it('refuses a share aimed at a team, rather than failing', async () => {
// A real home directory, so the refusal is the recipient's and not
// the entry's.
const username = `nt_${Math.random().toString(36).slice(2, 10)}`;
const creds = await createTestUser(server, {
username,
password: 'Nt-Pw-12345!',
});
const row = (await server.stores.user.getByUsername(username))!;
const user = { id: row.id, username };
const actor = { user: row } as never;
expect(creds).toBeTruthy();
// The recipient cannot exist, so this is a 404 and not a crash.
await expect(
server.services.share.share(actor, {
recipient: { teamHandle: 'nobody' },
mode: 'read',
path: `/${user.username}`,
} as never),
).rejects.toMatchObject({ statusCode: 404 });
});
it('serves the homepage shell with the teams tab off', async () => {
const user = await makeUser();
const row = (await server.stores.user.getById(user.id))!;
let html = '';
const res = { send: (out: string) => (html = out) };
await server.services.homepage.send(
{
req: { headers: {}, query: {}, get: () => undefined },
res,
actor: { user: row },
} as never,
{ title: 'Puter' } as never,
{} as never,
);
const params = JSON.parse(
/gui\((\{.*?\})\);/s.exec(html)![1].replaceAll('\\u003c', '<'),
);
expect(params.teams_ui).toBe(false);
});
});
@@ -28,6 +28,12 @@ export interface WriteResult {
insertId: number | bigint;
affectedRows: number;
anyRowsAffected: boolean;
/**
* Warnings the statement raised; nonzero means a value may have been stored
* differently from how it was sent. Absent when the engine can't vouch that
* a clean insert stored exactly what was sent, on a fresh id.
*/
warningStatus?: number;
}
export interface BatchEntry {
@@ -330,6 +336,30 @@ export class AbstractDatabaseClient extends PuterClient {
});
}
/**
* Whole `periodSeconds` buckets since the epoch, from a unix-seconds
* column. MySQL's `/` yields a decimal, which would split a bucket.
*/
periodBucket(secondsExpression: string, periodSeconds: number): string {
return this.case({
mysql: `(${secondsExpression} DIV ${periodSeconds})`,
otherwise: `(${secondsExpression} / ${periodSeconds})`,
});
}
/**
* Counts each (`userExpression`, bucket) pair once. `COUNT(DISTINCT a, b)`
* is MySQL-only, so the pair folds into one number — injective while the
* bucket stays under the multiplier, which hourly reaches in the 2980s.
*/
countDistinctUserPeriods(
userExpression: string,
secondsExpression: string,
periodSeconds: number,
): string {
return `COUNT(DISTINCT ${userExpression} * 10000000 + ${this.periodBucket(secondsExpression, periodSeconds)})`;
}
nullCoalesce(...expressions: readonly string[]): string {
if (expressions.length === 0) {
throw new Error('nullCoalesce requires at least one expression');
@@ -435,6 +435,105 @@ describe('MySQLDatabaseClient — query interface', () => {
]);
});
it.each([
{ method: 'pread', poolIndex: 0, maxTimeInQueue: 30 },
{ method: 'read', poolIndex: 1, maxTimeInQueue: 10 },
] as const)(
'$method: flushes lone reads at once until 8 are in flight, then waits $maxTimeInQueue ms and coalesces',
async ({ method, poolIndex, maxTimeInQueue }) => {
const client = await startClient({ replica: { host: 'replica' } });
const pool = createdPools[poolIndex];
const held: Array<() => void> = [];
pool.respond = (sql) =>
new Promise((resolve) => {
const sets = sql
.split(';')
.slice(0, -1)
.map((stmt) => [{ stmt }]);
held.push(() => resolve(rowSets(...sets)));
});
vi.useFakeTimers({
toFake: [
'setTimeout',
'clearTimeout',
'setImmediate',
'clearImmediate',
],
});
try {
const reads: Promise<unknown>[] = [];
for (let i = 0; i < 8; i++) {
reads.push(client[method](`SELECT ${i}`));
await vi.advanceTimersByTimeAsync(0);
}
expect(pool.calls).toHaveLength(8);
reads.push(client[method]('SELECT 8'));
await vi.advanceTimersByTimeAsync(maxTimeInQueue - 1);
reads.push(client[method]('SELECT 9'));
expect(pool.calls).toHaveLength(8);
await vi.advanceTimersByTimeAsync(1);
expect(pool.calls.map((c) => c.sql).at(-1)).toBe(
'SELECT 8;SELECT 9; SELECT 1',
);
for (const release of held.splice(0)) release();
const rows = await Promise.all(reads);
expect(rows.at(-2)).toEqual([{ stmt: 'SELECT 8' }]);
expect(rows.at(-1)).toEqual([{ stmt: 'SELECT 9' }]);
} finally {
vi.useRealTimers();
}
},
);
it.each([
{ setup: 'single node', replica: undefined, waits: true },
{ setup: 'replica', replica: { host: 'replica' }, waits: false },
])(
'$setup: 8 primary reads in flight hold back a lone read: $waits',
async ({ replica, waits }) => {
const client = await startClient(replica ? { replica } : {});
const held: Array<() => void> = [];
for (const pool of createdPools) {
pool.respond = (sql) =>
new Promise((resolve) => {
const sets = sql
.split(';')
.slice(0, -1)
.map((stmt) => [{ stmt }]);
held.push(() => resolve(rowSets(...sets)));
});
}
const sent = () => createdPools.flatMap((pool) => pool.calls);
vi.useFakeTimers({
toFake: [
'setTimeout',
'clearTimeout',
'setImmediate',
'clearImmediate',
],
});
try {
const reads: Promise<unknown>[] = [];
for (let i = 0; i < 8; i++) {
reads.push(client.pread(`SELECT ${i}`));
await vi.advanceTimersByTimeAsync(0);
}
reads.push(client.read('SELECT 8'));
await vi.advanceTimersByTimeAsync(0);
expect(sent()).toHaveLength(waits ? 8 : 9);
await vi.advanceTimersByTimeAsync(10);
expect(sent()).toHaveLength(9);
for (const release of held.splice(0)) release();
await Promise.all(reads);
} finally {
vi.useRealTimers();
}
},
);
it('sends a write on its own, outside a transaction', async () => {
const client = await startClient();
const pool = createdPools[0];
@@ -472,6 +571,23 @@ describe('MySQLDatabaseClient — query interface', () => {
});
});
it('passes the warning count through, so callers can tell a clean write', async () => {
const client = await startClient();
createdPools[0].respond = () =>
writeHeader({ insertId: 9, affectedRows: 1, warningStatus: 1 });
await expect(
client.write('INSERT IGNORE INTO `user` (`username`) VALUES (?)', [
'ada',
]),
).resolves.toEqual({
insertId: 9,
affectedRows: 1,
anyRowsAffected: true,
warningStatus: 1,
});
});
it('reports no rows affected when the header omits the counters', async () => {
const client = await startClient();
createdPools[0].respond = () => writeHeader({});
@@ -78,8 +78,13 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
});
console.log('[mysql] connected to primary');
// Read batchers on one pool share the cap on immediate flushes.
const primaryReadFlushes = { count: 0 };
this.db = this.createPrimaryBatcher(this.primaryPool);
this.dbPrimaryRead = this.createPrimaryReadBatcher(this.primaryPool);
this.dbPrimaryRead = this.createPrimaryReadBatcher(
this.primaryPool,
primaryReadFlushes,
);
if (dbConf.replica) {
this.replicaPool = this.createPool(dbConf.replica);
@@ -90,7 +95,12 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
this.configuration = Configuration.SINGLE;
}
this.dbReplica = this.createReplicaBatcher(this.replicaPool);
this.dbReplica = this.createReplicaBatcher(
this.replicaPool,
this.configuration === Configuration.SINGLE
? primaryReadFlushes
: { count: 0 },
);
await this.applyMigrationPaths(
() => this.primaryPool.promise().getConnection(),
@@ -164,12 +174,14 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
const header = result[0] as {
insertId?: number;
affectedRows?: number;
warningStatus?: number;
};
const affectedRows = header.affectedRows ?? 0;
return {
insertId: header.insertId ?? 0,
affectedRows,
anyRowsAffected: affectedRows > 0,
warningStatus: header.warningStatus,
};
}
@@ -274,9 +286,14 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
}
/** Reads that must see the primary, still coalesced. */
private createPrimaryReadBatcher(pool: Pool): SQLBatcher {
private createPrimaryReadBatcher(
pool: Pool,
inFlight: { count: number },
): SQLBatcher {
return new SQLBatcher(pool, {
maxTimeInQueue: 30,
maxInFlightImmediate: 8,
inFlight,
maxBatchSize: 5,
poolLabel: 'primary',
readOnly: true,
@@ -284,9 +301,14 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
});
}
private createReplicaBatcher(pool: Pool): SQLBatcher {
private createReplicaBatcher(
pool: Pool,
inFlight: { count: number },
): SQLBatcher {
return new SQLBatcher(pool, {
maxTimeInQueue: 10,
maxInFlightImmediate: 8,
inFlight,
maxBatchSize: 5,
poolLabel: 'replica',
readOnly: true,
+46 -11
View File
@@ -76,6 +76,7 @@ const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
export class SQLBatcher {
dbPool;
maxTimeInQueue;
maxInFlightImmediate;
maxBatchSize;
maxQueueSize;
failureThreshold;
@@ -84,7 +85,9 @@ export class SQLBatcher {
readOnly;
acquireTimeoutMs;
queue = [];
timeouts = [];
#flushTimer = null;
#flushTimerIsImmediate = false;
#inFlight;
#consecutiveFailures = 0;
#lastFailureAt = 0;
#metricAttrs;
@@ -93,6 +96,11 @@ export class SQLBatcher {
* @param {object} dbPool Mysql2 pool
* @param {object} [opts]
* @param {number} [opts.maxTimeInQueue] Ms an item may wait before flush
* @param {number} [opts.maxInFlightImmediate] While fewer flushes than this
* are in flight, a lone item flushes on the next turn instead of waiting
* maxTimeInQueue; past it, items wait and coalesce to spare connections
* @param {{ count: number }} [opts.inFlight] In-flight flush counter the
* cap reads; batchers sharing a pool can share one
* @param {number} [opts.maxBatchSize] Items coalesced per flush
* @param {number} [opts.maxQueueSize] Drop-oldest high-water mark
* @param {number} [opts.failureThreshold] Consecutive failures to open the
@@ -111,6 +119,8 @@ export class SQLBatcher {
dbPool,
{
maxTimeInQueue = 20,
maxInFlightImmediate = 0,
inFlight = { count: 0 },
maxBatchSize = 50,
maxQueueSize = DEFAULT_MAX_QUEUE_SIZE,
failureThreshold = DEFAULT_FAILURE_THRESHOLD,
@@ -122,6 +132,8 @@ export class SQLBatcher {
) {
this.dbPool = dbPool;
this.maxTimeInQueue = maxTimeInQueue;
this.maxInFlightImmediate = maxInFlightImmediate;
this.#inFlight = inFlight;
this.maxBatchSize = maxBatchSize;
this.maxQueueSize = maxQueueSize;
this.failureThreshold = failureThreshold;
@@ -185,17 +197,34 @@ export class SQLBatcher {
if (this.queue.length >= this.maxBatchSize) {
this.flush(this.queue.splice(0, this.maxBatchSize));
} else if (this.queue.length === 1) {
this.timeouts.push(
setTimeout(() => {
this.flush(this.queue.splice(0, this.queue.length));
}, this.maxTimeInQueue),
);
} else if (this.#flushTimer === null) {
this.#armFlushTimer();
}
return promise;
}
// A lone flush holds a pooled connection for a full round trip, so only
// the first maxInFlightImmediate flushes skip the wait.
#armFlushTimer() {
const run = () => {
this.#flushTimer = null;
this.flush(this.queue.splice(0, this.queue.length));
};
this.#flushTimerIsImmediate =
this.#inFlight.count < this.maxInFlightImmediate;
this.#flushTimer = this.#flushTimerIsImmediate
? setImmediate(run)
: setTimeout(run, this.maxTimeInQueue);
}
#clearFlushTimer() {
if (this.#flushTimer === null) return;
if (this.#flushTimerIsImmediate) clearImmediate(this.#flushTimer);
else clearTimeout(this.#flushTimer);
this.#flushTimer = null;
}
// Bounded wait for a pooled connection. Without a bound, a stalled
// database turns every flush into an indefinite hang — nothing fails,
// so neither the breaker nor callers' own timeouts ever engage.
@@ -252,12 +281,18 @@ export class SQLBatcher {
}
async flush(batch) {
const timeout = this.timeouts.shift();
if (timeout && !timeout._destroyed) {
clearTimeout(timeout);
}
this.#clearFlushTimer();
if (batch.length === 0) return;
this.#inFlight.count++;
try {
await this.#flushBatch(batch);
} finally {
this.#inFlight.count--;
}
}
async #flushBatch(batch) {
let connection;
try {
connection = await this.#acquireConnection();
@@ -69,6 +69,356 @@ const happyBatch = (sql: string) => {
];
};
// Each query on this connection waits until the test releases it, so its
// flush stays in flight.
const makeHeldConnection = () => {
const held: Array<() => void> = [];
const conn = makeConnection(
(sql) =>
new Promise((resolve) => {
held.push(() => resolve(happyBatch(sql)));
}),
);
const releaseAll = () => {
for (const release of held.splice(0)) release();
};
const sent = () => conn.query.mock.calls.map(([sql]) => sql);
return { conn, releaseAll, sent };
};
const fakeFlushTimers = () =>
vi.useFakeTimers({
toFake: [
'setTimeout',
'clearTimeout',
'setImmediate',
'clearImmediate',
],
});
describe('SQLBatcher flush timing', () => {
it('flushes a lone query on the next turn while under the in-flight cap', async () => {
const conn = makeConnection(happyBatch);
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxTimeInQueue: 60_000,
maxInFlightImmediate: 2,
});
const pending = batcher.query('SELECT a', []);
expect(getConnection).not.toHaveBeenCalled();
await new Promise((resolve) => setImmediate(resolve));
expect(getConnection).toHaveBeenCalledTimes(1);
expect((await pending)[0]).toEqual([{ n: 0 }]);
});
it('holds a lone query for maxTimeInQueue without an in-flight cap', async () => {
fakeFlushTimers();
try {
const conn = makeConnection(happyBatch);
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxTimeInQueue: 30,
});
const pending = batcher.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(29);
expect(getConnection).not.toHaveBeenCalled();
await vi.advanceTimersByTimeAsync(1);
expect(getConnection).toHaveBeenCalledTimes(1);
expect((await pending)[0]).toEqual([{ n: 0 }]);
} finally {
vi.useRealTimers();
}
});
it('batches queries from the same turn and sends later ones separately', async () => {
const conn = makeConnection(happyBatch);
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxInFlightImmediate: 8,
});
const sameTurn = [
batcher.query('SELECT a', []),
batcher.query('SELECT b', []),
batcher.query('SELECT c', []),
];
await new Promise((resolve) => setImmediate(resolve));
const nextTurn = batcher.query('SELECT d', []);
const results = await Promise.all([...sameTurn, nextTurn]);
expect(results.map((r) => r[0])).toEqual([
[{ n: 0 }],
[{ n: 1 }],
[{ n: 2 }],
[{ n: 0 }],
]);
expect(getConnection).toHaveBeenCalledTimes(2);
expect(conn.query.mock.calls.map(([sql]) => sql)).toEqual([
'SELECT a;SELECT b;SELECT c; SELECT 1',
'SELECT d; SELECT 1',
]);
});
it('gives a query enqueued during an in-flight flush its own flush while under the cap', async () => {
const { conn, releaseAll, sent } = makeHeldConnection();
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxInFlightImmediate: 2,
});
const turn = () => new Promise((resolve) => setImmediate(resolve));
const a = batcher.query('SELECT a', []);
await turn();
const b = batcher.query('SELECT b', []);
await turn();
expect(sent()).toEqual(['SELECT a; SELECT 1', 'SELECT b; SELECT 1']);
releaseAll();
expect((await a)[0]).toEqual([{ n: 0 }]);
expect((await b)[0]).toEqual([{ n: 0 }]);
});
it('holds a lone query for maxTimeInQueue at the in-flight cap and coalesces what follows', async () => {
fakeFlushTimers();
try {
const { conn, releaseAll, sent } = makeHeldConnection();
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxTimeInQueue: 30,
maxInFlightImmediate: 1,
});
const a = batcher.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(0);
expect(sent()).toEqual(['SELECT a; SELECT 1']);
const b = batcher.query('SELECT b', []);
await vi.advanceTimersByTimeAsync(29);
const c = batcher.query('SELECT c', []);
expect(sent()).toHaveLength(1);
await vi.advanceTimersByTimeAsync(1);
expect(sent()).toEqual([
'SELECT a; SELECT 1',
'SELECT b;SELECT c; SELECT 1',
]);
releaseAll();
expect((await a)[0]).toEqual([{ n: 0 }]);
expect((await b)[0]).toEqual([{ n: 0 }]);
expect((await c)[0]).toEqual([{ n: 1 }]);
// Settled flushes free the cap again.
const d = batcher.query('SELECT d', []);
await vi.advanceTimersByTimeAsync(0);
expect(sent()).toHaveLength(3);
releaseAll();
expect((await d)[0]).toEqual([{ n: 0 }]);
} finally {
vi.useRealTimers();
}
});
it('shares the in-flight cap between batchers given one counter', async () => {
fakeFlushTimers();
try {
const { conn, releaseAll, sent } = makeHeldConnection();
const { pool } = makePool(conn);
const inFlight = { count: 0 };
const opts = {
readOnly: true,
maxTimeInQueue: 30,
maxInFlightImmediate: 1,
inFlight,
};
const first = new SQLBatcher(pool, opts);
const second = new SQLBatcher(pool, opts);
const own = new SQLBatcher(pool, { ...opts, inFlight: undefined });
const a = first.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(0);
const b = second.query('SELECT b', []);
const c = own.query('SELECT c', []);
await vi.advanceTimersByTimeAsync(0);
expect(sent()).toEqual([
'SELECT a; SELECT 1',
'SELECT c; SELECT 1',
]);
await vi.advanceTimersByTimeAsync(30);
expect(sent()).toHaveLength(3);
expect(sent()[2]).toBe('SELECT b; SELECT 1');
releaseAll();
await Promise.all([a, b, c]);
expect(inFlight.count).toBe(0);
} finally {
vi.useRealTimers();
}
});
it('counts a flush as in flight until its per-item fallback settles', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
fakeFlushTimers();
try {
const held: Array<() => void> = [];
const conn = makeConnection((sql) => {
if (isBatchQuery(sql)) throw makeError('ECONNRESET');
return new Promise((resolve) => {
held.push(() => resolve([[{ ok: sql }], undefined]));
});
});
const { pool } = makePool(conn);
const sent = () => conn.query.mock.calls.map(([sql]) => sql);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxTimeInQueue: 30,
maxInFlightImmediate: 1,
acquireTimeoutMs: 0,
});
const a = batcher.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(0);
// The batch failed; the per-item retry of `a` is still running.
expect(sent()).toEqual(['SELECT a; SELECT 1', 'SELECT a']);
const b = batcher.query('SELECT b', []);
await vi.advanceTimersByTimeAsync(29);
expect(sent()).toHaveLength(2);
await vi.advanceTimersByTimeAsync(1);
expect(sent()).toEqual([
'SELECT a; SELECT 1',
'SELECT a',
'SELECT b; SELECT 1',
'SELECT b',
]);
for (const release of held.splice(0)) release();
expect((await a)[0]).toEqual([{ ok: 'SELECT a' }]);
expect((await b)[0]).toEqual([{ ok: 'SELECT b' }]);
} finally {
vi.useRealTimers();
warnSpy.mockRestore();
}
});
it('flushes a full batch at once past the cap and cancels the armed wait', async () => {
fakeFlushTimers();
try {
const { conn, releaseAll, sent } = makeHeldConnection();
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxBatchSize: 2,
maxTimeInQueue: 30,
maxInFlightImmediate: 1,
acquireTimeoutMs: 0,
});
const a = batcher.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(0);
const b = batcher.query('SELECT b', []);
expect(vi.getTimerCount()).toBe(1);
const c = batcher.query('SELECT c', []);
expect(getConnection).toHaveBeenCalledTimes(2);
expect(vi.getTimerCount()).toBe(0);
// A later lone query waits its own full maxTimeInQueue.
await vi.advanceTimersByTimeAsync(10);
const d = batcher.query('SELECT d', []);
await vi.advanceTimersByTimeAsync(29);
expect(getConnection).toHaveBeenCalledTimes(2);
await vi.advanceTimersByTimeAsync(1);
expect(getConnection).toHaveBeenCalledTimes(3);
expect(sent()).toEqual([
'SELECT a; SELECT 1',
'SELECT b;SELECT c; SELECT 1',
'SELECT d; SELECT 1',
]);
releaseAll();
await Promise.all([a, b, c, d]);
} finally {
vi.useRealTimers();
}
});
it('cancels the pending next-turn flush when a batch fills up', async () => {
fakeFlushTimers();
try {
const conn = makeConnection(happyBatch);
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxBatchSize: 2,
readOnly: true,
maxInFlightImmediate: 8,
acquireTimeoutMs: 0,
});
const a = batcher.query('SELECT a', []);
expect(vi.getTimerCount()).toBe(1);
const b = batcher.query('SELECT b', []);
expect(vi.getTimerCount()).toBe(0);
const c = batcher.query('SELECT c', []);
expect(vi.getTimerCount()).toBe(1);
await vi.runAllTimersAsync();
await Promise.all([a, b, c]);
expect(vi.getTimerCount()).toBe(0);
expect(getConnection).toHaveBeenCalledTimes(2);
} finally {
vi.useRealTimers();
}
});
it('leaves no timer armed after a failed flush, a drop or a breaker rejection', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
fakeFlushTimers();
try {
const getConnection = vi.fn(async () => {
throw makeError('ECONNREFUSED');
});
const batcher = new SQLBatcher(
{ promise: () => ({ getConnection }) },
{
readOnly: true,
maxQueueSize: 1,
failureThreshold: 1,
cooldownMs: 60_000,
maxInFlightImmediate: 8,
},
);
const dropped = expect(
batcher.query('SELECT a', []),
).rejects.toMatchObject({ reason: 'queueOverflow' });
const failed = expect(
batcher.query('SELECT b', []),
).rejects.toMatchObject({ reason: 'connAcquire' });
expect(vi.getTimerCount()).toBe(1);
await vi.runAllTimersAsync();
await dropped;
await failed;
expect(vi.getTimerCount()).toBe(0);
await expect(batcher.query('SELECT c', [])).rejects.toMatchObject({
reason: 'breakerOpen',
});
expect(vi.getTimerCount()).toBe(0);
expect(batcher.queue).toHaveLength(0);
} finally {
vi.useRealTimers();
warnSpy.mockRestore();
}
});
});
describe('SQLBatcher', () => {
it('resolves each batched item with its own result', async () => {
const conn = makeConnection(happyBatch);
@@ -27,7 +27,7 @@ import { DatabaseClientFactory } from './index.js';
import { SqliteDatabaseClient } from './SqliteDatabaseClient.js';
/** Highest schema version the migration table can reach. */
const CURRENT_SCHEMA_VERSION = 89;
const CURRENT_SCHEMA_VERSION = 90;
/**
* These suites migrate real files on disk. Idle they finish in well under a
@@ -930,6 +930,7 @@ describe('SqliteDatabaseClient — query interface', () => {
affectedRows: 1,
anyRowsAffected: true,
});
expect(inserted).not.toHaveProperty('warningStatus');
const missed = await client.write(
'UPDATE `widget` SET `name` = ? WHERE `id` = ?',
@@ -128,6 +128,7 @@ const AVAILABLE_MIGRATIONS: [number, string[]][] = [
[86, ['0091_icloud-dot-significance.sql']],
[87, ['0092_share-entry-owner.sql']],
[88, ['0093_fsentries-parent-uid-index.sql']],
[89, ['0094_deleted-apps.sql']],
];
export class SqliteDatabaseClient extends AbstractDatabaseClient {
@@ -206,6 +207,9 @@ export class SqliteDatabaseClient extends AbstractDatabaseClient {
insertId: info.lastInsertRowid,
affectedRows: info.changes,
anyRowsAffected: info.changes > 0,
// No warningStatus: SQLite raises none, but a new row can reuse a
// deleted row's rowid that other tables still name, so a clean
// insert doesn't prove the row is all the caller sent.
};
}
@@ -0,0 +1,26 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Uid and owner of each deleted app. Mirrors SQLite migration 0094.
--
-- Idempotent: `CREATE TABLE IF NOT EXISTS` lets the directory replay safely.
CREATE TABLE IF NOT EXISTS `deleted_apps` (
`uid` VARCHAR(40) NOT NULL,
`owner_user_id` INT UNSIGNED DEFAULT NULL,
PRIMARY KEY (`uid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
@@ -0,0 +1,26 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Uid and owner of each deleted app. Mirrors SQLite migration 0094 / MySQL
-- mysql_mig_48.
--
-- Idempotent via IF NOT EXISTS.
CREATE TABLE IF NOT EXISTS deleted_apps (
uid VARCHAR(40) PRIMARY KEY,
owner_user_id INTEGER DEFAULT NULL
);
@@ -0,0 +1,25 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Uid and owner of each deleted app. An origin-derived uid comes back when the
-- origin is used again; on a hosted subdomain now registered to someone else,
-- AuthService moves on to a fresh uid so the deleted app's data stays put.
CREATE TABLE IF NOT EXISTS `deleted_apps` (
"uid" TEXT PRIMARY KEY NOT NULL,
"owner_user_id" INTEGER DEFAULT NULL
);
@@ -1,3 +1,9 @@
import { trace } from '@opentelemetry/api';
import {
BasicTracerProvider,
InMemorySpanExporter,
SimpleSpanProcessor,
} from '@opentelemetry/sdk-trace-base';
import {
afterAll,
afterEach,
@@ -207,6 +213,122 @@ describe('EventClient', () => {
await target.emitAndWait(`${key}.child`, {}, {});
expect(resolved).toBe(true);
});
describe('tracing', () => {
const exporter = new InMemorySpanExporter();
const provider = new BasicTracerProvider({
spanProcessors: [new SimpleSpanProcessor(exporter)],
});
const spansFor = (eventKey: string) =>
exporter
.getFinishedSpans()
.filter(
(s) =>
s.name === 'event.emitAndWait' &&
s.attributes['event.key'] === eventKey,
);
beforeAll(() => {
trace.setGlobalTracerProvider(provider);
});
afterAll(async () => {
await provider.shutdown();
trace.disable();
});
it('resolves without a span when nothing listens', async () => {
const meta: { listener_failed?: boolean } = {};
await expect(
target.emitAndWait(`${key}.child`, {}, meta),
).resolves.toBeUndefined();
expect(meta).toEqual({});
expect(spansFor(`${key}.child`)).toHaveLength(0);
});
it('spans the wait whenever a listener matches, wildcards included', async () => {
target.on(`${key}.*`, vi.fn());
await expect(
target.emitAndWait(`${key}.child`, {}, {}),
).resolves.toBeUndefined();
expect(spansFor(`${key}.child`)).toHaveLength(1);
extensionStore.events[`${key}.ext`] = [vi.fn()];
try {
await target.emitAndWait(`${key}.ext`, {}, {});
} finally {
delete extensionStore.events[`${key}.ext`];
}
expect(spansFor(`${key}.ext`)).toHaveLength(1);
});
});
});
describe('emitAndWaitConcurrently', () => {
it('runs every listener at once, wildcards included, and awaits them all', async () => {
let inFlight = 0;
let peak = 0;
const contribute =
(share: number) => async (_k: string, data: unknown) => {
inFlight++;
peak = Math.max(peak, inFlight);
await new Promise((r) => setTimeout(r, 10));
inFlight--;
(data as { total: number }).total += share;
};
target.on(`${key}.*`, contribute(1));
target.on(`${key}.child`, contribute(10));
target.on(`${key}.child`, contribute(100));
const data = { total: 0 };
await target.emitAndWaitConcurrently(`${key}.child`, data, {});
expect(peak).toBe(3);
expect(data.total).toBe(111);
});
it('flags a failed exact-key listener and still awaits the rest', async () => {
const errSpy = vi
.spyOn(console, 'error')
.mockImplementation(() => {});
let finished = false;
target.on(key, () => {
throw new Error('boom');
});
target.on(key, async () => {
await new Promise((r) => setTimeout(r, 5));
finished = true;
});
const meta: Record<string, unknown> = {};
await target.emitAndWaitConcurrently(key, {}, meta);
expect(finished).toBe(true);
expect(meta.listener_failed).toBe(true);
expect(errSpy).toHaveBeenCalledWith(
'Error in event listener for event',
key,
expect.any(Error),
);
errSpy.mockRestore();
});
it('leaves the flag alone when only a wildcard listener fails', async () => {
const errSpy = vi
.spyOn(console, 'error')
.mockImplementation(() => {});
target.on(`${key}.*`, () => {
throw new Error('observer');
});
const meta: Record<string, unknown> = {};
await target.emitAndWaitConcurrently(`${key}.child`, {}, meta);
expect(meta.listener_failed).toBeUndefined();
errSpy.mockRestore();
});
it('resolves at once with nobody listening', async () => {
await expect(
target.emitAndWaitConcurrently(`${key}.nobody`, {}, {}),
).resolves.toBeUndefined();
});
});
describe('off', () => {
+38
View File
@@ -94,6 +94,7 @@ export class EventClient extends PuterClient {
data: EventMap[T],
meta: EventMetadata,
) {
if (!this.hasListeners(key)) return;
// Spanned because callers block on listeners — this is where time
// spent in extension hooks (e.g. `ip.validate`) gets attributed.
return withSpan('event.emitAndWait', { 'event.key': key }, async () => {
@@ -118,6 +119,43 @@ export class EventClient extends PuterClient {
});
}
/**
* Like `emitAndWait`, but every matched listener runs at once, so none sees
* another's mutation of `data` before all have finished. Only for events
* whose listeners each add an independent share, written after their own
* awaits (`const v = await x; data.n += v`).
*/
async emitAndWaitConcurrently<T extends keyof EventMap>(
key: T,
data: EventMap[T],
meta: EventMetadata,
) {
const matched = this.#matchKeys(key).flatMap((matchKey) =>
this.#listenersFor(matchKey).map((listener) => ({
listener,
// A wildcard listener observes the event; it does not implement it.
isExact: matchKey === key,
})),
);
if (matched.length === 0) return;
return withSpan('event.emitAndWait', { 'event.key': key }, async () => {
await Promise.all(
matched.map(async ({ listener, isExact }) => {
try {
await listener(key, data, meta);
} catch (e) {
if (isExact) meta.listener_failed = true;
console.error(
'Error in event listener for event',
key,
e,
);
}
}),
);
});
}
/**
* Whether anything would run if `key` were emitted, wildcards included.
*
+10 -5
View File
@@ -459,7 +459,11 @@ export type EventMap = {
[key: string]: unknown;
};
};
'storage.quota.bonus': { userId: number; extra: number };
/**
* Bytes to add to a user's storage allowance. Listeners run concurrently:
* add to `extra` after your own awaits, never `extra += await x`.
*/
'storage.quota.bonus': { userId: number; userUuid?: string; extra: number };
// ---- Key-value ----
/**
@@ -919,10 +923,11 @@ export type EventKey = keyof EventMap & string;
// Generates a wildcard for every non-final dot-separated prefix of K.
export type WildcardPrefixes<K extends string> =
K extends `${infer Head}.${infer Tail}`
? | `${Head}.*`
| (Tail extends `${string}.${string}`
? `${Head}.${WildcardPrefixes<Tail>}`
: never)
?
| `${Head}.*`
| (Tail extends `${string}.${string}`
? `${Head}.${WildcardPrefixes<Tail>}`
: never)
: never;
export type ListenKey = EventKey | WildcardPrefixes<EventKey>;
+62 -10
View File
@@ -3,18 +3,19 @@
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
* Puter is free software: you can redistribute it and/or modify it under the
* terms of the GNU Affero General Public License as published by the Free
* Software Foundation, either version 3 of the License, or (at your option) any
* later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
* details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
* along with this program. If not, see
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
*/
import { GetObjectCommand } from '@aws-sdk/client-s3';
@@ -24,7 +25,7 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import type { IConfig } from '../../types';
import { S3Client } from './S3Client';
import { S3Client, withPresignHeadroom } from './S3Client';
const LOCAL_BUCKET = 'puter-local';
@@ -487,3 +488,54 @@ describe('S3Client — short reads during multipart', () => {
expect(parts).toEqual([10, 10]);
});
});
describe('withPresignHeadroom', () => {
const HOUR = 60 * 60 * 1000;
const creds = (msLeft?: number, id = 'a') => ({
accessKeyId: id,
secretAccessKey: 's',
...(msLeft === undefined
? {}
: { expiration: new Date(Date.now() + msLeft) }),
});
it('reports expiry an hour early so the SDK refreshes while an hour remains', async () => {
const chain = vi.fn(async () => creds(5 * HOUR));
const identity = await withPresignHeadroom(chain)();
expect(chain).toHaveBeenCalledTimes(1);
const left = identity.expiration!.getTime() - Date.now();
expect(left).toBeGreaterThan(4 * HOUR - 1000);
expect(left).toBeLessThanOrEqual(4 * HOUR);
});
it('force-refreshes credentials that are inside the presign window', async () => {
const chain = vi.fn(async (options?: { forceRefresh?: boolean }) =>
options?.forceRefresh
? creds(6 * HOUR, 'fresh')
: creds(30 * 60 * 1000, 'stale'),
);
const identity = await withPresignHeadroom(chain)();
expect(chain).toHaveBeenLastCalledWith(
expect.objectContaining({ forceRefresh: true }),
);
expect(identity.accessKeyId).toBe('fresh');
});
it('leaves long-lived credentials alone', async () => {
const chain = vi.fn(async () => creds(undefined));
const identity = await withPresignHeadroom(chain)();
expect(chain).toHaveBeenCalledTimes(1);
expect(identity.expiration).toBeUndefined();
});
it('never reports an expiry inside the SDK refresh margin', async () => {
const chain = vi.fn(async () => creds(20 * 60 * 1000));
const identity = await withPresignHeadroom(chain)();
const left = identity.expiration!.getTime() - Date.now();
expect(left).toBeGreaterThan(5 * 60 * 1000);
});
});
+40 -1
View File
@@ -43,6 +43,43 @@ const DEFAULT_MULTIPART_PART_SIZE_BYTES = 5 * 1024 * 1024;
const FAUXQS_SAFE_PUT_OBJECT_LIMIT_BYTES = 10 * 1024 * 1024;
const LEGACY_STORAGE_BUCKET = 'puter-local';
// A presigned URL stops working when the credentials that signed it expire.
// Presigns last up to an hour, but the SDK only refreshes temporary
// credentials five minutes before expiry.
const PRESIGN_MAX_LIFETIME_MS = 60 * 60 * 1000;
const SDK_REFRESH_MARGIN_MS = 5 * 60 * 1000;
type CredentialChain = ReturnType<typeof fromNodeProviderChain>;
/**
* Keeps temporary credentials at least an hour from expiry when they sign. The
* SDK is handed each identity with its expiration moved an hour early, so its
* own refresh comes back here while an hour remains, and a credential already
* inside that window is force-refreshed from the chain.
*/
export function withPresignHeadroom(chain: CredentialChain): CredentialChain {
return async (options) => {
let identity = await chain(options);
const remaining = (i: typeof identity) =>
i.expiration ? i.expiration.getTime() - Date.now() : Infinity;
if (
remaining(identity) <
PRESIGN_MAX_LIFETIME_MS + SDK_REFRESH_MARGIN_MS
) {
identity = await chain({ ...options, forceRefresh: true });
}
if (!identity.expiration) return identity;
// If even fresh credentials are short-lived, report an expiry just past
// the SDK's margin rather than one in the past, which would make it
// re-resolve on every request.
const reported = Math.max(
identity.expiration.getTime() - PRESIGN_MAX_LIFETIME_MS,
Date.now() + SDK_REFRESH_MARGIN_MS + 60 * 1000,
);
return { ...identity, expiration: new Date(reported) };
};
}
type S3CommandSender = Pick<AwsS3Client, 'send'>;
export class S3Client extends PuterClient {
@@ -83,7 +120,9 @@ export class S3Client extends PuterClient {
if (useCredentialChain) {
this.useProviderChain = true;
this.awsConfig = { credentials: fromNodeProviderChain() };
this.awsConfig = {
credentials: withPresignHeadroom(fromNodeProviderChain()),
};
this.partSize = 64 * 1024 * 1024;
this.maxSingleUploadSize = 128 * 1024 * 1024;
} else {
@@ -338,6 +338,42 @@ describe('AppController POST /rao', () => {
expect(rows).toHaveLength(1);
});
it('keeps every open as a row, and counts one per account per hour', async () => {
const owner = await makeUser();
const other = await makeUser();
const app = await createApp(owner.actor);
const open = async (user: { actor: Actor; userId: number }) => {
const { res } = makeRes();
await withActor(user.actor, () =>
callRoute(
'post',
'/rao',
makeReq({ body: { app_uid: app.uid }, actor: user.actor }),
res,
),
);
};
for (let i = 0; i < 5; i++) await open(owner);
await open(other);
await server.controllers.apps.drainPendingAppOpens();
// Every open is still a row: the recent-apps list orders on them.
const rows = (await server.clients.db.read(
'SELECT `user_id` FROM `app_opens` WHERE `app_uid` = ?',
[app.uid],
)) as Array<{ user_id: number }>;
expect(rows).toHaveLength(6);
// The count is accounts-per-hour, so repeats do not move it.
const stats = await server.stores.app.getAppsStats([app.uid]);
expect(stats.get(app.uid)).toMatchObject({
open_count: 2,
user_count: 2,
});
});
// `app_opens` is analytics: the response carries nothing derived from it
// and the client never awaits the post. Holding the response open for a
// primary write put that write's latency inside the app launch it was
@@ -18,7 +18,7 @@
*/
import { TOTP } from 'otpauth';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { makeActor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { createSecret as otpCreateSecret } from '../../services/auth/OTPUtil.js';
@@ -28,7 +28,6 @@ import {
setupPuterTestEnv,
type PuterTestEnv,
} from '../../testUtil.js';
import type { IConfig } from '../../types';
/**
* HTTP-level coverage for `/auth/revoke-own-access-token` — the route that
@@ -424,7 +423,7 @@ describe('2FA enrollment over HTTP', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv({ teams_enabled: true } as IConfig);
env = await setupPuterTestEnv();
}, 120_000);
afterAll(async () => {
@@ -561,56 +560,6 @@ describe('2FA enrollment over HTTP', () => {
expect((await otpState(account.userId)).enabled).toBe(true);
});
it('lets a seat its team holds for 2FA enroll through the same gate', async () => {
const owner = await makeAccount();
// The rule can only be turned on by an owner who has 2FA.
await env.server.stores.user.update(owner.userId, { otp_enabled: 1 });
await env.server.stores.user.invalidateById(owner.userId);
const team = await env.server.services.team.createTeam(owner.userId, {
name: 'Acme',
});
const seatName = `seat_${uniq()}`;
const { userId, temporaryPassword } =
await env.server.services.team.provisionAccount(
team.uid,
owner.userId,
{ username: seatName },
);
await env.server.services.team.updateTeam(team.uid, owner.userId, {
require2fa: true,
});
const { token } = await env.server.services.auth.createSessionToken(
(await env.server.stores.user.getById(userId))!,
);
// First sign-in replaces the temporary password.
const changed = await protectedPost(
'/user-protected/change-password',
token,
{ password: temporaryPassword, new_pass: 'seat-own-password' },
);
expect(changed.status).toBe(200);
const held = await fetch(new URL('/teams', env.apiOrigin), {
headers: { authorization: `Bearer ${token}` },
});
expect(held.status).toBe(403);
expect(await held.text()).toContain('two_factor_required');
const res = await setup(token, { password: 'seat-own-password' });
expect(res.status).toBe(200);
const { secret } = (await res.json()) as { secret: string };
const enabled = await configure(token, 'enable', {
code: liveTotp(seatName, secret),
});
expect(enabled.status).toBe(200);
expect((await otpState(userId)).enabled).toBe(true);
const released = await fetch(new URL('/teams', env.apiOrigin), {
headers: { authorization: `Bearer ${token}` },
});
expect(released.status).toBe(200);
});
});
/**
@@ -680,3 +629,89 @@ describe('temp-account reauth over HTTP', () => {
expect(who.status).toBe(200);
});
});
describe('session token helpers over HTTP', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 120_000);
afterAll(async () => {
await env?.shutdown();
}, 120_000);
const signupTemp = async () => {
const res = await fetch(new URL('/signup', env.origin), {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ is_temp: true }),
});
expect(res.status).toBe(200);
return (await res.json()) as { token: string; user: { uuid: string } };
};
const get = (path: string, token: string) =>
fetch(new URL(path, env.origin), {
headers: { authorization: `Bearer ${token}` },
});
const claims = (token: string) =>
env.server.services.token.verify('auth', token) as {
type: string;
user_uid: string;
session_uid: string;
};
it('mint tokens for the session the request authenticated with', async () => {
const { token, user } = await signupTemp();
const session = claims(token);
const row = await env.server.stores.user.getByUuid(user.uuid);
const getById = vi.spyOn(env.server.stores.user, 'getById');
try {
const gui = await get('/get-gui-token', token);
expect(gui.status).toBe(200);
const { token: guiToken } = (await gui.json()) as {
token: string;
};
expect(claims(guiToken)).toMatchObject({
type: 'gui',
user_uid: user.uuid,
session_uid: session.session_uid,
});
const sync = await get('/session/sync-cookie', token);
expect(sync.status).toBe(204);
const cookie = sync.headers.get('set-cookie') ?? '';
const cookieToken = /puter_auth_token=([^;]+)/.exec(cookie)?.[1];
expect(claims(cookieToken!)).toMatchObject({
type: 'session',
user_uid: user.uuid,
session_uid: session.session_uid,
});
// The user is the row token auth already read.
expect(getById.mock.calls.filter(([id]) => id === row!.id)).toEqual(
[],
);
} finally {
getById.mockRestore();
}
});
it("refuse a deleted account's session before either handler runs", async () => {
const { token, user } = await signupTemp();
const row = await env.server.stores.user.getByUuid(user.uuid);
await env.server.clients.db.write('DELETE FROM `user` WHERE `id` = ?', [
row!.id,
]);
await env.server.stores.user.markDeleted(row!);
expect((await get('/get-gui-token', token)).status).toBe(401);
const sync = await get('/session/sync-cookie', token);
expect(sync.status).toBe(401);
expect(sync.headers.get('set-cookie') ?? '').not.toContain(
'puter_auth_token=',
);
});
});
@@ -28,6 +28,7 @@
* play along. Aligns with AGENTS.md: "Prefer test server over mocking deps."
*/
import { installSeatStub } from '../../testFixtures/seatStub.js';
import bcrypt from 'bcrypt';
import jwt from 'jsonwebtoken';
import { TOTP } from 'otpauth';
@@ -2259,6 +2260,62 @@ describe('AuthController grant flows', () => {
).not.toContain(permission);
});
it('revoke-user-app `*`: deletes the worker sessions that app deployed for the user, and only those', async () => {
const app = await server.stores.app.create(
{
name: `tr-worker-${uuidv4()}`,
title: 'TestRevokeWorkerApp',
index_url: `https://tr-worker-${uuidv4()}.example.test/`,
},
{ ownerUserId: issuer.id },
);
const workerName = () => `wk-${uuidv4().slice(0, 8)}`;
const appWorker = await server.stores.session.getOrCreateWorker(
issuer.id,
{ appUid: app.uid, workerName: workerName() },
);
const otherApp = await server.stores.session.getOrCreateWorker(
issuer.id,
{ appUid: `app-${uuidv4()}`, workerName: workerName() },
);
const userScoped = await server.stores.session.getOrCreateWorker(
issuer.id,
{ appUid: null, workerName: workerName() },
);
const exists = async (uuid: string) =>
(
await server.clients.db.read(
'SELECT 1 FROM `sessions` WHERE `uuid` = ?',
[uuid],
)
).length > 0;
// A single permission leaves the app's workers running as before.
await inCtx(issuerActor, () =>
controller.handleRevokeUserApp(
makeReq(
{ app_uid: app.uid, permission: 'service:x:ii:read' },
{ actor: issuerActor },
),
makeRes(),
),
);
expect(await exists(appWorker.uuid)).toBe(true);
await inCtx(issuerActor, () =>
controller.handleRevokeUserApp(
makeReq(
{ app_uid: app.uid, permission: '*' },
{ actor: issuerActor },
),
makeRes(),
),
);
expect(await exists(appWorker.uuid)).toBe(false);
expect(await exists(otherApp.uuid)).toBe(true);
expect(await exists(userScoped.uuid)).toBe(true);
});
it('grant-user-app: 400 on non-string or oversized origin/app_uid/permission', async () => {
const cases = [
{ origin: { host: 'evil' }, permission: 'service:x:ii:read' },
@@ -3541,6 +3598,46 @@ describe('AuthController.handleGetUserAppToken + handleCheckApp', () => {
);
});
// The handler's own lookup is the row the grant and the token use.
it('reads the app row once per get-user-app-token and check-app', async () => {
const getByUid = vi.spyOn(server.stores.app, 'getByUid');
const lookupsOf = () =>
getByUid.mock.calls.filter(([uid]) => uid === app.uid).length;
try {
const tokenRes = makeRes();
await inCtx(actor, () =>
controller.handleGetUserAppToken(
makeReq({ app_uid: app.uid }, { actor }),
tokenRes,
),
);
expect(lookupsOf()).toBe(1);
const minted = tokenRes.body as { token: string };
getByUid.mockClear();
const checkRes = makeRes();
await inCtx(actor, () =>
controller.handleCheckApp(
makeReq({ app_uid: app.uid }, { actor }),
checkRes,
),
);
expect(lookupsOf()).toBe(1);
const checked = checkRes.body as { token: string };
// Both tokens name the same app session.
const sessionOf = (token: string) =>
(
server.services.token.verify('auth', token) as {
session_uid: string;
}
).session_uid;
expect(sessionOf(checked.token)).toBe(sessionOf(minted.token));
} finally {
getByUid.mockRestore();
}
});
it('after get-user-app-token, check-app reports authenticated:true and returns a token', async () => {
// Ensure the flag is granted (re-run is idempotent).
await inCtx(actor, () =>
@@ -6218,29 +6315,25 @@ describe('AuthController password recovery', () => {
it('send-pass-recovery-email: refuses a team seat, and writes no token', async () => {
// The seat's address is admin-supplied and unverified; recovery there
// would be a takeover channel. Its recovery is the admin's reset.
const { user: owner } = await makeUserAndActor();
const { user: seat } = await makeUserAndActor();
const team = await server.stores.team.create({
ownerUserId: owner.id,
name: 'Acme',
});
await server.stores.user.update(seat.id, { password: null });
await server.stores.team.addMember(team.uid, seat.id, {
orgOwned: true,
});
const res = makeRes();
await controller.handleSendPassRecoveryEmail(
makeReq({ username: seat.username }),
res,
);
expect((res.body as { message: string }).message).toMatch(
/If that account exists/i,
);
const after = await server.stores.user.getById(seat.id, {
force: true,
});
expect(after!.pass_recovery_token).toBeFalsy();
const restore = installSeatStub(server as never, [seat.id]);
try {
const res = makeRes();
await controller.handleSendPassRecoveryEmail(
makeReq({ username: seat.username }),
res,
);
expect((res.body as { message: string }).message).toMatch(
/If that account exists/i,
);
const after = await server.stores.user.getById(seat.id, {
force: true,
});
expect(after!.pass_recovery_token).toBeFalsy();
} finally {
restore();
}
});
// `system` is seeded by migration; `admin` only by the self-hosted boot.
@@ -6629,16 +6722,9 @@ describe('AuthController user-protected mutations (validation paths)', () => {
});
it('change-email: 403 for an account its team provisioned', async () => {
const { user: owner } = await makeUserAndActor();
const { user: seat, actor } = await makeUserAndActor();
const team = await server.stores.team.create({
ownerUserId: owner.id,
name: 'Acme',
});
await server.stores.user.update(seat.id, { password: null });
await server.stores.team.addMember(team.uid, seat.id, {
orgOwned: true,
});
installSeatStub(server as never, [seat.id]);
await expect(
controller.handleChangeEmail(
@@ -6654,17 +6740,9 @@ describe('AuthController user-protected mutations (validation paths)', () => {
it('change-username: 403 for an account its team provisioned', async () => {
// The console lists members by username and the audit log records them
// by it; a self-service rename would desync both.
const { user: owner } = await makeUserAndActor();
const { user: seat, actor } = await makeUserAndActor();
const team = await server.stores.team.create({
ownerUserId: owner.id,
name: 'Acme',
});
// addMember refuses to adopt an account that has one.
await server.stores.user.update(seat.id, { password: null });
await server.stores.team.addMember(team.uid, seat.id, {
orgOwned: true,
});
installSeatStub(server as never, [seat.id]);
await expect(
controller.handleChangeUsername(
@@ -8058,15 +8136,10 @@ describe('AuthController 2FA flows', () => {
it('lets a team seat enable 2FA without a confirmed email', async () => {
// A seat has no address by design (PUT-1792) while its team can
// require 2FA, so this check would bar it forever.
const owner = await makeUserAndActor();
const team = await server.services.team.createTeam(owner.user.id, {
name: 'Acme',
});
const seatName = `s_${uniq()}`;
await server.services.team.provisionAccount(team.uid, owner.user.id, {
username: seatName,
});
const seat = await server.stores.user.getByUsername(seatName);
const { user: made } = await makeUserAndActor({ email_confirmed: 0 });
await server.stores.user.update(made.id, { password: null });
const seat = await server.stores.user.getById(made.id, { force: true });
installSeatStub(server as never, [seat!.id]);
const actor = {
user: {
id: seat!.id,
@@ -8087,15 +8160,10 @@ describe('AuthController 2FA flows', () => {
});
it('refuses to disable 2FA while the team requires it', async () => {
const owner = await makeUserAndActor({ otp_enabled: 1 });
const team = await server.services.team.createTeam(owner.user.id, {
name: 'Acme',
});
const seatName = `s_${uniq()}`;
await server.services.team.provisionAccount(team.uid, owner.user.id, {
username: seatName,
});
const seat = await server.stores.user.getByUsername(seatName);
const { user: made } = await makeUserAndActor();
await server.stores.user.update(made.id, { password: null });
const seat = await server.stores.user.getById(made.id, { force: true });
installSeatStub(server as never, [seat!.id], { require2fa: true });
const actor = {
user: {
id: seat!.id,
@@ -8106,9 +8174,6 @@ describe('AuthController 2FA flows', () => {
} as Actor;
const { secret } = await setup2fa(actor);
await enable2fa(actor, { code: liveTotp(seat!.username, secret) });
await server.services.team.updateTeam(team.uid, owner.user.id, {
require2fa: true,
});
await expect(
controller.handleDisable2fa(makeReq({}, { actor }), makeRes()),
@@ -8252,7 +8317,7 @@ describe('AuthController 2FA flows', () => {
otp_recovery_codes: 'a,b,c',
});
const res = makeRes();
await controller.handleDisable2fa(makeReq({}, { actor }), res);
await controller.handleDisable2fa(makeReq({ code: liveTotp(user.username, 'TESTSECRETBASE32') }, { actor }), res);
expect(res.body).toEqual({ success: true });
const after = await server.stores.user.getById(user.id, {
force: true,
@@ -8399,16 +8464,9 @@ describe('AuthController.handleDeleteOwnUser', () => {
it('refuses, and keeps the row, for an account its team provisioned', async () => {
// The team is billed for the seat and closing it is theirs to do, from
// the console that keeps the audit trail.
const { user: owner } = await makeUserAndActor();
const { user: seat, actor } = await makeUserAndActor();
const team = await server.stores.team.create({
ownerUserId: owner.id,
name: 'Acme',
});
await server.stores.user.update(seat.id, { password: null });
await server.stores.team.addMember(team.uid, seat.id, {
orgOwned: true,
});
installSeatStub(server as never, [seat.id]);
await expect(
controller.handleDeleteOwnUser(makeReq({}, { actor }), makeRes()),
@@ -9347,56 +9405,6 @@ describe('AuthController.handleGetDevProfile additional branches', () => {
});
});
describe('AuthController.handleGetGuiToken / handleSessionSyncCookie additional branches', () => {
it('get-gui-token: 404 when actor has a session but the user row is gone', async () => {
const { user, actor } = await makeUserAndActor();
const sessionRes = await server.services.auth.createSessionToken(
user,
{},
);
const sessionUid = (sessionRes.session as { uuid: string }).uuid;
const sessionedActor = {
...actor,
session: { uid: sessionUid },
} as Actor;
// Pull the user row out from under the session.
await server.clients.db.write('DELETE FROM `user` WHERE `id` = ?', [
user.id,
]);
await server.stores.user.invalidateById(user.id);
await expect(
controller.handleGetGuiToken(
makeReq({}, { actor: sessionedActor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 404 });
});
it('session/sync-cookie: 404 when actor has a session but the user row is gone', async () => {
const { user, actor } = await makeUserAndActor();
const sessionRes = await server.services.auth.createSessionToken(
user,
{},
);
const sessionUid = (sessionRes.session as { uuid: string }).uuid;
const sessionedActor = {
...actor,
session: { uid: sessionUid },
} as Actor;
await server.clients.db.write('DELETE FROM `user` WHERE `id` = ?', [
user.id,
]);
await server.stores.user.invalidateById(user.id);
const res = makeRes();
await controller.handleSessionSyncCookie(
makeReq({}, { actor: sessionedActor, tokenSource: 'header' }),
res,
);
expect(res.statusCode).toBe(404);
});
});
describe('AuthController.handleSendConfirmEmail additional branches', () => {
it('throws 404 when the actor user row no longer exists', async () => {
const { user, actor } = await makeUserAndActor();
@@ -10288,17 +10296,9 @@ describe('AuthController — app-data grants', () => {
describe('AuthController — a seat address nobody confirmed', () => {
/** A seat whose address its team typed in; unconfirmed. */
const makeSeat = async () => {
const { user: owner } = await makeUserAndActor();
const { user: seat } = await makeUserAndActor();
const team = await server.stores.team.create({
ownerUserId: owner.id,
name: 'Acme',
});
// A seat is created password-less and given a temporary one after.
await server.stores.user.update(seat.id, { password: null });
await server.stores.team.addMember(team.uid, seat.id, {
orgOwned: true,
});
installSeatStub(server as never, [seat.id]);
await server.stores.user.update(seat.id, {
password: await bcrypt.hash('temporary-password', 8),
});
+119 -59
View File
@@ -111,6 +111,7 @@ import { normalizeAbsolutePath } from '../../services/fs/resolveNode.js';
import type { FSEntry } from '../../stores/fs/FSEntry.js';
import { PuterController } from '../types.js';
import { startWebSession } from './webSession.js';
import { completeOidcLoginReturn } from './oidcLoginReturn.js';
const FINGERPRINT_MAX_LENGTH = 128;
// One consent prompt covers a handful of scopes at most. The cap keeps a
@@ -595,6 +596,7 @@ export class AuthController extends PuterController {
user_uid: string;
purpose: string;
auth_id?: string;
oidcContinuation?: unknown;
}>('otp', token);
} catch {
throw new HttpError(400, 'Invalid token.', {
@@ -628,7 +630,7 @@ export class AuthController extends PuterController {
await this.#enforceAuthIdMatch(req, user, decoded.auth_id ?? null);
await this.#completeLogin(req, res, user);
await this.#completeLogin(req, res, user, decoded.oidcContinuation);
}
// -- Login: recovery code ----------------------------------------
@@ -664,6 +666,7 @@ export class AuthController extends PuterController {
user_uid: string;
purpose: string;
auth_id?: string;
oidcContinuation?: unknown;
}>('otp', token);
} catch {
throw new HttpError(400, 'Invalid token.', {
@@ -688,6 +691,11 @@ export class AuthController extends PuterController {
}
await this.#assertSecondFactorAttemptsRemain(user.uuid);
if (typeof code !== 'string') {
throw new HttpError(400, 'code must be a string.', {
legacyCode: 'bad_request',
});
}
const hashed = hashRecoveryCode(code);
const codes = ((user.otp_recovery_codes as string) || '')
.split(',')
@@ -700,15 +708,15 @@ export class AuthController extends PuterController {
}
await this.#consumeOtpLoginToken(token);
// Consume the recovery code
codes.splice(idx, 1);
await this.stores.user.update(user.id, {
otp_recovery_codes: codes.join(','),
});
if (!(await this.stores.user.consumeOtpRecoveryCode(user.id, hashed))) {
await this.#recordSecondFactorFailure(user.uuid);
res.json({ proceed: false });
return;
}
await this.#enforceAuthIdMatch(req, user, decoded.auth_id ?? null);
await this.#completeLogin(req, res, user);
await this.#completeLogin(req, res, user, decoded.oidcContinuation);
}
// -- Signup ------------------------------------------------------
@@ -1932,7 +1940,7 @@ export class AuthController extends PuterController {
// A seat's address is admin-supplied and never verified, so whoever
// holds that inbox could take the seat over. Its recovery channel is
// the team admin's password reset, not this one.
if (await this.stores.team.getOrgSeat(user.id)) {
if (await this.stores.team?.getOrgSeat(user.id)) {
res.json({ message: genericMessage });
return;
}
@@ -2133,10 +2141,16 @@ export class AuthController extends PuterController {
const password_hash = await bcrypt.hash(password, 8);
let result;
try {
result = await this.clients.db.write(
'UPDATE `user` SET `password` = ?, `pass_recovery_token` = NULL, `change_email_confirm_token` = NULL, ' +
'`requires_password_change` = 0, `temp_password_expires_at` = NULL WHERE `id` = ? AND `pass_recovery_token` = ?',
[password_hash, user.id, decoded.token],
result = await this.stores.user.update(
user.id,
{
password: password_hash,
pass_recovery_token: null,
change_email_confirm_token: null,
requires_password_change: false,
temp_password_expires_at: null,
},
{ expectedRecoveryToken: decoded.token },
);
} catch (e) {
if (!isOwnedEmailConflict(e)) throw e;
@@ -2152,22 +2166,16 @@ export class AuthController extends PuterController {
{ legacyCode: 'email_already_in_use' as never },
);
}
const affected =
(result as { affectedRows?: number; changes?: number })
?.affectedRows ??
(result as { affectedRows?: number; changes?: number })?.changes ??
0;
if (affected === 0) {
if (!result) {
throw new HttpError(400, 'Token has already been used.', {
legacyCode: 'bad_request',
});
}
await this.stores.user.invalidateById(user.id);
// Best effort: the password is already committed, and an audit write
// must not skip the eviction below.
await this.services.team
.recordPasswordSelfChange(user.id as number)
.catch((err: unknown) => {
?.recordPasswordSelfChange(user.id as number)
?.catch((err: unknown) => {
console.warn('[team] password self-change audit failed:', err);
});
@@ -2222,8 +2230,8 @@ export class AuthController extends PuterController {
temp_password_expires_at: null,
});
await this.services.team
.recordPasswordSelfChange(user.id)
.catch((err: unknown) => {
?.recordPasswordSelfChange(user.id)
?.catch((err: unknown) => {
console.warn('[team] password self-change audit failed:', err);
});
@@ -2257,7 +2265,7 @@ export class AuthController extends PuterController {
// A provisioned account's name belongs to the team that made it: the
// console lists its members by username and the audit log records them
// by username, so a self-service rename would desync both.
if (await this.stores.team.getOrgSeat(req.actor!.user.id!)) {
if (await this.stores.team?.getOrgSeat(req.actor!.user.id!)) {
throw new HttpError(
403,
'Your team set this username. Ask a team admin to change it.',
@@ -2351,6 +2359,34 @@ export class AuthController extends PuterController {
res.json({ username: new_username });
}
#requireAccountTwoFactor(user: UserRow, code: unknown): void {
if (user.otp_enabled) {
if (typeof code !== 'string' || !/^\d{6}$/.test(code)) {
throw new HttpError(
400,
'Enter the 6-digit code from your authenticator app.',
{
legacyCode: 'bad_request',
fields: { factor: 'otp' },
},
);
}
let verified = false;
try {
verified =
typeof user.otp_secret === 'string' &&
verifyOtp(user.username, user.otp_secret, code);
} catch {
// Invalid or unavailable secrets cannot authorize a change.
}
if (!verified)
throw new HttpError(401, 'Incorrect code.', {
legacyCode: 'code_mismatch',
fields: { factor: 'otp' },
});
}
}
@Post('/user-protected/change-email', {
userProtected: true,
rateLimit: {
@@ -2363,7 +2399,7 @@ export class AuthController extends PuterController {
async handleChangeEmail(req: Request, res: Response): Promise<void> {
// The address is where admin-issued credentials and team notices go;
// same reasoning as the username and deletion guards above.
if (await this.stores.team.getOrgSeat(req.actor!.user.id!)) {
if (await this.stores.team?.getOrgSeat(req.actor!.user.id!)) {
throw new HttpError(
403,
'Your team set this address. Ask a team admin to change it.',
@@ -2371,6 +2407,17 @@ export class AuthController extends PuterController {
);
}
const user =
req.userProtected?.user ??
(await this.stores.user.getById(req.actor!.user.id!, {
force: true,
}));
if (!user)
throw new HttpError(404, 'User not found', {
legacyCode: 'not_found',
});
this.#requireAccountTwoFactor(user, req.body?.code);
const { new_email } = req.body ?? {};
if (!new_email || typeof new_email !== 'string') {
throw new HttpError(400, '`new_email` is required', {
@@ -2412,6 +2459,7 @@ export class AuthController extends PuterController {
token: confirm_token,
user_id: req.actor!.user.id,
purpose: 'change-email',
twoFactorVerified: Boolean(user.otp_enabled),
},
{ expiresIn: '1h' },
);
@@ -2429,10 +2477,7 @@ export class AuthController extends PuterController {
console.warn('[change-email] new-address email failed:', e);
}
// Notify the old address too
const user = await this.stores.user.getById(req.actor!.user.id!, {
force: true,
});
if (user?.email) {
if (user.email) {
try {
await (
this.clients.email as unknown as {
@@ -2489,19 +2534,21 @@ export class AuthController extends PuterController {
});
}
const rows = (await this.clients.db.read(
'SELECT * FROM `user` WHERE `change_email_confirm_token` = ? ORDER BY `id` ASC LIMIT 1',
[decoded.token],
)) as Array<Record<string, unknown>>;
const user = rows[0] as
| {
id: number;
email_confirmed?: number | boolean;
password?: string | null;
unconfirmed_change_email?: string;
}
| undefined;
if (!user || !user.unconfirmed_change_email) {
const user =
typeof decoded.user_id === 'number' &&
Number.isInteger(decoded.user_id) &&
decoded.user_id > 0
? await this.stores.user.getById(decoded.user_id, {
force: true,
})
: null;
if (
!user ||
user.change_email_confirm_token !== decoded.token ||
typeof user.unconfirmed_change_email !== 'string' ||
!user.unconfirmed_change_email ||
(user.otp_enabled && decoded.twoFactorVerified !== true)
) {
throw new HttpError(400, 'Invalid or expired token.', {
legacyCode: 'token_expired' as never,
});
@@ -3265,6 +3312,12 @@ export class AuthController extends PuterController {
app_uid,
meta ?? undefined,
);
// Withdrawing everything also retires the workers the app deployed
// under this user; their tokens would otherwise act as the app.
await this.services.auth.deleteWorkerSessionsForApp(
req.actor!.user!.id as number,
app_uid,
);
} else {
for (const entry of list) {
await this.services.permission.revokeUserAppPermission(
@@ -3744,6 +3797,7 @@ export class AuthController extends PuterController {
APP_AUTHENTICATED_FLAG,
{},
{},
app,
);
const tokenPromise: Promise<{ token: string; expiresAt?: number }> =
@@ -3753,6 +3807,7 @@ export class AuthController extends PuterController {
token: await this.services.auth.getUserAppToken(
req.actor!,
app_uid,
app,
),
}))();
@@ -3855,6 +3910,7 @@ export class AuthController extends PuterController {
result.token = await this.services.auth.getUserAppToken(
req.actor!,
app!.uid,
app!,
);
}
res.json(result);
@@ -4074,7 +4130,7 @@ export class AuthController extends PuterController {
if (action === 'enable') {
// A seat has no email by design, so this would bar it forever.
const seat = await this.stores.team.getOrgSeat(user.id);
const seat = await this.stores.team?.getOrgSeat(user.id);
if (!user.email_confirmed && !seat) {
throw new HttpError(
403,
@@ -4148,7 +4204,7 @@ export class AuthController extends PuterController {
legacyCode: 'not_found',
});
const seat = await this.stores.team.getOrgSeat(user.id);
const seat = await this.stores.team?.getOrgSeat(user.id);
if (seat && Number(seat.require_2fa) === 1) {
throw new HttpError(
409,
@@ -4157,6 +4213,8 @@ export class AuthController extends PuterController {
);
}
this.#requireAccountTwoFactor(user, req.body?.code);
await this.stores.user.update(user.id, {
otp_enabled: false,
otp_recovery_codes: null,
@@ -4234,13 +4292,10 @@ export class AuthController extends PuterController {
throw new HttpError(400, 'No session bound to this actor', {
legacyCode: 'session_required' as never,
});
const user = await this.stores.user.getById(req.actor.user.id!);
if (!user)
throw new HttpError(404, 'User not found', {
legacyCode: 'not_found',
});
// Token auth built this actor from the user's row in this request, and
// a deleted user fails it.
const guiToken = this.services.auth.createGuiToken(
user,
req.actor.user as UserRow,
req.actor.session.uid,
);
res.json({ token: guiToken });
@@ -4270,13 +4325,9 @@ export class AuthController extends PuterController {
res.status(400).end();
return;
}
const user = await this.stores.user.getById(req.actor.user.id!);
if (!user) {
res.status(404).end();
return;
}
// As in `/get-gui-token`: the actor is the user's row as auth read it.
const sessionToken = this.services.auth.createSessionTokenForSession(
user,
req.actor.user as UserRow,
req.actor.session.uid,
);
res.cookie(this.config.cookie_name ?? 'puter_token', sessionToken, {
@@ -4406,7 +4457,7 @@ export class AuthController extends PuterController {
const userId = req.actor!.user.id!;
// The team owns the account and is billed for it; only they may close
// it, through the console that keeps the audit trail.
if (await this.stores.team.getOrgSeat(userId)) {
if (await this.stores.team?.getOrgSeat(userId)) {
throw new HttpError(
403,
'Your team owns this account. Ask a team admin to remove it.',
@@ -4416,7 +4467,7 @@ export class AuthController extends PuterController {
// The other direction: deleting an owner sets `owner_user_id` NULL and
// cascades their membership, leaving a team no one can administer —
// not even an admin, since every team route resolves by membership.
if (await this.stores.team.countOwned(userId)) {
if (await this.stores.team?.countOwned(userId)) {
throw new HttpError(
403,
'Delete the teams you own before closing this account.',
@@ -4425,7 +4476,7 @@ export class AuthController extends PuterController {
}
// A deleted team's seats are suspended, not gone, and only its owner
// can retire them.
if (await this.stores.team.countOwnedTeamsWithAccounts(userId)) {
if (await this.stores.team?.countOwnedTeamsWithAccounts(userId)) {
throw new HttpError(
403,
'Close the accounts your teams created before closing this one.',
@@ -4578,7 +4629,15 @@ export class AuthController extends PuterController {
req: Request,
res: Response,
user: UserRow,
oidcContinuation?: unknown,
): Promise<void> {
const redirectUrl =
oidcContinuation === undefined
? undefined
: completeOidcLoginReturn(res, user, oidcContinuation, {
config: this.config,
oidc: this.services.oidc,
});
const gui_token = await startWebSession(req, res, user, {
config: this.config,
auth: this.services.auth,
@@ -4604,6 +4663,7 @@ export class AuthController extends PuterController {
next_step: 'complete',
token: gui_token,
user: details,
...(redirectUrl ? { redirectUrl } : {}),
});
}
}
@@ -0,0 +1,371 @@
import { TOTP } from 'otpauth';
import { v4 as uuidv4 } from 'uuid';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import {
createTestUser,
setupPuterTestEnv,
type PuterTestEnv,
} from '../../testUtil.js';
import { installSeatStub } from '../../testFixtures/seatStub.js';
describe('account change second-factor verification', () => {
let env: PuterTestEnv;
const password = 'email-change-password';
const links = new Map<string, string>();
beforeAll(async () => {
env = await setupPuterTestEnv();
vi.spyOn(env.server.clients.email, 'send').mockImplementation(
async (to, template, values) => {
if (template === 'email_verification_link')
links.set(to, String(values?.link));
return null;
},
);
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const account = async (enabled = true, oidcOnly = false) => {
const username = `email2fa_${uuidv4().slice(0, 8)}`;
const credentials = await createTestUser(env.server, {
username,
password,
});
const user = (await env.server.stores.user.getByUsername(username))!;
const otp = new TOTP({
issuer: 'puter.com',
label: username,
algorithm: 'SHA1',
digits: 6,
});
await env.server.stores.user.update(user.id, {
email: `${username}@example.com`,
email_confirmed: true,
otp_enabled: enabled,
otp_secret: otp.secret.base32,
...(oidcOnly ? { password: null } : {}),
});
return { ...credentials, id: user.id, uuid: user.uuid, otp };
};
const change = (
token: string,
body: Record<string, unknown>,
revalidation?: string,
path = '/user-protected/change-email',
) =>
fetch(new URL(path, env.origin), {
method: 'POST',
headers: {
'content-type': 'application/json',
cookie: `puter_auth_token=${token}${revalidation ? `; puter_revalidation=${revalidation}` : ''}`,
},
body: JSON.stringify(body),
});
it.each([undefined, 'invalid-code', 123456, {}])(
'rejects an enabled account with code %j before staging a change',
async (code) => {
const user = await account();
const response = await change(user.token, {
password,
new_email: '[email protected]',
code,
});
expect(response.status).toBeGreaterThanOrEqual(400);
expect(response.status).toBeLessThan(500);
expect(await response.json()).toMatchObject({ factor: 'otp' });
const after = await env.server.stores.user.getById(user.id, {
force: true,
});
expect(after!.change_email_confirm_token).toBeFalsy();
expect(after!.unconfirmed_change_email).toBeFalsy();
},
);
it('rejects an incorrect six-digit code without staging the address', async () => {
const user = await account();
const code = ['000000', '000001', '000002', '000003'].find(
(token) => user.otp.validate({ token, window: 1 }) === null,
);
const response = await change(user.token, {
password,
new_email: '[email protected]',
code,
});
expect(response.status).toBe(401);
expect(await response.json()).toMatchObject({
code: 'code_mismatch',
factor: 'otp',
});
expect(
(await env.server.stores.user.getById(user.id, { force: true }))!
.change_email_confirm_token,
).toBeFalsy();
});
it('refuses a change when the enabled account secret is unavailable', async () => {
const user = await account();
await env.server.stores.user.update(user.id, {
otp_secret: 'invalid!',
});
expect(
(
await change(user.token, {
password,
new_email: '[email protected]',
code: '000000',
})
).status,
).toBe(401);
});
it('requires a code even after OIDC revalidation', async () => {
const user = await account(true, true);
const proof = env.server.services.token.sign(
'oidc-state',
{
user_uuid: user.uuid,
purpose: 'revalidate',
},
{ expiresIn: '5m' },
);
expect(
(
await change(
user.token,
{ new_email: '[email protected]' },
proof,
)
).status,
).toBe(400);
expect(
(
await change(
user.token,
{
new_email: '[email protected]',
code: user.otp.generate(),
},
proof,
)
).status,
).toBe(200);
});
it.each([true, false])(
'completes the change with the required factors (2FA enabled: %s)',
async (enabled) => {
const user = await account(enabled);
const newEmail = `${uuidv4()}@example.com`;
expect(
(
await change(user.token, {
password,
new_email: newEmail,
...(enabled ? { code: user.otp.generate() } : {}),
})
).status,
).toBe(200);
const link = links.get(newEmail)!;
expect(link).toBeTruthy();
expect((await fetch(link, { redirect: 'manual' })).status).toBe(
200,
);
expect(
(await env.server.stores.user.getById(user.id, {
force: true,
}))!.email,
).toBe(newEmail);
},
);
it('requires a new request if 2FA is enabled after the link was issued', async () => {
const user = await account(false);
const newEmail = `${uuidv4()}@example.com`;
expect(
(await change(user.token, { password, new_email: newEmail }))
.status,
).toBe(200);
await env.server.stores.user.update(user.id, { otp_enabled: true });
expect(
(await fetch(links.get(newEmail)!, { redirect: 'manual' })).status,
).toBe(400);
expect(
(await env.server.stores.user.getById(user.id, { force: true }))!
.email,
).not.toBe(newEmail);
});
it('refuses an existing confirmation link that never verified 2FA', async () => {
const user = await account();
const token = uuidv4();
await env.server.stores.user.update(user.id, {
change_email_confirm_token: token,
unconfirmed_change_email: '[email protected]',
});
const link = env.server.services.token.sign(
'otp',
{
token,
user_id: user.id,
purpose: 'change-email',
},
{ expiresIn: '1h' },
);
const response = await fetch(
new URL(`/change_email/confirm?token=${link}`, env.origin),
{ redirect: 'manual' },
);
expect(response.status).toBe(400);
expect(
(await env.server.stores.user.getById(user.id, { force: true }))!
.email,
).toBe(`${user.username}@example.com`);
});
describe('disabling 2FA', () => {
it.each([undefined, 'invalid-code', 123456, {}])(
'refuses code %j and retains the second-factor configuration',
async (code) => {
const user = await account();
await env.server.stores.user.update(user.id, {
otp_recovery_codes: 'one,two,three',
});
const response = await change(
user.token,
{ password, code },
undefined,
'/user-protected/disable-2fa',
);
expect(response.status).toBe(400);
expect(await response.json()).toMatchObject({ factor: 'otp' });
const after = await env.server.stores.user.getById(user.id, {
force: true,
});
expect(after!.otp_enabled).toBeTruthy();
expect(after!.otp_secret).toBe(user.otp.secret.base32);
expect(after!.otp_recovery_codes).toBe('one,two,three');
},
);
it('rejects an incorrect six-digit code', async () => {
const user = await account();
const code = ['000000', '000001', '000002', '000003'].find(
(token) => user.otp.validate({ token, window: 1 }) === null,
);
const response = await change(
user.token,
{ password, code },
undefined,
'/user-protected/disable-2fa',
);
expect(response.status).toBe(401);
expect(await response.json()).toMatchObject({
code: 'code_mismatch',
factor: 'otp',
});
expect(
(await env.server.stores.user.getById(user.id, {
force: true,
}))!.otp_enabled,
).toBeTruthy();
});
it.each([false, true])(
'accepts a current code after proving identity (OIDC-only: %s)',
async (oidcOnly) => {
const user = await account(true, oidcOnly);
const proof = oidcOnly
? env.server.services.token.sign(
'oidc-state',
{
user_uuid: user.uuid,
purpose: 'revalidate',
},
{ expiresIn: '5m' },
)
: undefined;
const missing = await change(
user.token,
oidcOnly ? {} : { password },
proof,
'/user-protected/disable-2fa',
);
expect(missing.status).toBe(400);
const response = await change(
user.token,
{
...(oidcOnly ? {} : { password }),
code: user.otp.generate(),
},
proof,
'/user-protected/disable-2fa',
);
expect(response.status).toBe(200);
const after = await env.server.stores.user.getById(user.id, {
force: true,
});
expect(after!.otp_enabled).toBeFalsy();
expect(after!.otp_secret).toBeNull();
expect(after!.otp_recovery_codes).toBeNull();
},
);
it('does not accept a code without password or OIDC revalidation', async () => {
const user = await account();
const response = await change(
user.token,
{ code: user.otp.generate() },
undefined,
'/user-protected/disable-2fa',
);
expect(response.status).toBe(403);
expect(await response.json()).toMatchObject({
code: 'password_required',
});
expect(
(await env.server.stores.user.getById(user.id, {
force: true,
}))!.otp_enabled,
).toBeTruthy();
});
it('does not let a valid code override a team requirement for 2FA', async () => {
const member = await account(true, true);
installSeatStub(env.server as never, [member.id], {
require2fa: true,
});
const proof = env.server.services.token.sign('oidc-state', {
user_uuid: member.uuid, purpose: 'revalidate',
}, { expiresIn: '5m' });
const response = await change(
member.token,
{ code: member.otp.generate() },
proof,
'/user-protected/disable-2fa',
);
expect(response.status).toBe(409);
expect(
(await env.server.stores.user.getById(member.id, {
force: true,
}))!.otp_enabled,
).toBeTruthy();
});
it('remains idempotent for an account with 2FA already disabled', async () => {
const user = await account(false);
expect(
(
await change(
user.token,
{ password },
undefined,
'/user-protected/disable-2fa',
)
).status,
).toBe(200);
});
});
});
@@ -0,0 +1,96 @@
import crypto from 'node:crypto';
import type { Response } from 'express';
import { HttpError } from '../../core/http/HttpError.js';
import type { OIDCService } from '../../services/auth/OIDCService.js';
import type { UserRow } from '../../stores/user/UserStore.js';
import type { IConfig } from '../../types.js';
import { sessionCookieFlags } from '../../util/cookieFlags.js';
export const OIDC_POPUP_RETURN_COOKIE_NAME = 'puter_oidc_popup_return';
export const OIDC_POPUP_RETURN_EXPIRY_SEC = 300;
export interface OidcLoginContinuation {
userUuid: string;
redirectUri: string;
embeddedInPopup: boolean;
openerOrigin: string | null;
messageId: string | null;
switchedToLogin: boolean;
}
interface OidcReturnDeps {
config: IConfig;
oidc: OIDCService;
}
/** Bind an attested popup return to the browser that completed sign-in. */
export function mintOidcPopupReturn(
res: Response,
payload: Record<string, unknown>,
deps: OidcReturnDeps,
): string {
const nonce = crypto.randomBytes(32).toString('base64url');
res.cookie(OIDC_POPUP_RETURN_COOKIE_NAME, nonce, {
...sessionCookieFlags(deps.config, { crossSite: false }),
httpOnly: true,
maxAge: OIDC_POPUP_RETURN_EXPIRY_SEC * 1000,
path: '/',
});
return deps.oidc.signPopupReturn({ action: 'sign-in', ...payload, nonce });
}
/** Call only after both the provider and the account's second factor succeed. */
export function completeOidcLoginReturn(
res: Response,
user: Pick<UserRow, 'uuid'>,
continuation: unknown,
deps: OidcReturnDeps,
): string {
if (!continuation || typeof continuation !== 'object') {
throw new HttpError(400, 'Invalid login continuation.', {
legacyCode: 'bad_request',
});
}
const data = continuation as OidcLoginContinuation;
if (
data.userUuid !== user.uuid ||
typeof data.redirectUri !== 'string' ||
typeof data.embeddedInPopup !== 'boolean' ||
typeof data.switchedToLogin !== 'boolean' ||
(data.openerOrigin !== null && typeof data.openerOrigin !== 'string') ||
(data.messageId !== null && typeof data.messageId !== 'string')
) {
throw new HttpError(400, 'Invalid login continuation.', {
legacyCode: 'bad_request',
});
}
const origin = (deps.config.origin ?? '').replace(/\/$/, '');
let target = data.redirectUri || origin || '/';
try {
if (origin && new URL(target).origin !== new URL(origin).origin)
target = origin;
} catch {
target = origin || '/';
}
const append = (key: string, value: string) => {
target += `${target.includes('?') ? '&' : '?'}${encodeURIComponent(key)}=${encodeURIComponent(value)}`;
};
if (data.embeddedInPopup) {
append('oidc_login', 'true');
append(
'opener_state',
mintOidcPopupReturn(
res,
{
opener_origin: data.openerOrigin,
msg_id: data.messageId,
oidc_login: true,
user_uuid: user.uuid,
},
deps,
),
);
}
if (data.switchedToLogin) append('oidc_switched', 'login');
return target;
}
@@ -0,0 +1,139 @@
import { v4 as uuidv4 } from 'uuid';
import {
afterAll,
afterEach,
beforeAll,
describe,
expect,
it,
vi,
} from 'vitest';
import { hashRecoveryCode } from '../../services/auth/OTPUtil.js';
import {
createTestUser,
setupPuterTestEnv,
type PuterTestEnv,
} from '../../testUtil.js';
describe('recovery code consumption', () => {
let env: PuterTestEnv;
const password = 'recovery-code-password';
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
afterEach(() => {
vi.restoreAllMocks();
});
// Hold the same cached snapshot until every concurrent request has read it.
const overlapReads = (userUuid: string, count: number) => {
const redis = env.server.clients.redis;
const get = redis.get.bind(redis);
const ready = Promise.withResolvers<void>();
let reads = 0;
vi.spyOn(redis, 'get').mockImplementation(async (key: string) => {
const value = await get(key);
if (key.includes(userUuid) && reads < count) {
if (++reads === count) ready.resolve();
await ready.promise;
}
return value;
});
};
const post = async (path: string, body: Record<string, unknown>) => {
const response = await fetch(new URL(path, env.origin), {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(body),
});
return { response, body: await response.json() };
};
const account = async (codes: string[]) => {
const username = `recovery_${uuidv4().slice(0, 8)}`;
await createTestUser(env.server, { username, password });
const user = (await env.server.stores.user.getByUsername(username))!;
await env.server.stores.user.update(user.id, {
otp_enabled: true,
otp_secret: 'TESTSECRETBASE32',
otp_recovery_codes: codes.map(hashRecoveryCode).join(','),
});
const challenge = async () => {
const { response, body } = await post('/login', {
username,
password,
});
expect(response.status).toBe(202);
expect(body.next_step).toBe('otp');
return body.otp_jwt_token as string;
};
return { user, challenge };
};
it('allows only one concurrent redemption of the same code', async () => {
const { user, challenge } = await account(['RECOVER1', 'RECOVER2']);
const tokens = await Promise.all([challenge(), challenge()]);
overlapReads(user.uuid, tokens.length);
const results = await Promise.all(
tokens.map((token) =>
post('/login/recovery-code', { token, code: 'RECOVER1' }),
),
);
expect(
results.filter((result) => result.body.next_step === 'complete'),
).toHaveLength(1);
expect(
results.filter((result) => result.body.proceed === false),
).toHaveLength(1);
const after = await env.server.stores.user.getById(user.id, {
force: true,
});
expect(after!.otp_recovery_codes).toBe(hashRecoveryCode('RECOVER2'));
});
it('consumes different concurrent codes without restoring either one', async () => {
const codes = ['RECOVER1', 'RECOVER2', 'RECOVER3'];
const { user, challenge } = await account(codes);
const tokens = await Promise.all(codes.map(() => challenge()));
overlapReads(user.uuid, tokens.length);
const results = await Promise.all(
codes.map((code, index) =>
post('/login/recovery-code', { token: tokens[index], code }),
),
);
expect(
results.every((result) => result.body.next_step === 'complete'),
).toBe(true);
const after = await env.server.stores.user.getById(user.id, {
force: true,
});
expect(after!.otp_recovery_codes).toBe('');
const replay = await post('/login/recovery-code', {
token: tokens[0],
code: codes[0],
});
expect(replay.body).toMatchObject({ proceed: false });
});
it.each([{ code: 123456 }, { code: {} }, { code: [] }])(
'rejects a non-string recovery code $code without consuming a code',
async ({ code }) => {
const { user, challenge } = await account(['RECOVER1']);
const result = await post('/login/recovery-code', {
token: await challenge(),
code,
});
expect(result.response.status).toBe(400);
expect(
(await env.server.stores.user.getById(user.id, {
force: true,
}))!.otp_recovery_codes,
).toBe(hashRecoveryCode('RECOVER1'));
},
);
});
@@ -823,9 +823,10 @@ describe('DriverController.#handleCall tracing', () => {
'driver.method': 'set',
'driver.name': 'puter-kvstore',
});
// The call path exercises the permission gate and the KV store's
// dynamo writes — both should have produced spans of their own.
expect(names).toContain('permission.scan');
// The KV store's dynamo writes produce spans of their own. The
// permission gate doesn't: every user holds `service` by default, which
// `check` answers without a scan.
expect(names).not.toContain('permission.scan');
expect(names.some((n) => n.startsWith('ddb.'))).toBe(true);
});
});
@@ -17,7 +17,9 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { GetObjectCommand } from '@aws-sdk/client-s3';
import { Readable } from 'node:stream';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
/**
@@ -549,3 +551,111 @@ describe('POST /fs/startWrite storage quota over HTTP', () => {
expect(responseBody.code).toBe('bad_request');
});
});
describe('file reads over HTTP when storage fails mid-download', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const writeFile = async (name: string, content: string) => {
const { username } = env.users.user;
const user = await env.server.stores.user.getByUsername(username);
const body = Buffer.from(content);
const { fsEntry } = await env.server.services.fs.write(user!.id, {
fileMetadata: {
path: `/${username}/Documents/${name}`,
size: body.byteLength,
contentType: 'text/plain',
},
fileContent: body,
});
return fsEntry;
};
// Storage hands over one chunk and then fails, the way a dropped
// connection or a checksum mismatch surfaces on a read in flight.
const breakReadsOf = (objectKey: string) => {
const s3 = env.server.stores.s3Object;
const client = env.server.clients.s3.get(s3.resolveRegion(null));
const send = client.send.bind(client);
return vi.spyOn(client, 'send').mockImplementation(((
command: unknown,
) => {
if (
!(command instanceof GetObjectCommand) ||
command.input.Key !== objectKey
) {
return send(command as never);
}
let reads = 0;
const body = new Readable({
read() {
reads += 1;
if (reads === 1) this.push('partial');
// Late enough for the headers and first chunk to go out.
else if (reads === 2)
setTimeout(
() => this.destroy(new Error('Checksum mismatch')),
25,
);
},
});
return Promise.resolve({
Body: body,
ContentLength: 64,
ContentType: 'text/plain',
});
}) as never);
};
it.each([
['/fs/read', 'path'],
['/read', 'file'],
])(
'GET %s fails that response and keeps serving others',
async (route, param) => {
const { username, token } = env.users.user;
const urlFor = (name: string) => {
const url = new URL(route, env.apiOrigin);
url.searchParams.set(param, `/${username}/Documents/${name}`);
return url;
};
const suffix = route.replace(/\W/g, '');
const broken = await writeFile(
`broken-${suffix}.txt`,
'x'.repeat(64),
);
await writeFile(`healthy-${suffix}.txt`, 'still fine');
const send = breakReadsOf(broken.uuid);
let status = 0;
let completed = false;
try {
const response = await fetch(urlFor(`broken-${suffix}.txt`), {
headers: { Authorization: `Bearer ${token}` },
});
status = response.status;
await response.arrayBuffer();
completed = true;
} catch {
// Expected: the transfer is cut short.
} finally {
send.mockRestore();
}
expect(status).toBe(200);
expect(completed).toBe(false);
const healthy = await fetch(urlFor(`healthy-${suffix}.txt`), {
headers: { Authorization: `Bearer ${token}` },
});
expect(healthy.status).toBe(200);
expect(await healthy.text()).toBe('still fine');
},
);
});
@@ -1948,6 +1948,27 @@ describe('FSController.readEntry (file streaming)', () => {
expect(captured.statusCode).toBe(206);
});
it('returns 206 with Range honored when offset and byte_count query params are supplied', async () => {
const { actor, userId } = await makeUser();
const username = actor.user!.username!;
const body = Buffer.from('abcdefghij');
const target = `/${username}/Documents/ranged-query.bin`;
await writeFile(userId, target, body, 'application/octet-stream');
const { res, captured } = makeStreamingRes();
await withActor(actor, () =>
controller.readEntry(
makeReq({
query: { path: target, offset: '2', byte_count: '4' },
actor,
}),
res,
),
);
expect(captured.statusCode).toBe(206);
expect(Buffer.concat(captured.bodyChunks).toString()).toBe('cdef');
});
it('throws 404 when the path does not exist', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
+241 -112
View File
@@ -20,12 +20,15 @@
import Busboy from 'busboy';
import type { Request, Response } from 'express';
import { posix as pathPosix } from 'node:path';
import { pipeline } from 'node:stream/promises';
import type { Actor } from '../../core/actor.js';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { Controller, Get, Post } from '../../core/http/decorators.js';
import { FS_ROUTE_REFUSAL } from '../../services/acl/ACLService.js';
import { pipeToResponse } from '../../core/http/pipeToResponse.js';
import {
FS_ROUTE_REFUSAL,
type ResourceDescriptor,
} from '../../services/acl/ACLService.js';
import { clientParentUid } from '../../services/fs/rootListing.js';
import {
expandTildePath,
@@ -42,7 +45,11 @@ import type {
UploadedBatchWriteItem,
UploadProgressTrackerLike,
} from '../../services/fs/types.js';
import type { FSEntry, FSEntryWriteInput } from '../../stores/fs/FSEntry.js';
import type {
FSEntry,
FSEntryWriteInput,
PendingUploadSession,
} from '../../stores/fs/FSEntry.js';
import {
runWithConcurrencyLimit,
runWithConcurrencyLimitSettled,
@@ -75,6 +82,8 @@ import {
type GuiItemEventName,
loadLegacyAssociatedApps,
replacedByMove,
resolveOrNotFound,
resolveReadRange,
signEntryThumbnail,
} from './legacyFsHelpers.js';
import type {
@@ -150,8 +159,14 @@ export class FSController extends PuterController {
);
}
// Write routes admit scoped access tokens for the same reason the reads
// do (see the note above `/stat`): every handler authorizes the target
// through the ACL, which intersects a token's grant with its issuer's, so
// a token writes exactly where it was granted write — the parity WebDAV
// already has. Upload sessions are re-checked on complete, sign and abort.
@Post('/startWrite', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
@@ -218,6 +233,7 @@ export class FSController extends PuterController {
@Post('/startBatchWrite', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
@@ -324,6 +340,7 @@ export class FSController extends PuterController {
@Post('/completeWrite', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
@@ -366,6 +383,7 @@ export class FSController extends PuterController {
@Post('/completeBatchWrite', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
@@ -396,26 +414,26 @@ export class FSController extends PuterController {
userId,
requests,
);
const updatedResponse = await runWithConcurrencyLimit(
response,
const withSideEffects = await this.#applyWriteResponsesSideEffects(
userId,
response.map((writeResponse, index) => {
const write: WriteResponse = {
fsEntry: writeResponse.fsEntry,
wasOverwrite: writeResponse.wasOverwrite,
requestedThumbnail: writeResponse.requestedThumbnail,
contentHashSha256: null,
};
return {
response: write,
guiMetadata: requests[index]?.guiMetadata,
};
}),
DEFAULT_BATCH_WRITE_SIDE_EFFECT_CONCURRENCY,
async (writeResponse, index) => {
const requestBody = requests[index];
const withSideEffects =
await this.#applyWriteResponseSideEffects(
userId,
{
fsEntry: writeResponse.fsEntry,
wasOverwrite: writeResponse.wasOverwrite,
requestedThumbnail:
writeResponse.requestedThumbnail,
contentHashSha256: null,
},
requestBody?.guiMetadata,
);
return { ...writeResponse, fsEntry: withSideEffects.fsEntry };
},
);
const updatedResponse = response.map((writeResponse, index) => ({
...writeResponse,
fsEntry: withSideEffects[index]?.fsEntry ?? writeResponse.fsEntry,
}));
res.json(
updatedResponse.map((r) => this.#withRequiredClientFsEntry(r)),
);
@@ -423,6 +441,7 @@ export class FSController extends PuterController {
@Post('/abortWrite', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
rateLimit: FS_MULTIPART_LIMIT,
})
@@ -437,12 +456,31 @@ export class FSController extends PuterController {
});
}
// An unknown id stays a no-op, as it always was.
try {
await this.services.fs.getUploadSessions(userId, [
req.body.uploadId,
]);
} catch (error) {
if (!(error instanceof HttpError && error.statusCode === 404)) {
throw error;
}
res.json({ ok: true });
return;
}
// Same re-check as complete/sign: a scoped token may only abort an
// upload whose target it could write.
await this.#assertUploadSessionWriteAccess(req, userId, [
req.body.uploadId,
]);
await this.services.fs.abortUrlWrite(userId, req.body.uploadId);
res.json({ ok: true });
}
@Post('/signMultipartParts', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
requireCredits: true,
rateLimit: FS_MULTIPART_LIMIT,
@@ -463,18 +501,22 @@ export class FSController extends PuterController {
{ legacyCode: 'bad_request' },
);
}
await this.#assertUploadSessionWriteAccess(req, userId, [
req.body?.uploadId,
]);
const sessions = await this.#assertUploadSessionWriteAccess(
req,
userId,
[req.body?.uploadId],
);
const response = await this.services.fs.signMultipartParts(
userId,
req.body,
sessions?.[0],
);
res.json(this.#withoutStorageInternals(response));
}
@Post('/write', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
requireCredits: true,
rateLimit: FS_WRITE_LIMIT,
@@ -530,6 +572,7 @@ export class FSController extends PuterController {
@Post('/batchWrite', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
requireCredits: true,
rateLimit: FS_WRITE_LIMIT,
@@ -865,17 +908,13 @@ export class FSController extends PuterController {
preparedBatch,
uploadedItems,
);
const updatedResponses = await runWithConcurrencyLimit(
writeResponses,
const updatedResponses = await this.#applyWriteResponsesSideEffects(
userId,
writeResponses.map((writeResponse, index) => ({
response: writeResponse,
guiMetadata: preparedBatch?.items[index]?.guiMetadata,
})),
32,
async (writeResponse, index) => {
const preparedItem = preparedBatch?.items[index];
return this.#applyWriteResponseSideEffects(
userId,
writeResponse,
preparedItem?.guiMetadata,
);
},
);
res.json(
updatedResponses.map((r) => this.#withRequiredClientFsEntry(r)),
@@ -988,17 +1027,13 @@ export class FSController extends PuterController {
preparedBatch,
uploadedItems,
);
const updatedResponses = await runWithConcurrencyLimit(
writeResponses,
const updatedResponses = await this.#applyWriteResponsesSideEffects(
userId,
writeResponses.map((writeResponse, index) => ({
response: writeResponse,
guiMetadata: filteredRequests[index]?.guiMetadata,
})),
32,
async (writeResponse, index) => {
const requestBody = filteredRequests[index];
return this.#applyWriteResponseSideEffects(
userId,
writeResponse,
requestBody?.guiMetadata,
);
},
);
res.json(
updatedResponses.map((r) => this.#withRequiredClientFsEntry(r)),
@@ -1021,7 +1056,9 @@ export class FSController extends PuterController {
const actor = this.#requireActor(req);
const body = this.#toObjectRecord(req.body);
const entry = await this.#resolveEntryForRequest(body);
await this.#assertAccess(actor, entry.path, 'see');
// One descriptor for both checks, so the chain is read at most once.
const descriptor = this.services.acl.fsDescriptor(entry.path);
await this.#assertAccess(actor, descriptor, 'see');
const wantsSize = this.#toBoolean(body.return_size);
const wantsShares = this.#toBoolean(body.return_shares);
@@ -1039,7 +1076,7 @@ export class FSController extends PuterController {
const mayReadSize =
entry.isDir &&
wantsSize &&
(await this.#canAccess(actor, entry.path, 'list'));
(await this.#canAccess(actor, descriptor, 'list'));
const [subtreeSize, suggestedApps, shareFlags, shares, parentUid] =
await Promise.all([
@@ -1459,10 +1496,7 @@ export class FSController extends PuterController {
);
}
const range =
typeof req.headers.range === 'string'
? req.headers.range
: undefined;
const range = resolveReadRange(req.headers.range, query);
const download = await this.services.fs.readContent(entry, {
range,
});
@@ -1488,19 +1522,14 @@ export class FSController extends PuterController {
);
res.status(range ? 206 : 200);
try {
await pipeline(download.body, res);
} catch {
// Client disconnect or upstream stream error — pipeline already
// tore down both ends. Response is partially sent; nothing to do.
return;
}
await pipeToResponse(download.body, res);
}
// -- Mutation routes ------------------------------------------------
@Post('/mkdir', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
@@ -1546,6 +1575,7 @@ export class FSController extends PuterController {
@Post('/touch', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
@@ -1586,6 +1616,7 @@ export class FSController extends PuterController {
@Post('/rename', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
@@ -1609,6 +1640,7 @@ export class FSController extends PuterController {
@Post('/delete', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
@@ -1635,6 +1667,7 @@ export class FSController extends PuterController {
@Post('/move', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
@@ -1688,6 +1721,7 @@ export class FSController extends PuterController {
@Post('/copy', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
requireCredits: true,
rateLimit: FS_MUTATE_LIMIT,
@@ -1739,6 +1773,7 @@ export class FSController extends PuterController {
@Post('/mkshortcut', {
subdomain: 'api',
allowAccessToken: true,
requireVerified: true,
rateLimit: FS_MUTATE_LIMIT,
})
@@ -1790,7 +1825,6 @@ export class FSController extends PuterController {
}
async #resolveEntryForRequest(source: Record<string, unknown>) {
const mod = await import('../../services/fs/resolveNode.js');
const username = Context.get('actor')?.user?.username;
const rawPath =
typeof source.path === 'string' ? source.path : undefined;
@@ -1810,15 +1844,8 @@ export class FSController extends PuterController {
? source.id
: undefined,
};
const entry = await mod.resolveNode(this.stores.fsEntry, ref, {
required: true,
});
if (!entry) {
throw new HttpError(404, 'Entry not found', {
legacyCode: 'not_found',
});
}
return entry;
// The resolved path is the owner's real one, so a miss must not echo it.
return resolveOrNotFound(this.stores.fsEntry, ref);
}
async #assertCanDedupeCreate(actor: Actor, targetPath: string) {
@@ -1835,16 +1862,20 @@ export class FSController extends PuterController {
/** As `#assertAccess`, but for what to include rather than what to serve. */
async #canAccess(
actor: Actor,
path: string,
path: string | ResourceDescriptor,
mode: 'see' | 'list' | 'read' | 'write',
): Promise<boolean> {
const acl = this.services.acl;
return acl.check(actor, acl.fsDescriptor(path), mode);
return acl.check(
actor,
typeof path === 'string' ? acl.fsDescriptor(path) : path,
mode,
);
}
async #assertAccess(
actor: Actor,
path: string,
path: string | ResourceDescriptor,
mode: 'see' | 'list' | 'read' | 'write',
) {
await this.services.acl.assertFsAccess(
@@ -2240,7 +2271,8 @@ export class FSController extends PuterController {
// the ActorUser type. Access via the escape hatch until a proper
// storage-quota mechanism is in place.
const actorUser = req.actor?.user as
Record<string, unknown> | undefined;
| Record<string, unknown>
| undefined;
const candidates = [
this.#toStorageCapacityCandidate(actorUser?.free_storage),
@@ -2478,19 +2510,24 @@ export class FSController extends PuterController {
* `startWrite` checked has to be re-checked against the session's recorded
* target every time the caller signs more parts or completes the upload —
* otherwise a revoked sharee still lands bytes in the owner's tree.
*
* Returns the sessions it checked, in request order, for signing to reuse;
* undefined when any id was left to the service to read and reject.
* Completion reads its own, so a session settled during this check still
* gets a 409.
*/
async #assertUploadSessionWriteAccess(
req: Request,
userId: number,
uploadIds: Array<string | undefined>,
): Promise<void> {
): Promise<PendingUploadSession[] | undefined> {
// A malformed id is left to the service, which owns that error shape.
const knownUploadIds = uploadIds.filter(
(uploadId): uploadId is string =>
typeof uploadId === 'string' && uploadId.length > 0,
);
if (knownUploadIds.length === 0) {
return;
return undefined;
}
const sessions = await this.services.fs.getUploadSessions(
userId,
@@ -2505,6 +2542,9 @@ export class FSController extends PuterController {
})),
{ pathAlreadyNormalized: true },
);
return knownUploadIds.length === uploadIds.length
? sessions
: undefined;
}
#toEventGuiMetadata(
@@ -2619,16 +2659,16 @@ export class FSController extends PuterController {
return this.#estimateDataUrlSize(thumbnail) > MAX_THUMBNAIL_BYTES;
}
async #applyThumbnailAfterWrite(
userId: number,
/** The thumbnail a write should store on `fsEntry`, or null for none. */
async #thumbnailToStore(
fsEntry: FSEntry,
requestedThumbnail: string | null | undefined,
): Promise<FSEntry> {
): Promise<string | null> {
if (!requestedThumbnail || this.#isAppDataPath(fsEntry.path)) {
return fsEntry;
return null;
}
if (this.#isOversizedThumbnailDataUrl(requestedThumbnail)) {
return fsEntry;
return null;
}
const thumbnailPayload = {
@@ -2649,15 +2689,10 @@ export class FSController extends PuterController {
? thumbnailPayload.url
: null;
if (finalThumbnail === fsEntry.thumbnail || finalThumbnail === null) {
return fsEntry;
if (finalThumbnail === fsEntry.thumbnail) {
return null;
}
return this.services.fs.updateEntryThumbnail(
userId,
fsEntry.uuid,
finalThumbnail,
);
return finalThumbnail;
}
#toThumbnailPrepareItem(
@@ -2887,36 +2922,130 @@ export class FSController extends PuterController {
response: WriteResponse,
guiMetadata: WriteGuiMetadata | undefined,
): Promise<WriteResponse> {
let fsEntry = response.fsEntry;
const [result] = await this.#applyWriteResponsesSideEffects(
userId,
[{ response, guiMetadata }],
1,
);
return result ?? response;
}
const hashEventPromise = this.#runNonCritical(async () => {
await this.#emitWriteHashEvent(
response.contentHashSha256,
fsEntry.uuid,
);
}, 'emitWriteHashEvent');
/**
* Each write's hash event, thumbnail and GUI event. Thumbnails are vetted
* `concurrency` writes at a time, then the caller's own go to the store in
* one update; a write into someone else's folder is updated alone.
*/
async #applyWriteResponsesSideEffects(
userId: number,
writes: Array<{
response: WriteResponse;
guiMetadata: WriteGuiMetadata | undefined;
}>,
concurrency: number,
): Promise<WriteResponse[]> {
type Write = (typeof writes)[number];
const results: WriteResponse[] = new Array(writes.length);
const finish = async (
index: number,
{ response, guiMetadata }: Write,
fsEntry: FSEntry,
hashEventPromise: Promise<void>,
) => {
await this.#runNonCritical(async () => {
await this.#emitGuiWriteEvent(
response.wasOverwrite
? 'outer.gui.item.updated'
: 'outer.gui.item.added',
fsEntry,
guiMetadata,
);
}, 'emitGuiWriteEvent');
await hashEventPromise;
results[index] = { ...response, fsEntry };
};
await this.#runNonCritical(async () => {
fsEntry = await this.#applyThumbnailAfterWrite(
userId,
response.fsEntry,
response.requestedThumbnail,
);
}, 'applyThumbnailAfterWrite');
const toStore: Array<{
index: number;
write: Write;
thumbnail: string;
hashEventPromise: Promise<void>;
}> = [];
await runWithConcurrencyLimit(
writes,
concurrency,
async (write, index) => {
const { fsEntry, contentHashSha256, requestedThumbnail } =
write.response;
const hashEventPromise = this.#runNonCritical(async () => {
await this.#emitWriteHashEvent(
contentHashSha256,
fsEntry.uuid,
);
}, 'emitWriteHashEvent');
let thumbnail = null as string | null;
await this.#runNonCritical(async () => {
thumbnail = await this.#thumbnailToStore(
fsEntry,
requestedThumbnail,
);
}, 'applyThumbnailAfterWrite');
if (thumbnail === null) {
await finish(index, write, fsEntry, hashEventPromise);
return;
}
toStore.push({ index, write, thumbnail, hashEventPromise });
},
);
if (toStore.length === 0) return results;
await this.#runNonCritical(async () => {
await this.#emitGuiWriteEvent(
response.wasOverwrite
? 'outer.gui.item.updated'
: 'outer.gui.item.added',
fsEntry,
guiMetadata,
);
}, 'emitGuiWriteEvent');
// The update is scoped to the caller's rows, so batching anyone else's
// would only fail the batch.
const own = toStore.filter(
({ write }) => write.response.fsEntry.userId === userId,
);
const groups = [
own,
...toStore
.filter(({ write }) => write.response.fsEntry.userId !== userId)
.map((item) => [item]),
].filter((group) => group.length > 0);
const stored = new Map<number, PromiseSettledResult<FSEntry>>();
await runWithConcurrencyLimit(groups, concurrency, async (group) => {
let outcomes: PromiseSettledResult<FSEntry>[];
try {
outcomes = await this.services.fs.updateEntryThumbnails(
userId,
group.map(({ write, thumbnail }) => ({
uuid: write.response.fsEntry.uuid,
thumbnail,
})),
);
} catch (error) {
outcomes = group.map(() => ({
status: 'rejected',
reason: error,
}));
}
group.forEach(({ index }, i) => {
const outcome = outcomes[i];
if (outcome) stored.set(index, outcome);
});
});
await hashEventPromise;
return { ...response, fsEntry };
await runWithConcurrencyLimit(
toStore,
concurrency,
async ({ index, write, hashEventPromise }) => {
let fsEntry = write.response.fsEntry;
await this.#runNonCritical(async () => {
const outcome = stored.get(index);
if (outcome?.status === 'rejected') throw outcome.reason;
if (outcome) fsEntry = outcome.value;
}, 'applyThumbnailAfterWrite');
await finish(index, write, fsEntry, hashEventPromise);
},
);
return results;
}
#shouldIgnoreUploadPath(targetPath: string): boolean {
@@ -26,6 +26,7 @@ import type { Actor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { toPendingUploadSessionKey } from '../../stores/fs/pendingUploadSessionHelpers.js';
import { generateDefaultFsentries } from '../../util/userProvisioning.js';
import type { FSController } from './FSController.js';
import { FS_BATCH_WRITE_MAX_ITEMS, FS_MULTIPART_MAX_PARTS } from './limits.js';
@@ -1256,8 +1257,271 @@ describe('FSController.completeWrite', () => {
),
).rejects.toMatchObject({ statusCode: 404 });
});
// One read for the access check and one for the status decision; the
// completion is recorded from the second without reading it back.
it('reads the session twice and records the completion from the second copy', async () => {
const { actor, username } = await makeUser();
const target = `/${username}/Documents/complete-once.txt`;
const started = await startSignedWrite(actor, target, 4);
await fetch(started.url!, { method: 'PUT', body: 'wxyz' });
const before = (await server.stores.fsEntry.getPendingEntryBySessionId(
started.sessionId,
))!;
const reads = countSessionReads();
try {
await withActor(actor, () =>
controller.completeWrite(
makeReq<CompleteWriteRequest>({
body: { uploadId: started.sessionId },
actor,
}),
makeRes().res,
),
);
expect(reads.of(started.sessionId)).toBe(2);
} finally {
reads.restore();
}
const after = (await server.stores.fsEntry.getPendingEntryBySessionId(
started.sessionId,
))!;
expect(after).toEqual({
...before,
status: 'completed',
failureReason: null,
updatedAt: after.completedAt,
consumedAt: after.completedAt,
completedAt: expect.any(Number),
});
});
it('reads each session of a batch completion twice', async () => {
const { actor, username } = await makeUser();
const started = [];
for (const name of ['batch-once-a.txt', 'batch-once-b.txt']) {
const session = await startSignedWrite(
actor,
`/${username}/Documents/${name}`,
2,
);
await fetch(session.url!, { method: 'PUT', body: 'ab' });
started.push(session);
}
const reads = countSessionReads();
try {
const { res, captured } = makeRes();
await withActor(actor, () =>
controller.completeBatchWrites(
makeReq<CompleteWriteRequest[]>({
body: started.map((session) => ({
uploadId: session.sessionId,
})),
actor,
}),
res,
),
);
expect(
(captured.body as Array<{ fsEntry: { path: string } }>).map(
(item) => item.fsEntry.path,
),
).toEqual([
`/${username}/Documents/batch-once-a.txt`,
`/${username}/Documents/batch-once-b.txt`,
]);
for (const session of started) {
expect(reads.of(session.sessionId)).toBe(2);
}
} finally {
reads.restore();
}
for (const session of started) {
await expect(
server.stores.fsEntry.getPendingEntryBySessionId(
session.sessionId,
),
).resolves.toMatchObject({ status: 'completed' });
}
});
it('leaves a malformed id in a batch to the service', async () => {
const { actor, username } = await makeUser();
const started = await startSignedWrite(
actor,
`/${username}/Documents/batch-malformed.txt`,
2,
);
await expect(
withActor(actor, () =>
controller.completeBatchWrites(
makeReq<CompleteWriteRequest[]>({
body: [
{ uploadId: started.sessionId },
{ uploadId: '' },
],
actor,
}),
makeRes().res,
),
),
).rejects.toMatchObject({ statusCode: 404 });
});
// The access check reads the session before the status decision; another
// request that settles it in between must still turn this one away.
const settleDuringAccessCheck = (settle: () => Promise<void>) => {
const fs = server.services.fs;
const read = fs.getUploadSessions.bind(fs);
return vi
.spyOn(fs, 'getUploadSessions')
.mockImplementationOnce(async (userId, uploadIds) => {
const sessions = await read(userId, uploadIds);
await settle();
return sessions;
});
};
it('turns away a completion whose session another one finished during the access check', async () => {
const { actor, userId, username } = await makeUser();
const target = `/${username}/Documents/raced-complete.txt`;
const started = await startSignedWrite(actor, target, 4);
await fetch(started.url!, { method: 'PUT', body: 'wxyz' });
const spy = settleDuringAccessCheck(async () => {
await server.services.fs.completeUrlWrite(userId, {
uploadId: started.sessionId,
});
});
try {
await expect(
withActor(actor, () =>
controller.completeWrite(
makeReq<CompleteWriteRequest>({
body: { uploadId: started.sessionId },
actor,
}),
makeRes().res,
),
),
).rejects.toMatchObject({
statusCode: 409,
message: expect.stringContaining('not pending'),
});
} finally {
spy.mockRestore();
}
await expect(
server.stores.fsEntry.getPendingEntryBySessionId(started.sessionId),
).resolves.toMatchObject({ status: 'completed' });
// A completed session's object backs the live file; a late abort must
// leave it alone.
await server.services.fs.abortUrlWrite(userId, started.sessionId);
const entry = (await server.stores.fsEntry.getEntryByPath(target))!;
const { body } = await server.services.fs.readContent(entry);
expect(await readAll(body)).toBe('wxyz');
});
it('turns away a completion whose session was aborted during the access check', async () => {
const { actor, userId, username } = await makeUser();
const target = `/${username}/Documents/raced-abort.txt`;
const started = await startSignedWrite(actor, target, 4);
await fetch(started.url!, { method: 'PUT', body: 'wxyz' });
const spy = settleDuringAccessCheck(() =>
server.services.fs.abortUrlWrite(userId, started.sessionId),
);
try {
await expect(
withActor(actor, () =>
controller.completeWrite(
makeReq<CompleteWriteRequest>({
body: { uploadId: started.sessionId },
actor,
}),
makeRes().res,
),
),
).rejects.toMatchObject({
statusCode: 409,
message: expect.stringContaining('not pending'),
});
} finally {
spy.mockRestore();
}
await expect(
server.stores.fsEntry.getPendingEntryBySessionId(started.sessionId),
).resolves.toMatchObject({
status: 'aborted',
failureReason: 'Upload aborted by caller',
});
await expect(
server.stores.fsEntry.getEntryByPath(target),
).resolves.toBeNull();
});
it('turns away a batch completion whose session another one finished during the access check', async () => {
const { actor, userId, username } = await makeUser();
const target = `/${username}/Documents/raced-batch.txt`;
const started = await startSignedWrite(actor, target, 2);
await fetch(started.url!, { method: 'PUT', body: 'ab' });
const spy = settleDuringAccessCheck(async () => {
await server.services.fs.batchCompleteUrlWrite(userId, [
{ uploadId: started.sessionId },
]);
});
try {
await expect(
withActor(actor, () =>
controller.completeBatchWrites(
makeReq<CompleteWriteRequest[]>({
body: [{ uploadId: started.sessionId }],
actor,
}),
makeRes().res,
),
),
).rejects.toMatchObject({
statusCode: 409,
message: expect.stringContaining('not pending'),
});
} finally {
spy.mockRestore();
}
await expect(
server.stores.fsEntry.getPendingEntryBySessionId(started.sessionId),
).resolves.toMatchObject({ status: 'completed' });
});
});
const readAll = async (body: Readable): Promise<string> => {
const chunks: Buffer[] = [];
for await (const chunk of body) chunks.push(Buffer.from(chunk));
return Buffer.concat(chunks).toString('utf8');
};
/** Counts system KV reads of each upload session's key. */
const countSessionReads = () => {
const get = vi.spyOn(server.stores.kv, 'get');
return {
of: (sessionId: string) =>
get.mock.calls
.flatMap(([params]) =>
[(params as { key: unknown }).key].flat(),
)
.filter((key) => key === toPendingUploadSessionKey(sessionId))
.length,
restore: () => get.mockRestore(),
};
};
describe('FSController.abortWrite', () => {
it('rejects a request with no uploadId', async () => {
const { actor } = await makeUser();
@@ -1381,6 +1645,48 @@ describe('FSController.signMultipartParts', () => {
).rejects.toMatchObject({ statusCode: 404 });
});
it('reads the session once to sign more parts', async () => {
const { actor, username } = await makeUser();
const { res: startRes, captured: startCaptured } = makeRes();
await withActor(actor, () =>
controller.startWrite(
makeReq<SignedWriteRequest>({
body: {
fileMetadata: {
path: `/${username}/Documents/sign-once.bin`,
size: 8,
},
uploadMode: 'multipart',
},
actor,
}),
startRes,
),
);
const started = startCaptured.body as ClientSignedWriteResponse;
const reads = countSessionReads();
try {
const { res, captured } = makeRes();
await withActor(actor, () =>
controller.signMultipartParts(
makeReq<SignMultipartPartsRequest>({
body: { uploadId: started.sessionId, partNumbers: [1] },
actor,
}),
res,
),
);
expect(captured.body).toMatchObject({
uploadId: started.sessionId,
multipartPartUrls: [expect.anything()],
});
expect(reads.of(started.sessionId)).toBe(1);
} finally {
reads.restore();
}
});
// The session is unknown, so reaching the lookup would answer 404.
it.each([
[FS_MULTIPART_MAX_PARTS + 1, 400],
@@ -1447,6 +1753,66 @@ describe('FSController.batchWrites (json)', () => {
}
});
it('stores a batch of thumbnails in one update and announces each with its thumbnail', async () => {
const { actor, username } = await makeUser();
const names = ['th-a.png', 'th-b.png', 'th-c.png', 'th-none.png'];
const thumbnailFor = (name: string) =>
name === 'th-none.png'
? undefined
: `data:image/png;base64,${Buffer.from(name).toString('base64')}`;
const writeSpy = vi.spyOn(server.clients.db, 'write');
const announced: Array<{ response: Record<string, unknown> }> = [];
const listener = (_key: string, data: unknown) => {
announced.push(data as { response: Record<string, unknown> });
};
server.clients.event.on('outer.gui.item.added', listener as never);
const { res, captured } = makeRes();
let statements: string[];
try {
await withActor(actor, () =>
controller.batchWrites(
makeReq<WriteRequest[]>({
body: names.map((name) => ({
fileMetadata: {
path: `/${username}/Documents/${name}`,
size: 1,
},
fileContent: 'x',
thumbnailData: thumbnailFor(name),
})) as WriteRequest[],
actor,
}),
res,
),
);
} finally {
statements = writeSpy.mock.calls.map(([sql]) => String(sql));
writeSpy.mockRestore();
server.clients.event.off('outer.gui.item.added', listener as never);
}
expect(
statements.filter((sql) => sql.includes('SET thumbnail')),
).toHaveLength(1);
const body = captured.body as Array<{
fsEntry: { path: string; thumbnail?: string | null };
}>;
expect(body.map((item) => item.fsEntry.path)).toEqual(
names.map((name) => `/${username}/Documents/${name}`),
);
for (const name of names) {
const path = `/${username}/Documents/${name}`;
const expected = thumbnailFor(name) ?? null;
await expect(
server.stores.fsEntry.getEntryByUuidFromPrimary(
(await server.stores.fsEntry.getEntryByPath(path))!.uuid,
),
).resolves.toMatchObject({ thumbnail: expected });
const event = announced.find((data) => data.response.path === path);
expect(event?.response.thumbnail ?? null).toBe(expected);
}
});
it('returns [] for a non-array body', async () => {
const { actor } = await makeUser();
const { res, captured } = makeRes();
@@ -0,0 +1,304 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { GetObjectCommand } from '@aws-sdk/client-s3';
import { randomBytes } from 'node:crypto';
import { Readable } from 'node:stream';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
/**
* The v1 download routes over real HTTP: what reaches the client, and what
* happens to the storage stream when the client goes away mid-download.
*/
describe('legacy file downloads over HTTP', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const writeFile = async (name: string, content: Buffer) => {
const { username } = env.users.user;
const user = await env.server.stores.user.getByUsername(username);
const path = `/${username}/Documents/${name}`;
const { fsEntry } = await env.server.services.fs.write(user!.id, {
fileMetadata: {
path,
size: content.byteLength,
contentType: 'application/octet-stream',
},
fileContent: content,
});
return { path, uuid: fsEntry.uuid };
};
const signedReadUrl = async (path: string): Promise<URL> => {
const signed = await fetch(new URL('/sign', env.apiOrigin), {
method: 'POST',
headers: {
Authorization: `Bearer ${env.users.user.token}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({ items: [{ path, action: 'read' }] }),
});
expect(signed.status).toBe(200);
const readUrl = new URL(
(await signed.json()).signatures[0].read_url as string,
);
return new URL(`${readUrl.pathname}${readUrl.search}`, env.apiOrigin);
};
const accessTokenFor = async (uuid: string): Promise<string> => {
const user = await env.server.stores.user.getByUsername(
env.users.user.username,
);
return env.server.services.auth.createAccessToken(
{ user } as never,
[[`fs:${uuid}:read`]],
{ label: 'legacy-download-http' },
);
};
type Route = '/read' | '/token-read' | '/file' | '/down';
const routes: Route[] = ['/read', '/token-read', '/file', '/down'];
const requestFor = async (
route: Route,
file: { path: string; uuid: string },
init: RequestInit & { headers?: Record<string, string> } = {},
): Promise<Response> => {
const { token, apiToken } = env.users.user;
let url: URL;
let base: RequestInit & { headers?: Record<string, string> } = {};
if (route === '/read') {
url = new URL('/read', env.apiOrigin);
url.searchParams.set('file', file.path);
base = { headers: { Authorization: `Bearer ${token}` } };
} else if (route === '/token-read') {
url = new URL('/token-read', env.apiOrigin);
url.searchParams.set('uid', file.uuid);
url.searchParams.set('token', await accessTokenFor(file.uuid));
} else if (route === '/file') {
url = await signedReadUrl(file.path);
} else {
url = new URL('/down', env.origin);
url.searchParams.set('path', file.path);
base = {
method: 'POST',
headers: { Authorization: `Bearer ${apiToken}` },
};
}
return fetch(url, {
...base,
...init,
headers: { ...base.headers, ...init.headers },
});
};
it.each(routes)(
'GET %s returns the stored bytes and headers unchanged',
async (route) => {
const content = randomBytes(256 * 1024 + 7);
const name = `bytes-${route.replace(/\W/g, '')}.bin`;
const file = await writeFile(name, content);
const raw = await requestFor(route, file, {
headers: { 'Accept-Encoding': 'identity' },
});
expect(raw.status).toBe(200);
expect(raw.headers.get('content-length')).toBe(
String(content.byteLength),
);
expect(raw.headers.get('content-type')).toBe(
'application/octet-stream',
);
expect(raw.headers.get('content-disposition')).toBe(
`${route === '/down' ? 'attachment' : 'inline'}; filename="${name}"`,
);
expect(raw.headers.get('etag')).toEqual(expect.any(String));
expect(raw.headers.get('last-modified')).toEqual(
expect.any(String),
);
expect(Buffer.from(await raw.arrayBuffer()).equals(content)).toBe(
true,
);
// Compressed on the wire by default; the bytes still match.
const compressed = await requestFor(route, file);
expect(compressed.headers.get('content-encoding')).toBe('gzip');
expect(
Buffer.from(await compressed.arrayBuffer()).equals(content),
).toBe(true);
},
);
// Storage hands over one chunk and then goes quiet, as a large object
// does while the client is still reading.
const stallReadsOf = (objectKey: string) => {
const s3 = env.server.stores.s3Object;
const client = env.server.clients.s3.get(s3.resolveRegion(null));
const send = client.send.bind(client);
const bodies: Readable[] = [];
const spy = vi.spyOn(client, 'send').mockImplementation(((
command: unknown,
) => {
if (
!(command instanceof GetObjectCommand) ||
command.input.Key !== objectKey
) {
return send(command as never);
}
let pushed = false;
const body = new Readable({
read() {
if (!pushed) {
pushed = true;
this.push(Buffer.alloc(16 * 1024, 1));
}
},
});
bodies.push(body);
return Promise.resolve({
Body: body,
ContentLength: 1024 * 1024,
ContentType: 'application/octet-stream',
});
}) as never);
return { spy, bodies };
};
it.each(routes)(
'GET %s releases the storage stream when the client aborts',
async (route) => {
const file = await writeFile(
`abort-${route.replace(/\W/g, '')}.bin`,
Buffer.alloc(1024 * 1024, 1),
);
const { spy, bodies } = stallReadsOf(file.uuid);
try {
const abort = new AbortController();
// Uncompressed, so the first chunk reaches the client as-is.
const response = await requestFor(route, file, {
signal: abort.signal,
headers: { 'Accept-Encoding': 'identity' },
});
expect(response.status).toBe(200);
const reader = response.body!.getReader();
await reader.read();
abort.abort();
expect(bodies).toHaveLength(1);
await vi.waitFor(() => expect(bodies[0]!.destroyed).toBe(true));
} finally {
spy.mockRestore();
}
// The server keeps serving after the aborted transfer.
const healthy = await writeFile(
`after-abort-${route.replace(/\W/g, '')}.txt`,
Buffer.from('still fine'),
);
const next = await requestFor(route, healthy);
expect(next.status).toBe(200);
expect(await next.text()).toBe('still fine');
},
);
// Storage fails, either before its first byte or after one chunk.
const failReadsOf = (objectKey: string, sendFirst: boolean) => {
const s3 = env.server.stores.s3Object;
const client = env.server.clients.s3.get(s3.resolveRegion(null));
const send = client.send.bind(client);
return vi.spyOn(client, 'send').mockImplementation(((
command: unknown,
) => {
if (
!(command instanceof GetObjectCommand) ||
command.input.Key !== objectKey
) {
return send(command as never);
}
let reads = 0;
const body = new Readable({
read() {
reads++;
if (sendFirst && reads === 1) {
this.push(Buffer.alloc(16 * 1024, 1));
return;
}
setImmediate(() =>
this.destroy(new Error('storage socket reset')),
);
},
});
return Promise.resolve({
Body: body,
ContentLength: 1024 * 1024,
ContentType: 'application/octet-stream',
});
}) as never);
};
it.each(
routes.flatMap((route) => [
[route, true] as const,
[route, false] as const,
]),
)(
'GET %s drops the connection when storage fails (bytes sent first: %s)',
async (route, sendFirst) => {
const file = await writeFile(
`fail-${sendFirst ? 'mid' : 'start'}-${route.replace(/\W/g, '')}.bin`,
Buffer.alloc(1024 * 1024, 1),
);
const spy = failReadsOf(file.uuid, sendFirst);
try {
// No error body after a partial one: the client sees the
// transfer cut off, either at the headers or mid-body.
const outcome = await (async () => {
try {
const response = await requestFor(route, file, {
headers: { 'Accept-Encoding': 'identity' },
});
await response.arrayBuffer();
return `completed ${response.status}`;
} catch {
return 'cut off';
}
})();
expect(outcome).toBe('cut off');
} finally {
spy.mockRestore();
}
const healthy = await writeFile(
`after-fail-${sendFirst ? 'mid' : 'start'}-${route.replace(/\W/g, '')}.txt`,
Buffer.from('still fine'),
);
const next = await requestFor(route, healthy);
expect(next.status).toBe(200);
expect(await next.text()).toBe('still fine');
},
);
});
@@ -467,6 +467,88 @@ describe('LegacyFSController.stat', () => {
expect(body.size).toBe(0);
});
it('reports is_empty from the same children a listing returns', async () => {
const { actor, userId } = await makeUser();
const username = actor.user!.username!;
const emptyPath = `/${username}/Documents/probe-empty`;
const fullPath = `/${username}/Documents/probe-full`;
for (const path of [emptyPath, fullPath]) {
await withActor(actor, () =>
controller.mkdir(
makeReq({ body: { path }, actor }),
makeRes().res,
),
);
}
await server.services.fs.write(userId, {
fileMetadata: {
path: `${fullPath}/inner.txt`,
size: 5,
contentType: 'text/plain',
},
fileContent: 'hello',
} as never);
for (const path of [emptyPath, fullPath]) {
const entry = (await server.stores.fsEntry.getEntryByPath(path))!;
const listed = await server.stores.fsEntry.listChildren(
entry.uuid,
{ limit: 1 },
);
const { res, captured } = makeRes();
await withActor(actor, () =>
controller.stat(
makeReq({ body: { path, return_size: true }, actor }),
res,
),
);
const body = captured.body as Record<string, unknown>;
expect(body.is_empty).toBe(listed.length === 0);
expect(body.size).toBe(path === fullPath ? 5 : 0);
expect(body.owner).toEqual({ username });
}
});
it('surfaces an earlier failure ahead of a failed subtree size, with nothing left unhandled', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const path = `/${username}/Documents/size-order`;
await withActor(actor, () =>
controller.mkdir(makeReq({ body: { path }, actor }), makeRes().res),
);
const flagsFailure = new Error('share flags failed');
const sizeFailure = new Error('subtree size failed');
const flags = vi
.spyOn(server.services.share, 'shareFlags')
.mockRejectedValueOnce(flagsFailure);
const size = vi
.spyOn(server.services.fs, 'getSubtreeSize')
.mockRejectedValue(sizeFailure);
try {
await expect(
withActor(actor, () =>
controller.stat(
makeReq({ body: { path, return_size: true }, actor }),
makeRes().res,
),
),
).rejects.toBe(flagsFailure);
// On its own, the size failure is what the caller sees.
await expect(
withActor(actor, () =>
controller.stat(
makeReq({ body: { path, return_size: true }, actor }),
makeRes().res,
),
),
).rejects.toBe(sizeFailure);
} finally {
flags.mockRestore();
size.mockRestore();
}
});
it('reports the share flag, and the recipients when asked', async () => {
const { actor } = await makeUser();
const recipient = await makeUser();
@@ -3650,6 +3732,63 @@ describe('LegacyFSController.read file streaming', () => {
);
expect(captured.headers['Content-Type']).toMatch(/text\/html/);
});
it('honors offset and byte_count query params on /read', async () => {
const { actor, userId } = await makeUser();
const username = actor.user!.username!;
const body = Buffer.from('0123456789abcdef');
const target = `/${username}/Documents/range-test.txt`;
await server.services.fs.write(userId, {
fileMetadata: {
path: target,
size: body.byteLength,
contentType: 'text/plain',
},
fileContent: body,
});
const { res, captured } = makeStreamingRes();
await withActor(actor, () =>
controller.read(
makeReq({
query: { file: target, offset: '4', byte_count: '6' },
actor,
}),
res,
),
);
await new Promise<void>((resolve) => setImmediate(resolve));
expect(captured.statusCode).toBe(206);
expect(Buffer.concat(captured.bodyChunks).toString()).toBe('456789');
});
it('rejects invalid offset query param with 400', async () => {
const { actor, userId } = await makeUser();
const username = actor.user!.username!;
const body = Buffer.from('test');
const target = `/${username}/Documents/invalid-offset.txt`;
await server.services.fs.write(userId, {
fileMetadata: {
path: target,
size: body.byteLength,
contentType: 'text/plain',
},
fileContent: body,
});
const { res } = makeStreamingRes();
await expect(
withActor(actor, () =>
controller.read(
makeReq({
query: { file: target, offset: '-1' },
actor,
}),
res,
),
),
).rejects.toMatchObject({ statusCode: 400 });
});
});
// ── /file directory listing ─────────────────────────────────────────
@@ -17,6 +17,7 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Team2faLookup } from '../../core/http/middleware/gates.js';
import Busboy from 'busboy';
import type { NextFunction, Request, RequestHandler, Response } from 'express';
import { contentType as contentTypeFromMime } from 'mime-types';
@@ -30,6 +31,7 @@ import {
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { RouteOptions } from '../../core/http/index.js';
import { pipeToResponse } from '../../core/http/pipeToResponse.js';
import {
assertNotSuspended,
assertTeam2fa,
@@ -51,6 +53,7 @@ import {
} from '../../util/fileSigning.js';
import { APP_ICON_SIZES } from '../../util/appIcon.js';
import { toAppSummary } from '../../util/appView.js';
import { awaitLater } from '../../util/concurrency.js';
import {
joinChildPath,
normalizeAbsolutePath,
@@ -100,6 +103,7 @@ import {
type GuiItemEventName,
loadLegacyAssociatedApps,
replacedByMove,
resolveReadRange,
resolveV1Selector,
signEntry,
signingConfigFromAppConfig,
@@ -209,12 +213,19 @@ export class LegacyFSController extends PuterController {
// Limits come from `./limits` and carry an explicit `scope`, so these
// draw from the same per-user budget as their v2 counterparts rather
// than handing a caller a second allowance for the same operation.
const mutate = { ...apiOptions, rateLimit: FS_MUTATE_LIMIT };
// Read-side routes admit scoped access tokens: the ACL check each
// handler runs already intersects a token's grant with its issuer's,
// so the token reaches exactly what it was minted for. That is what a
// background events handler acts through — its `user` is a token
// scoped to the subscription's grant.
//
// The entry routes — reads and writes alike — admit scoped access
// tokens: the ACL check each handler runs already intersects a token's
// grant with its issuer's, so the token reaches exactly what it was
// minted for. That is what a background events handler acts through —
// its `user` is a token scoped to the subscription's grant — and it
// matches what WebDAV admits. Routes reaching past a single entry
// (search, df, sign, the app helpers) stay closed to them.
const mutate = {
...apiOptions,
allowAccessToken: true,
rateLimit: FS_MUTATE_LIMIT,
} as RouteOptions;
const reads = { ...apiOptions, allowAccessToken: true } as RouteOptions;
router.post('/stat', { ...reads, rateLimit: FS_STAT_LIMIT }, this.stat);
router.post(
@@ -264,6 +275,8 @@ export class LegacyFSController extends PuterController {
'/batch',
{
...spends,
// Every op authorizes its own target.
allowAccessToken: true,
rateLimit: FS_BATCH_LIMIT,
concurrent: FS_BATCH_CONCURRENT,
},
@@ -494,6 +507,16 @@ export class LegacyFSController extends PuterController {
'see',
);
// Started early but awaited where it used to run, so its error still
// surfaces after the ones below. awaitLater covers the early exits
// that never reach that await.
const subtreeSize =
entry.isDir && getBoolean(body, 'return_size')
? awaitLater(
this.services.fs.getSubtreeSize(userId, entry.path),
)
: null;
const [suggestedApps, appsById, shareFlags, parentUid] =
await Promise.all([
this.services.suggestedApps.getSuggestedApps(entry),
@@ -521,12 +544,7 @@ export class LegacyFSController extends PuterController {
});
// Optional hydrations:
if (entry.isDir && getBoolean(body, 'return_size')) {
shaped.size = await this.services.fs.getSubtreeSize(
userId,
entry.path,
);
}
if (subtreeSize) shaped.size = await subtreeSize;
// `return_versions` has no parity yet; the empty array keeps `forEach`
// callers working. `return_owner` is a no-op — `owner` is always set.
if (getBoolean(body, 'return_versions')) shaped.versions = [];
@@ -914,7 +932,9 @@ export class LegacyFSController extends PuterController {
// Trash, and `null`/`{}` when restoring. See
// `src/gui/src/helpers.js` → `window.move_items`.
newMetadata: (body.new_metadata ?? undefined) as
Record<string, unknown> | null | undefined,
| Record<string, unknown>
| null
| undefined,
});
const overwrittenEntry = replacedByMove(overwriteTarget, moved, name);
const oldPath = source.path;
@@ -1269,10 +1289,7 @@ export class LegacyFSController extends PuterController {
});
}
const range =
typeof req.headers.range === 'string'
? req.headers.range
: undefined;
const range = resolveReadRange(req.headers.range, query);
const download = await this.services.fs.readContent(entry, {
range,
});
@@ -1305,10 +1322,7 @@ export class LegacyFSController extends PuterController {
);
res.status(range ? 206 : 200);
download.body.on('error', (err) => {
res.destroy(err);
});
download.body.pipe(res);
void pipeToResponse(download.body, res);
};
tokenRead = async (req: Request, res: Response): Promise<void> => {
@@ -1333,7 +1347,7 @@ export class LegacyFSController extends PuterController {
// that guard every other authenticated FS route have to run here.
assertNotSuspended(actor!.user);
assertVerifiedAccount(actor!.user);
await assertTeam2fa(actor!.user, this.stores.team);
await assertTeam2fa(actor!.user, this.stores.team as Team2faLookup);
req.actor = assertResolvedActor(actor!);
Context.set('actor', actor);
@@ -1391,7 +1405,8 @@ export class LegacyFSController extends PuterController {
}
type SignedOrEmpty =
(SignedFile & { path?: string }) | Record<string, never>;
| (SignedFile & { path?: string })
| Record<string, never>;
const result: { signatures: SignedOrEmpty[]; token?: string } = {
signatures: [],
};
@@ -1439,11 +1454,13 @@ export class LegacyFSController extends PuterController {
}
// ACL: always require read; downgrade write→read silently.
// One descriptor for both modes reads the chain at most once.
const descriptor = this.services.acl.fsDescriptor(entry.path);
await assertAccess(
this.services.acl,
this.services.fs,
actor,
entry.path,
descriptor,
'read',
);
let finalAction: 'read' | 'write' = 'read';
@@ -1455,7 +1472,7 @@ export class LegacyFSController extends PuterController {
// publishes the descriptor, so there is nothing to hide.
const writeOk = await this.services.acl.check(
actor,
this.services.acl.fsDescriptor(entry.path),
descriptor,
'write',
);
finalAction = writeOk ? 'write' : 'read';
@@ -1785,10 +1802,7 @@ export class LegacyFSController extends PuterController {
}
// File: stream bytes with Range support.
const range =
typeof req.headers.range === 'string'
? req.headers.range
: undefined;
const range = resolveReadRange(req.headers.range, query);
const download = await this.services.fs.readContent(entry, {
range,
});
@@ -1819,10 +1833,7 @@ export class LegacyFSController extends PuterController {
);
res.status(range ? 206 : 200);
download.body.on('error', (err) => {
res.destroy(err);
});
download.body.pipe(res);
void pipeToResponse(download.body, res);
};
/** GET|POST /df — user storage allowance. */
@@ -1986,7 +1997,10 @@ export class LegacyFSController extends PuterController {
const subjectRef = body.subject;
const appRef = body.app;
const mode = (getString(body, 'mode') ?? 'read') as
'see' | 'list' | 'read' | 'write';
| 'see'
| 'list'
| 'read'
| 'write';
if (!subjectRef || !appRef)
throw new HttpError(400, '`subject` and `app` are required', {
legacyCode: 'bad_request',
@@ -2092,10 +2106,7 @@ export class LegacyFSController extends PuterController {
);
res.status(range ? 206 : 200);
download.body.on('error', (err) => {
res.destroy(err);
});
download.body.pipe(res);
void pipeToResponse(download.body, res);
};
async #emitGuiEvent(
@@ -17,13 +17,16 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { describe, expect, it } from 'vitest';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import type { PuterServer } from '../../server.js';
import { createTestUser, setupTestServer } from '../../testUtil.js';
import {
asRecord,
fsEntryMimeType,
getBoolean,
getString,
loadLegacyAssociatedApps,
resolveReadRange,
signEntry,
signEntryThumbnail,
signingConfigFromAppConfig,
@@ -360,7 +363,7 @@ describe('toLegacyEntry', () => {
it('probes for children to set is_empty on a directory', async () => {
const fsEntryStore = {
listChildren: async () => [{ uuid: 'child' }],
hasChildrenByParentUid: async () => true,
} as never;
const shaped = await toLegacyEntry(
undefined,
@@ -373,7 +376,7 @@ describe('toLegacyEntry', () => {
it('treats a failed child probe as non-empty rather than throwing', async () => {
const fsEntryStore = {
listChildren: async () => {
hasChildrenByParentUid: async () => {
throw new Error('db down');
},
} as never;
@@ -385,6 +388,61 @@ describe('toLegacyEntry', () => {
expect(shaped.is_empty).toBe(false);
});
it('leaves is_empty unset for a directory with no store to ask', async () => {
const shaped = await toLegacyEntry(
undefined,
baseEntry({ isDir: true }),
);
expect(shaped).not.toHaveProperty('is_empty');
});
it('runs the child probe, owner and thumbnail lookups together, keeping key order', async () => {
const started: string[] = [];
let release!: () => void;
const gate = new Promise<void>((resolve) => {
release = resolve;
});
const fsEntryStore = {
hasChildrenByParentUid: async () => {
started.push('children');
await gate;
return false;
},
} as never;
const userStore = {
getById: async () => {
started.push('owner');
await gate;
return { username: 'alice' };
},
};
const eventClient = {
emitAndWait: async (_key: string, data: { thumbnail: string }) => {
started.push('thumbnail');
await gate;
data.thumbnail = 'https://signed.example/t.png';
},
} as never;
const pending = toLegacyEntry(
eventClient,
baseEntry({ isDir: true, thumbnail: 's3://bucket/t.png' }),
{ fsEntryStore, userStore },
);
await vi.waitFor(() => expect(started).toHaveLength(3));
release();
const shaped = await pending;
expect(shaped).toMatchObject({
is_empty: true,
owner: { username: 'alice' },
thumbnail: 'https://signed.example/t.png',
});
const keys = Object.keys(shaped);
expect(keys.slice(-2)).toEqual(['is_empty', 'owner']);
expect(keys.indexOf('thumbnail')).toBe(keys.indexOf('is_public') + 1);
});
it('hydrates the owner as a username-only object', async () => {
const userStore = {
getById: async () => ({ username: 'alice', email: '[email protected]' }),
@@ -522,3 +580,128 @@ describe('signEntry', () => {
expect(inSeconds(signed.expires)).toBeLessThanOrEqual(30);
});
});
describe('toLegacyEntry is_empty against stored children', () => {
let server: PuterServer;
beforeAll(async () => {
server = await setupTestServer();
}, 60_000);
afterAll(async () => {
await server?.shutdown();
});
it('reports an empty directory as empty, and one with a child as not', async () => {
const username = `lfh${Math.random().toString(36).slice(2, 10)}`;
await createTestUser(server, { username, password: 'lfh-password' });
const fsEntryStore = server.stores.fsEntry;
const dir = await fsEntryStore.createNonFileEntry({
parent: (await fsEntryStore.getEntryByPath(
`/${username}/Documents`,
))!,
name: 'maybe-empty',
kind: 'directory',
});
const shape = async () =>
toLegacyEntry(
undefined,
(await fsEntryStore.getEntryByUuid(dir.uuid))!,
{ fsEntryStore },
);
expect((await shape()).is_empty).toBe(true);
await fsEntryStore.createNonFileEntry({
parent: dir,
name: 'child',
kind: 'directory',
});
expect((await shape()).is_empty).toBe(false);
});
});
describe('resolveReadRange', () => {
it('returns the explicit headersRange when provided', () => {
expect(
resolveReadRange('bytes=10-20', { offset: 0, byte_count: 5 }),
).toBe('bytes=10-20');
});
it('returns undefined when neither headersRange nor range query params are present', () => {
expect(resolveReadRange(undefined, {})).toBeUndefined();
expect(resolveReadRange('', {})).toBeUndefined();
});
it('derives a byte range from offset and byte_count', () => {
expect(resolveReadRange(undefined, { offset: 0, byte_count: 5 })).toBe(
'bytes=0-4',
);
expect(
resolveReadRange(undefined, { offset: 10, byte_count: 20 }),
).toBe('bytes=10-29');
expect(
resolveReadRange(undefined, { offset: '4', byte_count: '6' }),
).toBe('bytes=4-9');
});
it('derives an open range when only offset is provided', () => {
expect(resolveReadRange(undefined, { offset: 10 })).toBe('bytes=10-');
expect(resolveReadRange(undefined, { offset: '0' })).toBe('bytes=0-');
});
it('derives a prefix range when only byte_count is provided', () => {
expect(resolveReadRange(undefined, { byte_count: 15 })).toBe(
'bytes=0-14',
);
expect(resolveReadRange(undefined, { byte_count: '1' })).toBe(
'bytes=0-0',
);
});
it('throws 400 on invalid or negative offset', () => {
expect(() =>
resolveReadRange(undefined, { offset: -1, byte_count: 10 }),
).toThrow('Invalid offset');
expect(() =>
resolveReadRange(undefined, { offset: 1.5, byte_count: 10 }),
).toThrow('Invalid offset');
expect(() =>
resolveReadRange(undefined, { offset: 'abc', byte_count: 10 }),
).toThrow('Invalid offset');
});
it('throws 400 on non-decimal or unsafe numeric strings', () => {
for (const offset of [
'1e3',
'0x10',
' 4',
'1e21',
'9007199254740993',
]) {
expect(() => resolveReadRange(undefined, { offset })).toThrow(
'Invalid offset',
);
}
expect(() =>
resolveReadRange(undefined, {
offset: String(Number.MAX_SAFE_INTEGER),
byte_count: '3',
}),
).toThrow('Invalid byte_count');
});
it('throws 400 on non-positive or invalid byte_count', () => {
expect(() =>
resolveReadRange(undefined, { offset: 0, byte_count: 0 }),
).toThrow('Invalid byte_count');
expect(() =>
resolveReadRange(undefined, { offset: 0, byte_count: -5 }),
).toThrow('Invalid byte_count');
expect(() =>
resolveReadRange(undefined, { offset: 0, byte_count: 2.5 }),
).toThrow('Invalid byte_count');
expect(() =>
resolveReadRange(undefined, { offset: 0, byte_count: 'xyz' }),
).toThrow('Invalid byte_count');
});
});
+93 -34
View File
@@ -28,6 +28,7 @@ import {
type ACLService,
type AclMode,
type AclRefusalOptions,
type ResourceDescriptor,
} from '../../services/acl/ACLService.js';
import type { Actor } from '../../core/actor.js';
import { Context } from '../../core/context.js';
@@ -98,6 +99,57 @@ export function getBoolean(
return undefined;
}
// Non-negative safe integer from a number or a plain digit string; anything
// else (exponents, hex, whitespace) is rejected.
function parseByteParam(raw: unknown, name: string, min: number): number {
const value =
typeof raw === 'number'
? raw
: typeof raw === 'string' && /^\d+$/.test(raw)
? Number(raw)
: NaN;
if (!Number.isSafeInteger(value) || value < min) {
throw new HttpError(400, `Invalid ${name}`, {
legacyCode: 'bad_request',
});
}
return value;
}
/**
* Resolves an HTTP Range header string from explicit `headersRange` or fallback
* `offset` / `byte_count` query parameters.
*/
export function resolveReadRange(
headersRange: unknown,
query: Record<string, unknown>,
): string | undefined {
if (typeof headersRange === 'string' && headersRange.length > 0) {
return headersRange;
}
const isSet = (v: unknown) => v !== undefined && v !== null && v !== '';
const offset = isSet(query.offset)
? parseByteParam(query.offset, 'offset', 0)
: undefined;
const byteCount = isSet(query.byte_count)
? parseByteParam(query.byte_count, 'byte_count', 1)
: undefined;
if (byteCount !== undefined) {
const last = (offset ?? 0) + byteCount - 1;
if (!Number.isSafeInteger(last)) {
throw new HttpError(400, 'Invalid byte_count', {
legacyCode: 'bad_request',
});
}
return `bytes=${offset ?? 0}-${last}`;
}
if (offset !== undefined) {
return `bytes=${offset}-`;
}
return undefined;
}
// Accepts either `{ path }` or `{ uid }` or `{ id }` or `{ parent, name }`
// from a legacy body field. Returns a resolved entry (throwing 404 if not
// found or 400 if no usable ref is present).
@@ -129,7 +181,7 @@ const MAX_SELECTOR_DEPTH = 4;
* Resolve `ref`, reporting any miss as the same 404 an unreadable entry gets,
* so a lookup never echoes a resolved path or tells missing from hidden.
*/
async function resolveOrNotFound(
export async function resolveOrNotFound(
fsEntryStore: FSEntryStore,
ref: NodeRef,
): Promise<FSEntry> {
@@ -229,7 +281,7 @@ export async function assertAccess(
aclService: ACLService,
_fsService: FSService,
actor: Actor,
path: string,
path: string | ResourceDescriptor,
mode: AclMode,
): Promise<void> {
await aclService.assertFsAccess(actor, path, mode, LEGACY_REFUSAL);
@@ -474,39 +526,46 @@ export async function toLegacyEntry(
...(opts.isShared === undefined ? {} : { is_shared: opts.isShared }),
};
// `is_empty` — only meaningful for directories. Single-row probe so we
// don't pay for listing every child.
if (entry.isDir && opts.fsEntryStore) {
try {
const children = await opts.fsEntryStore.listChildren(entry.uuid, {
limit: 1,
});
response.is_empty = children.length === 0;
} catch {
response.is_empty = false;
}
} else if (!entry.isDir) {
response.is_empty = false;
}
const { fsEntryStore, userStore } = opts;
const [isEmpty, ownerUsername, thumbnail] = await Promise.all([
// `is_empty` — only meaningful for directories.
entry.isDir && fsEntryStore
? (async () => {
try {
return !(await fsEntryStore.hasChildrenByParentUid(
entry.uuid,
));
} catch {
return false;
}
})()
: undefined,
// `owner` — username-only. Matches the legacy safe-entry contract.
userStore
? (async () => {
try {
const owner = await userStore.getById(entry.userId);
return owner && typeof owner.username === 'string'
? owner.username
: undefined;
} catch {
return undefined; // best-effort
}
})()
: undefined,
// Let the thumbnail extension swap an s3:// key for a signed URL.
signEntryThumbnail(
eventClient,
entry.uuid,
response.thumbnail as string | null,
),
]);
// `owner` — username-only. Matches the legacy safe-entry contract.
if (opts.userStore) {
try {
const owner = await opts.userStore.getById(entry.userId);
if (owner && typeof owner.username === 'string') {
response.owner = { username: owner.username };
}
} catch {
/* best-effort */
}
}
// Let the thumbnail extension swap an s3:// key for a signed URL.
response.thumbnail = await signEntryThumbnail(
eventClient,
entry.uuid,
response.thumbnail as string | null,
);
if (isEmpty !== undefined) response.is_empty = isEmpty;
else if (!entry.isDir) response.is_empty = false;
if (ownerUsername !== undefined)
response.owner = { username: ownerUsername };
response.thumbnail = thumbnail;
return response;
}
@@ -0,0 +1,354 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import {
createTestUser,
setupPuterTestEnv,
type PuterTestEnv,
} from '../../testUtil.js';
// A scoped access token writes through the `/fs` routes wherever its grant
// reaches — the reach it already has over WebDAV — and nowhere else. The ACL
// each handler runs intersects the token's grant with its issuer's, so these
// check both edges of that, plus the routes that stay closed to tokens.
describe('scoped access tokens on the /fs write routes', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 180_000);
afterAll(async () => {
await env?.shutdown();
});
const post = async (route: string, token: string, body: unknown) => {
const res = await fetch(new URL(route, env.apiOrigin), {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${token}`,
},
body: JSON.stringify(body),
});
const text = await res.text();
let json: Record<string, unknown> | null = null;
try {
json = text ? (JSON.parse(text) as Record<string, unknown>) : null;
} catch {
// Some refusals are plain text.
}
return { status: res.status, json, text };
};
/** A user, a token granted `mode` on their Documents, and some fixtures. */
const setup = async (mode: 'read' | 'write') => {
const user = await createTestUser(env.server, {
username: `stw${Math.random().toString(36).slice(2, 9)}`,
password: 'puter-test-user-password',
});
const home = `/${user.username}`;
const documents = await post('/stat', user.token, {
path: `${home}/Documents`,
});
await post('/touch', user.token, { path: `${home}/Desktop/kept.txt` });
await post('/touch', user.token, { path: `${home}/Documents/in.txt` });
const minted = await post('/auth/create-access-token', user.token, {
permissions: [`fs:${documents.json!.uid}:${mode}`],
});
expect(minted.status).toBe(200);
return {
user,
home,
token: minted.json!.token as string,
exists: async (path: string) =>
(await post('/stat', user.token, { path })).status === 200,
};
};
it('writes inside its grant on the legacy routes', async () => {
const { home, token, exists } = await setup('write');
const docs = `${home}/Documents`;
expect(
(await post('/mkdir', token, { path: `${docs}/d` })).status,
).toBe(200);
expect(
(await post('/touch', token, { path: `${docs}/d/t.txt` })).status,
).toBe(200);
const t = await post('/stat', token, { path: `${docs}/d/t.txt` });
expect(
(
await post('/rename', token, {
uid: t.json!.uid,
new_name: 'r.txt',
})
).status,
).toBe(200);
expect(
(
await post('/copy', token, {
source: `${docs}/d/r.txt`,
destination: docs,
})
).status,
).toBe(200);
expect(
(
await post('/move', token, {
source: `${docs}/r.txt`,
destination: `${docs}/d`,
new_name: 'm.txt',
})
).status,
).toBe(200);
expect(
(await post('/delete', token, { paths: [`${docs}/d/m.txt`] }))
.status,
).toBe(200);
const batch = await post('/batch', token, {
operations: [
{ op: 'mkdir', path: `${docs}/b` },
{ op: 'delete', path: `${docs}/d/r.txt` },
],
});
expect(batch.status, batch.text).toBe(200);
expect(await exists(`${docs}/b`)).toBe(true);
expect(await exists(`${docs}/d/r.txt`)).toBe(false);
expect(await exists(`${docs}/d/m.txt`)).toBe(false);
}, 180_000);
it('writes inside its grant on the v2 routes', async () => {
const { home, token, exists } = await setup('write');
const docs = `${home}/Documents`;
expect(
(await post('/fs/mkdir', token, { path: `${docs}/v` })).status,
).toBe(200);
expect(
(await post('/fs/touch', token, { path: `${docs}/v/t.txt` }))
.status,
).toBe(200);
const written = await post('/fs/write', token, {
fileMetadata: {
path: `${docs}/v/w.txt`,
size: 3,
contentType: 'text/plain',
},
fileContent: 'abc',
});
expect(written.status, written.text).toBe(200);
expect(
(
await post('/fs/rename', token, {
path: `${docs}/v/t.txt`,
new_name: 'r.txt',
})
).status,
).toBe(200);
expect(
(
await post('/fs/copy', token, {
source: `${docs}/v/r.txt`,
destination: docs,
})
).status,
).toBe(200);
expect(
(
await post('/fs/move', token, {
source: `${docs}/r.txt`,
destination: `${docs}/v`,
new_name: 'm.txt',
})
).status,
).toBe(200);
expect(
(
await post('/fs/mkshortcut', token, {
parent: docs,
target: `${docs}/v/w.txt`,
name: 'w-link',
})
).status,
).toBe(200);
expect(
(await post('/fs/delete', token, { path: `${docs}/v/m.txt` }))
.status,
).toBe(200);
const started = await post('/fs/startWrite', token, {
fileMetadata: { path: `${docs}/v/up.bin`, size: 4 },
});
expect(started.status, started.text).toBe(200);
expect(
(
await post('/fs/abortWrite', token, {
uploadId: started.json!.sessionId,
})
).status,
).toBe(200);
expect(await exists(`${docs}/v/w.txt`)).toBe(true);
expect(await exists(`${docs}/w-link`)).toBe(true);
expect(await exists(`${docs}/v/m.txt`)).toBe(false);
}, 180_000);
it('is refused everywhere outside its grant', async () => {
const { home, token, exists } = await setup('write');
const docs = `${home}/Documents`;
const desktop = `${home}/Desktop`;
const attempts = [
['/mkdir', { path: `${desktop}/x` }],
['/touch', { path: `${desktop}/x.txt` }],
['/move', { source: `${docs}/in.txt`, destination: desktop }],
['/copy', { source: `${desktop}/kept.txt`, destination: docs }],
['/delete', { paths: [`${desktop}/kept.txt`] }],
['/delete', { paths: [home] }],
['/fs/mkdir', { path: `${desktop}/x` }],
[
'/fs/write',
{
fileMetadata: {
path: `${desktop}/kept.txt`,
size: 1,
overwrite: true,
},
fileContent: 'x',
},
],
['/fs/move', { source: `${docs}/in.txt`, destination: desktop }],
['/fs/copy', { source: `${desktop}/kept.txt`, destination: docs }],
['/fs/rename', { path: `${desktop}/kept.txt`, new_name: 'y.txt' }],
['/fs/delete', { path: `${desktop}/kept.txt` }],
['/fs/delete', { path: home }],
] as const;
for (const [route, body] of attempts) {
const res = await post(route, token, body);
expect(
[403, 404],
`${route} ${JSON.stringify(body)} → ${res.status}`,
).toContain(res.status);
}
const batch = await post('/batch', token, {
operations: [{ op: 'mkdir', path: `${desktop}/b` }],
});
expect(batch.status).not.toBe(200);
expect(await exists(`${desktop}/kept.txt`)).toBe(true);
expect(await exists(`${desktop}/x`)).toBe(false);
expect(await exists(`${desktop}/b`)).toBe(false);
expect(await exists(`${docs}/in.txt`)).toBe(true);
expect(await exists(`${docs}/kept.txt`)).toBe(false);
}, 180_000);
it('refuses every write to a read-only token', async () => {
const { home, token, exists } = await setup('read');
const docs = `${home}/Documents`;
const attempts = [
['/mkdir', { path: `${docs}/x` }],
['/touch', { path: `${docs}/x.txt` }],
['/delete', { paths: [`${docs}/in.txt`] }],
['/fs/mkdir', { path: `${docs}/x` }],
[
'/fs/write',
{
fileMetadata: { path: `${docs}/x.txt`, size: 1 },
fileContent: 'x',
},
],
[
'/fs/startWrite',
{ fileMetadata: { path: `${docs}/x.bin`, size: 1 } },
],
['/fs/rename', { path: `${docs}/in.txt`, new_name: 'y.txt' }],
['/fs/delete', { path: `${docs}/in.txt` }],
] as const;
for (const [route, body] of attempts) {
const res = await post(route, token, body);
expect(res.status, `${route} → ${res.status}`).toBe(403);
}
expect(await exists(`${docs}/in.txt`)).toBe(true);
expect(await exists(`${docs}/x`)).toBe(false);
}, 180_000);
// An upload session belongs to the account, not to the credential that
// opened it, so each later step re-checks the target against the caller.
it('cannot touch an upload the session opened outside its grant', async () => {
const { user, home, token } = await setup('write');
const started = await post('/fs/startWrite', user.token, {
fileMetadata: { path: `${home}/Desktop/up.bin`, size: 4 },
});
expect(started.status).toBe(200);
const uploadId = started.json!.sessionId as string;
for (const [route, body] of [
['/fs/abortWrite', { uploadId }],
['/fs/completeWrite', { uploadId }],
['/fs/signMultipartParts', { uploadId, partNumbers: [1] }],
] as const) {
const res = await post(route, token, body);
expect(
[403, 404],
`${route} → ${res.status} ${res.text}`,
).toContain(res.status);
}
// Still open: the session's own abort finds it to abort.
const sessions = await env.server.services.fs.getUploadSessions(
(await env.server.stores.user.getByUsername(user.username))!.id,
[uploadId],
);
expect(sessions[0]?.status).not.toBe('aborted');
expect(
(await post('/fs/abortWrite', user.token, { uploadId })).status,
).toBe(200);
}, 180_000);
it('aborting an unknown upload stays a no-op', async () => {
const { token } = await setup('write');
const res = await post('/fs/abortWrite', token, {
uploadId: 'no-such-upload',
});
expect(res.status).toBe(200);
}, 180_000);
it('keeps the routes that reach past one entry closed', async () => {
const { home, token } = await setup('write');
const docs = `${home}/Documents`;
for (const [route, body] of [
['/sign', { items: [{ path: `${docs}/in.txt`, action: 'read' }] }],
['/search', { text: 'in' }],
['/fs/search', { query: 'in' }],
['/df', {}],
['/readdir-subdomains', {}],
] as const) {
const res = await post(route, token, body);
expect(res.status, `${route} → ${res.status}`).toBe(403);
expect(res.json?.message ?? res.text).toMatch(
/Access tokens are not allowed/,
);
}
}, 180_000);
});
@@ -194,4 +194,87 @@ describe('writes inside a folder shared with write', () => {
(moved.json!.moved as { path: string }).path,
).toBe(`${s.maskedDir}/renamed.txt`);
}, 180_000);
// The thumbnail update is scoped to the writer's own rows, so a file the
// owner ends up owning keeps none, while the writer's own file in the same
// batch gets its thumbnail.
it('batchWrite stores thumbnails on the writer’s own files only', async () => {
const s = await shareAFolder();
const png =
'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==';
const ownPath = `/${s.recipient.username}/Documents/own.png`;
const written = await post('/fs/batchWrite', s.recipient.token, [
{
fileMetadata: { path: `${s.maskedDir}/shared.png`, size: 1 },
fileContent: 'a',
thumbnailData: png,
},
{
fileMetadata: { path: ownPath, size: 1 },
fileContent: 'b',
thumbnailData: png,
},
]);
expect(written.status).toBe(200);
const stored = async (path: string) =>
env.server.stores.fsEntry.getEntryByUuidFromPrimary(
(await env.server.stores.fsEntry.getEntryByPath(path))!.uuid,
);
expect((await stored(`${s.realDir}/shared.png`))?.thumbnail).toBeNull();
expect((await stored(ownPath))?.thumbnail).toMatch(/^s3:\/\//);
}, 180_000);
// The recipient only knows the masked path, so refusals name entries the
// way the recipient addressed them.
it('keeps the owner’s real path out of error messages', async () => {
const s = await shareAFolder();
await post('/touch', s.owner.token, {
path: `${s.realDir}/other.txt`,
});
const cases = [
{
res: await post('/fs/stat', s.recipient.token, {
path: `${s.maskedDir}/missing.txt`,
}),
status: 404,
message: 'Subject does not exist',
},
{
res: await post('/fs/mkdir', s.recipient.token, {
path: `${s.maskedDir}/note.txt`,
}),
status: 409,
message: 'An entry named `note.txt` already exists',
},
{
res: await post('/fs/mkdir', s.recipient.token, {
path: `${s.maskedDir}/missing/child`,
}),
status: 404,
message: 'Parent path does not exist: missing',
},
{
res: await post('/fs/rename', s.recipient.token, {
uid: s.noteUid,
new_name: 'other.txt',
}),
status: 409,
message: 'An entry named `other.txt` already exists',
},
{
res: await post('/mkdir', s.recipient.token, {
path: `${s.maskedDir}/note.txt`,
}),
status: 409,
message: 'An entry named `note.txt` already exists',
},
];
for (const { res, status, message } of cases) {
expect(res.status).toBe(status);
expect(res.json!.message).toBe(message);
expect(JSON.stringify(res.json)).not.toContain(s.realDir);
}
}, 180_000);
});
-2
View File
@@ -32,7 +32,6 @@ import { NotificationController } from './notification/NotificationController.js
import { OIDCController } from './oidc/OIDCController.js';
import { PuterAIController } from './puterai/PuterAIController.js';
import { ShareController } from './share/ShareController.js';
import { TeamController } from './team/TeamController.js';
import { StaticAssetsController } from './static/StaticAssetsController.js';
import { StaticPagesController } from './static/StaticPagesController.js';
import { SystemController } from './system/SystemController.js';
@@ -60,7 +59,6 @@ export const puterControllers = {
notification: NotificationController,
events: EventsController,
share: ShareController,
team: TeamController,
webdav: WebDAVController,
oidc: OIDCController,
wisp: WispController,
@@ -17,6 +17,7 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { installSeatStub } from '../../testFixtures/seatStub.js';
import type { Request, RequestHandler, Response } from 'express';
import {
afterAll,
@@ -233,20 +234,14 @@ const callRoute = async (
);
};
const oidc = (): OIDCService =>
server.services.oidc as unknown as OIDCService;
const oidc = (): OIDCService => server.services.oidc as unknown as OIDCService;
// ── GET /auth/oidc/providers ────────────────────────────────────────
describe('OIDCController GET /auth/oidc/providers', () => {
it('lists every provider whose config validates', async () => {
const { res, captured } = makeRes();
await callRoute(
'get',
'/auth/oidc/providers',
makeReq({}),
res,
);
await callRoute('get', '/auth/oidc/providers', makeReq({}), res);
// We configured a single `custom` provider; the live service
// walks the config and returns its id.
expect(captured.body).toEqual({ providers: ['custom'] });
@@ -417,9 +412,9 @@ describe('OIDCController GET /auth/oidc/:provider/start', () => {
}),
res,
);
const state = new URL(
captured.redirectUrl ?? '',
).searchParams.get('state');
const state = new URL(captured.redirectUrl ?? '').searchParams.get(
'state',
);
const decoded = oidc().verifyState(state!);
expect(String(decoded?.redirect_uri)).toBe(TEST_ORIGIN);
}
@@ -624,10 +619,7 @@ describe('OIDCController login callback', () => {
// The real OIDCService linked the new user — verify via the
// live store.
const linkedUser = await oidc().findUserByProviderSub(
'custom',
sub,
);
const linkedUser = await oidc().findUserByProviderSub('custom', sub);
expect(linkedUser).not.toBeNull();
expect(linkedUser?.email).toBe(email);
});
@@ -791,14 +783,12 @@ describe('OIDCController login callback', () => {
// Seed by running the linked-sub path first via createUserFromOIDC.
const sub = `sub-${Math.random().toString(36).slice(2, 8)}`;
const email = `oidc-${Math.random().toString(36).slice(2, 8)}@test.local`;
const created = await runWithContext(
{ req: makeReq({}) },
() =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
const created = await runWithContext({ req: makeReq({}) }, () =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
);
expect(created.success).toBe(true);
const firstUserId = created.user!.id;
@@ -838,14 +828,12 @@ describe('OIDCController login callback', () => {
it('redirects with auth_error when the user is suspended', async () => {
const sub = `sub-${Math.random().toString(36).slice(2, 8)}`;
const email = `sus-${Math.random().toString(36).slice(2, 8)}@test.local`;
const created = await runWithContext(
{ req: makeReq({}) },
() =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
const created = await runWithContext({ req: makeReq({}) }, () =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
);
expect(created.success).toBe(true);
// Flip the suspended bit on the existing row.
@@ -980,14 +968,12 @@ describe('OIDCController login callback', () => {
it('keeps an /app/<name> landing in the error redirect (suspended user)', async () => {
const sub = `sub-${Math.random().toString(36).slice(2, 8)}`;
const email = `sus-app-${Math.random().toString(36).slice(2, 8)}@test.local`;
const created = await runWithContext(
{ req: makeReq({}) },
() =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
const created = await runWithContext({ req: makeReq({}) }, () =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
);
expect(created.success).toBe(true);
await server.stores.user.update(created.user!.id, { suspended: 1 });
@@ -1075,14 +1061,12 @@ describe('OIDCController login callback', () => {
it('uses popup-style error URL (msg_id + opener_origin) when the popup-state user is suspended', async () => {
const sub = `sub-${Math.random().toString(36).slice(2, 8)}`;
const email = `pop-sus-${Math.random().toString(36).slice(2, 8)}@test.local`;
const created = await runWithContext(
{ req: makeReq({}) },
() =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
const created = await runWithContext({ req: makeReq({}) }, () =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
);
expect(created.success).toBe(true);
await server.stores.user.update(created.user!.id, { suspended: 1 });
@@ -1232,13 +1216,7 @@ describe('OIDCController login callback', () => {
email,
free_storage: 100 * 1024 * 1024,
});
const team = await server.stores.team.create({
ownerUserId: owner.id,
name: 'Acme',
});
await server.stores.team.addMember(team.uid, seat.id, {
orgOwned: true,
});
installSeatStub(server as never, [seat.id]);
// Provisioning issues a temporary password once the seat exists.
await server.stores.user.update(seat.id, { password: 'hashed' });
return seat;
@@ -1334,14 +1312,12 @@ describe('OIDCController signup veto (abuse harness)', () => {
const email = `veto-${Math.random().toString(36).slice(2, 8)}@test.local`;
vetoWithTrail(email, 'trail-svc-1');
const created = await runWithContext(
{ req: makeReq({}) },
() =>
oidc().createUserFromOIDC('custom', {
sub: `sub-${Math.random().toString(36).slice(2, 8)}`,
email,
email_verified: true,
}),
const created = await runWithContext({ req: makeReq({}) }, () =>
oidc().createUserFromOIDC('custom', {
sub: `sub-${Math.random().toString(36).slice(2, 8)}`,
email,
email_verified: true,
}),
);
expect(created.success).toBe(false);
expect(created.code).toBe('signup_blocked');
@@ -1676,14 +1652,12 @@ describe('OIDCController signup callback', () => {
// resolves to `linked-sub` rather than creating a new user.
const sub = `sub-${Math.random().toString(36).slice(2, 8)}`;
const email = `existing-${Math.random().toString(36).slice(2, 8)}@test.local`;
const created = await runWithContext(
{ req: makeReq({}) },
() =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
const created = await runWithContext({ req: makeReq({}) }, () =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
);
expect(created.success).toBe(true);
@@ -1755,14 +1729,12 @@ describe('OIDCController signup callback', () => {
it('redirects suspended users to action=signup with message=account_suspended', async () => {
const sub = `sub-${Math.random().toString(36).slice(2, 8)}`;
const email = `sus-${Math.random().toString(36).slice(2, 8)}@test.local`;
const created = await runWithContext(
{ req: makeReq({}) },
() =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
const created = await runWithContext({ req: makeReq({}) }, () =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
);
expect(created.success).toBe(true);
await server.stores.user.update(created.user!.id, { suspended: 1 });
@@ -1899,14 +1871,12 @@ describe('OIDCController revalidate callback', () => {
// The linked account exists, but state.user_uuid points at someone else.
const sub = `sub-${Math.random().toString(36).slice(2, 8)}`;
const email = `reval-${Math.random().toString(36).slice(2, 8)}@test.local`;
const created = await runWithContext(
{ req: makeReq({}) },
() =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
const created = await runWithContext({ req: makeReq({}) }, () =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
);
expect(created.success).toBe(true);
@@ -1938,14 +1908,12 @@ describe('OIDCController revalidate callback', () => {
it('happy path: sets the revalidation cookie and redirects to redirect-done', async () => {
const sub = `sub-${Math.random().toString(36).slice(2, 8)}`;
const email = `reval-${Math.random().toString(36).slice(2, 8)}@test.local`;
const created = await runWithContext(
{ req: makeReq({}) },
() =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
const created = await runWithContext({ req: makeReq({}) }, () =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
);
expect(created.success).toBe(true);
const userUuid = created.user!.uuid;
@@ -1992,14 +1960,12 @@ describe('OIDCController revalidate callback', () => {
it('falls back to /auth/revalidate-done when state.redirect_uri is cross-origin', async () => {
const sub = `sub-${Math.random().toString(36).slice(2, 8)}`;
const email = `reval-${Math.random().toString(36).slice(2, 8)}@test.local`;
const created = await runWithContext(
{ req: makeReq({}) },
() =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
const created = await runWithContext({ req: makeReq({}) }, () =>
oidc().createUserFromOIDC('custom', {
sub,
email,
email_verified: true,
}),
);
expect(created.success).toBe(true);
const userUuid = created.user!.uuid;
@@ -2038,12 +2004,7 @@ describe('OIDCController revalidate callback', () => {
describe('OIDCController GET /auth/revalidate-done', () => {
it('renders the postMessage HTML body with the configured origin', async () => {
const { res, captured } = makeRes();
await callRoute(
'get',
'/auth/revalidate-done',
makeReq({}),
res,
);
await callRoute('get', '/auth/revalidate-done', makeReq({}), res);
expect(captured.contentType).toContain('text/html');
const body = String(captured.body);
expect(body).toContain('puter-revalidate-done');
@@ -2057,9 +2018,9 @@ describe('OIDCController GET /auth/revalidate-done', () => {
/**
* The proof exists because the popup return leg states two things the popup
* cannot check — the opener's origin and that a login completed — and a URL
* built from a verified `state` is byte-identical to one anybody can type.
* The opener's origin picks the app a token is minted for, so it has to be
* attested rather than read.
* built from a verified `state` is byte-identical to one anybody can type. The
* opener's origin picks the app a token is minted for, so it has to be attested
* rather than read.
*/
describe('OIDCController POST /auth/oidc/verify-popup-return', () => {
const NONCE = 'binding-nonce';
+72 -63
View File
@@ -24,6 +24,12 @@ import { HttpError } from '../../core/http/HttpError.js';
import type { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterController } from '../types.js';
import { startWebSession } from '../auth/webSession.js';
import {
completeOidcLoginReturn,
mintOidcPopupReturn,
OIDC_POPUP_RETURN_COOKIE_NAME,
type OidcLoginContinuation,
} from '../auth/oidcLoginReturn.js';
import type { UserRow } from '../../stores/user/UserStore.js';
import { sessionCookieFlags } from '../../util/cookieFlags.js';
import { normalizeBonusCode } from '../../util/signupBonus.js';
@@ -43,11 +49,6 @@ const REVALIDATION_EXPIRY_SEC = 300;
const OIDC_NONCE_COOKIE_NAME = 'puter_oidc_nonce';
const OIDC_NONCE_EXPIRY_SEC = 600;
// Single-use, and binds a popup-return proof to the browser that earned it.
// Expiry mirrors POPUP_RETURN_EXPIRY_SEC in OIDCService.
const OIDC_POPUP_RETURN_COOKIE_NAME = 'puter_oidc_popup_return';
const OIDC_POPUP_RETURN_EXPIRY_SEC = 300;
// The popup branch below hard-codes this return target.
const POPUP_RETURN_ACTION = 'sign-in';
@@ -284,11 +285,6 @@ function buildErrorRedirectUrl(
return `${base}${pagePath}?${params.toString()}`;
}
function appendQueryParam(url: string, key: string, value: string): string {
const sep = url.includes('?') ? '&' : '?';
return `${url}${sep}${encodeURIComponent(key)}=${encodeURIComponent(value)}`;
}
/** Length-safe constant-time string compare (never throws on mismatch). */
function constantTimeEqual(a: string, b: string): boolean {
const ba = Buffer.from(a);
@@ -803,18 +799,9 @@ if (window.opener) {
/** Sign a popup-return proof, bound to this browser and this popup. */
#mintPopupReturn(res: Response, payload: Record<string, unknown>): string {
const nonce = crypto.randomBytes(32).toString('base64url');
res.cookie(OIDC_POPUP_RETURN_COOKIE_NAME, nonce, {
// Redeemed same-origin.
...sessionCookieFlags(this.config, { crossSite: false }),
httpOnly: true,
maxAge: OIDC_POPUP_RETURN_EXPIRY_SEC * 1000,
path: '/',
});
return this.services.oidc.signPopupReturn({
action: POPUP_RETURN_ACTION,
...payload,
nonce,
return mintOidcPopupReturn(res, payload, {
config: this.config,
oidc: this.services.oidc,
});
}
@@ -872,7 +859,7 @@ if (window.opener) {
byEmail !== null &&
!byEmail.email_confirmed &&
userinfo.email_verified === true &&
(await this.services.team.releaseUnconfirmedSeatEmail(
(await this.services.team?.releaseUnconfirmedSeatEmail(
byEmail.id,
));
if (byEmail && !releasedFromSeat) {
@@ -1012,49 +999,71 @@ if (window.opener) {
stateDecoded: Record<string, unknown>,
extraQueryParams?: Record<string, string>,
): Promise<void> {
const origin = (this.config.origin ?? '').replace(/\/$/, '');
const continuation: OidcLoginContinuation = {
userUuid: user.uuid,
redirectUri:
typeof stateDecoded.redirect_uri === 'string'
? stateDecoded.redirect_uri
: origin || '/',
embeddedInPopup: stateDecoded.embedded_in_popup === true,
openerOrigin:
typeof stateDecoded.opener_origin === 'string'
? stateDecoded.opener_origin
: null,
messageId:
stateDecoded.msg_id == null
? null
: String(stateDecoded.msg_id),
switchedToLogin: extraQueryParams?.oidc_switched === 'login',
};
if (user.otp_enabled) {
const challenge = this.services.token.sign(
'otp',
{
purpose: 'otp-login',
user_uid: user.uuid,
jti: crypto.randomUUID(),
oidcContinuation: continuation,
},
{ expiresIn: '5m' },
);
res.set('Cache-Control', 'no-store');
res.set('Referrer-Policy', 'no-referrer');
let retryUrl = `${origin}/action/login`;
if (continuation.embeddedInPopup) {
const retry = new URL(
buildErrorRedirectUrl(
origin,
'login',
'other',
'unauthorized',
stateDecoded,
undefined,
(payload) => this.#mintPopupReturn(res, payload),
),
);
retry.searchParams.delete('auth_error');
retry.searchParams.delete('message');
retryUrl = retry.href;
}
const fragment = new URLSearchParams({
token: challenge,
retryUrl,
});
res.redirect(302, `${origin}/action/oidc-2fa#${fragment}`);
return;
}
await startWebSession(req, res, user, {
config: this.config,
auth: this.services.auth,
});
const origin = (this.config.origin ?? '').replace(/\/$/, '');
let target = (stateDecoded.redirect_uri as string) || origin || '/';
if (!isSameOrigin(target, origin)) {
target = origin || '/';
}
if (stateDecoded.embedded_in_popup) {
target = appendQueryParam(target, 'oidc_login', 'true');
// `opener_origin` and `oidc_login` reach the popup as bare query
// parameters, which say nothing about where they came from: the
// URL a verified state produces is byte-identical to one anybody
// can type. The popup treats the opener's origin as the app
// identity to mint a token for, so it needs the integrity this
// state already carries — re-signed here, at the one point where
// the round trip is known to have actually happened.
//
// `user_uuid` binds the proof to the account that just completed
// OIDC. Without it a proof from one login could be replayed in
// another signed-in browser to skip its account picker; the popup
// only honors `oidc_login` when this matches its current user.
target = appendQueryParam(
target,
'opener_state',
this.#mintPopupReturn(res, {
opener_origin: stateDecoded.opener_origin ?? null,
msg_id: stateDecoded.msg_id ?? null,
oidc_login: true,
user_uuid: user.uuid,
}),
);
}
if (extraQueryParams) {
for (const [k, v] of Object.entries(extraQueryParams)) {
if (v != null) target = appendQueryParam(target, k, v);
}
}
res.redirect(302, target);
res.redirect(
302,
completeOidcLoginReturn(res, user, continuation, {
config: this.config,
oidc: this.services.oidc,
}),
);
}
}
@@ -0,0 +1,323 @@
import { TOTP } from 'otpauth';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import {
setupPuterTestEnv,
createTestUser,
type PuterTestEnv,
} from '../../testUtil.js';
import type { IConfig } from '../../types.js';
import { hashRecoveryCode } from '../../services/auth/OTPUtil.js';
describe('OIDC account second-factor login', () => {
let env: PuterTestEnv;
let claims: { sub: string; email: string; email_verified: boolean };
beforeAll(async () => {
env = await setupPuterTestEnv({
cookie_name: 'puter_auth_token',
oidc: {
providers: {
custom: {
client_id: 'test-client',
client_secret: 'test-secret',
authorization_endpoint: 'https://idp.example/authorize',
token_endpoint: 'https://idp.example/token',
userinfo_endpoint: 'https://idp.example/userinfo',
},
},
},
} as IConfig);
vi.spyOn(
env.server.services.oidc,
'exchangeCodeForTokens',
).mockResolvedValue({ access_token: 'test-access' } as never);
vi.spyOn(env.server.services.oidc, 'getUserInfo').mockImplementation(
async () => claims,
);
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const account = async (enabled = true, linked = true) => {
const username = `oidc2fa-${uuidv4().slice(0, 8)}`;
await createTestUser(env.server, {
username,
password: 'oidc-test-password',
});
const user = (await env.server.stores.user.getByUsername(username))!;
const otp = new TOTP({
issuer: 'puter.com',
label: username,
algorithm: 'SHA1',
digits: 6,
});
claims = {
sub: uuidv4(),
email: `${username}@example.com`,
email_verified: true,
};
await env.server.stores.user.update(user.id, {
email: claims.email,
email_confirmed: true,
otp_enabled: enabled,
otp_secret: otp.secret.base32,
otp_recovery_codes: hashRecoveryCode('ABCDEFG2'),
});
if (linked) {
await env.server.services.oidc.linkProviderToUser(
user.id,
'custom',
claims,
);
}
return { user, otp };
};
const callback = async (
flow = 'login',
extra: Record<string, string> = {},
) => {
const start = new URL(
`/auth/oidc/custom/start?flow=${flow}`,
env.origin,
);
Object.entries(extra).forEach(([key, value]) =>
start.searchParams.set(key, value),
);
const begun = await fetch(start, { redirect: 'manual' });
expect(begun.status).toBe(302);
const state = new URL(begun.headers.get('location')!).searchParams.get(
'state',
)!;
const nonce = begun.headers
.getSetCookie()
.find((value) => value.startsWith('puter_oidc_nonce='))!
.split(';')[0]!;
return fetch(
new URL(
`/auth/oidc/callback/${flow}?code=test-code&state=${encodeURIComponent(state)}`,
env.origin,
),
{ redirect: 'manual', headers: { cookie: nonce } },
);
};
const verify = (token: string, code: string, recovery = false) =>
fetch(
new URL(
recovery ? '/login/recovery-code' : '/login/otp',
env.origin,
),
{
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ token, code }),
},
);
const challenge = (response: Response) => {
expect(response.status).toBe(302);
expect(
response.headers
.getSetCookie()
.some((value) => value.startsWith('puter_auth_token=')),
).toBe(false);
const location = new URL(response.headers.get('location')!);
expect(location.pathname).toBe('/action/oidc-2fa');
const token = new URLSearchParams(location.hash.slice(1)).get('token');
expect(token).toBeTruthy();
return token!;
};
it.each(['login', 'signup'])(
'requires the authenticator before completing existing-account %s',
async (flow) => {
const { otp } = await account();
const token = challenge(
await callback(flow, { return_to: '/dashboard' }),
);
const response = await verify(token, otp.generate());
expect(response.status).toBe(200);
expect(await response.json()).toMatchObject({
proceed: true,
next_step: 'complete',
redirectUrl: `${env.origin}/dashboard${flow === 'signup' ? '?oidc_switched=login' : ''}`,
});
expect(
response.headers
.getSetCookie()
.some((value) => value.startsWith('puter_auth_token=')),
).toBe(true);
expect((await verify(token, otp.generate())).status).toBe(400);
},
);
it('incorrect codes do not start a session and allow a valid recovery-code retry', async () => {
const { otp } = await account();
const token = challenge(await callback());
const wrong = ['000000', '000001', '000002'].find(
(code) => otp.validate({ token: code }) === null,
)!;
const rejected = await verify(token, wrong);
expect(await rejected.json()).toMatchObject({ proceed: false });
expect(rejected.headers.getSetCookie()).toHaveLength(0);
const accepted = await verify(token, 'ABCDEFG2', true);
expect(accepted.status).toBe(200);
expect(await accepted.json()).toMatchObject({
proceed: true,
redirectUrl: env.origin,
});
});
it('preserves an unauthenticated popup retry and attests login only after verification', async () => {
const { otp, user } = await account();
const response = await callback('login', {
embedded_in_popup: 'true',
msg_id: 'message-123',
opener_origin: 'https://app.example',
});
const token = challenge(response);
const fragment = new URLSearchParams(
new URL(response.headers.get('location')!).hash.slice(1),
);
const retry = new URL(fragment.get('retryUrl')!);
expect(retry.origin).toBe(env.origin);
expect(retry.searchParams.get('action')).toBe('login');
expect(retry.searchParams.get('embedded_in_popup')).toBe('true');
expect(retry.searchParams.get('msg_id')).toBe('message-123');
expect(retry.searchParams.has('auth_error')).toBe(false);
const retryNonce = response.headers
.get('set-cookie')!
.match(/puter_oidc_popup_return=([^;, ]+)/)![1];
const retried = await fetch(
new URL('/auth/oidc/verify-popup-return', env.origin),
{
method: 'POST',
headers: {
'content-type': 'application/json',
cookie: `puter_oidc_popup_return=${retryNonce}`,
},
body: JSON.stringify({
opener_state: retry.searchParams.get('opener_state'),
}),
},
);
expect(retried.status).toBe(200);
expect(await retried.json()).toMatchObject({
action: 'login',
oidc_login: false,
user_uuid: null,
msg_id: 'message-123',
opener_origin: 'https://app.example',
});
const accepted = await verify(token, otp.generate());
const data = await accepted.json();
const target = new URL(data.redirectUrl);
expect(target.pathname).toBe('/action/sign-in');
expect(target.searchParams.get('msg_id')).toBe('message-123');
expect(target.searchParams.get('oidc_login')).toBe('true');
const proof = env.server.services.oidc.verifyPopupReturn(
target.searchParams.get('opener_state')!,
);
expect(proof).toMatchObject({
user_uuid: user.uuid,
opener_origin: 'https://app.example',
msg_id: 'message-123',
oidc_login: true,
});
expect(
accepted.headers
.getSetCookie()
.some((value) => value.startsWith('puter_oidc_popup_return=')),
).toBe(true);
});
it('requires the authenticator when verified email first links an existing account', async () => {
const { otp } = await account(true, false);
const token = challenge(await callback());
expect((await verify(token, otp.generate())).status).toBe(200);
});
it('requires the authenticator for an OIDC-only account', async () => {
const { user, otp } = await account();
await env.server.stores.user.update(user.id, { password: null });
const token = challenge(await callback());
const response = await verify(token, otp.generate());
expect(await response.json()).toMatchObject({
proceed: true,
next_step: 'complete',
});
});
it('uses distinct short-lived challenges and refuses a modified token', async () => {
const { otp } = await account();
const first = challenge(await callback());
const second = challenge(await callback());
expect(first).not.toBe(second);
const decoded = env.server.services.token.verify<{
purpose: string;
jti: string;
exp: number;
iat: number;
}>('otp', first);
expect(decoded.purpose).toBe('otp-login');
expect(decoded.jti).toBeTruthy();
expect(decoded.exp - decoded.iat).toBe(300);
const modified = `${first.slice(0, -4)}abcd`;
const rejected = await verify(modified, otp.generate());
expect(rejected.status).toBe(400);
expect(rejected.headers.get('set-cookie')).toBeNull();
});
it('validates continuation account binding before creating any session', async () => {
const { otp } = await account();
const issued = challenge(await callback());
const decoded = env.server.services.token.verify<
Record<string, unknown>
>('otp', issued);
const continuation = decoded.oidcContinuation as Record<
string,
unknown
>;
const mismatched = env.server.services.token.sign('otp', {
...decoded,
jti: uuidv4(),
oidcContinuation: { ...continuation, userUuid: uuidv4() },
});
const rejected = await verify(mismatched, otp.generate());
expect(rejected.status).toBe(400);
expect(rejected.headers.get('set-cookie')).toBeNull();
});
it('clamps a signed continuation redirect to the configured origin', async () => {
const { otp } = await account();
const issued = challenge(await callback());
const decoded = env.server.services.token.verify<
Record<string, unknown>
>('otp', issued);
const continuation = decoded.oidcContinuation as Record<
string,
unknown
>;
const redirected = env.server.services.token.sign('otp', {
...decoded,
jti: uuidv4(),
oidcContinuation: {
...continuation,
redirectUri: 'https://elsewhere.example/',
},
});
const accepted = await verify(redirected, otp.generate());
expect(await accepted.json()).toMatchObject({
proceed: true,
redirectUrl: env.origin,
});
});
it('preserves ordinary OIDC login for accounts without 2FA', async () => {
await account(false);
const response = await callback('login', { return_to: '/desktop' });
expect(response.headers.get('location')).toBe(`${env.origin}/desktop`);
expect(
response.headers.get('set-cookie')?.includes('puter_auth_token='),
).toBe(true);
});
});
@@ -0,0 +1,140 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, Response } from 'express';
import { v4 as uuidv4 } from 'uuid';
import {
afterAll,
afterEach,
beforeAll,
describe,
expect,
it,
vi,
} from 'vitest';
import type { PuterServer } from '../../server.js';
import type { UserRow } from '../../stores/user/UserStore.js';
import { setupTestServer } from '../../testUtil.js';
import type { ProfileController } from './ProfileController.js';
// GET /profile without a username: the caller's own profile, read alongside
// their account row.
describe('ProfileController own profile', () => {
let server: PuterServer;
let controller: ProfileController;
beforeAll(async () => {
server = await setupTestServer();
controller = server.controllers.profile as unknown as ProfileController;
}, 120_000);
afterAll(async () => {
await server?.shutdown();
});
afterEach(() => {
vi.restoreAllMocks();
});
const makeUser = async (): Promise<UserRow> => {
const username = `pc-${Math.random().toString(36).slice(2, 10)}`;
return server.stores.user.create({
username,
uuid: uuidv4(),
password: null,
email: `${username}@test.local`,
requires_email_confirmation: false,
} as Parameters<typeof server.stores.user.create>[0]);
};
const getOwn = async (uuid: string) => {
let body: unknown;
const res = {
json: (value: unknown) => {
body = value;
},
} as unknown as Response;
await controller.get(
{ query: {}, actor: { user: { uuid } } } as unknown as Request,
res,
);
return body;
};
const isProfilePath = (path: unknown) =>
String(path).startsWith('/system/profiles/');
it('starts reading the profile before the account lookup returns', async () => {
const user = await makeUser();
await server.services.profile.updateProfile(user, { bio: 'hi' });
const order: string[] = [];
const getByUuid = server.stores.user.getByUuid.bind(server.stores.user);
vi.spyOn(server.stores.user, 'getByUuid').mockImplementation(
async (...args: Parameters<typeof getByUuid>) => {
await new Promise((resolve) => setTimeout(resolve, 10));
order.push('account');
return getByUuid(...args);
},
);
const getEntryByPath = server.stores.fsEntry.getEntryByPath.bind(
server.stores.fsEntry,
);
vi.spyOn(server.stores.fsEntry, 'getEntryByPath').mockImplementation(
async (...args: Parameters<typeof getEntryByPath>) => {
if (isProfilePath(args[0])) order.push('profile');
return getEntryByPath(...args);
},
);
await expect(getOwn(user.uuid)).resolves.toEqual({
picture: null,
displayName: null,
bio: 'hi',
});
expect(order).toEqual(['profile', 'account']);
});
it('404s a caller with no account, whatever the profile read did', async () => {
vi.spyOn(server.stores.fsEntry, 'getEntryByPath').mockRejectedValue(
new Error('profile read failed'),
);
await expect(getOwn(uuidv4())).rejects.toMatchObject({
statusCode: 404,
});
});
it('surfaces the account lookup failure before the profile read failure', async () => {
vi.spyOn(server.stores.user, 'getByUuid').mockRejectedValue(
new Error('account lookup failed'),
);
vi.spyOn(server.stores.fsEntry, 'getEntryByPath').mockRejectedValue(
new Error('profile read failed'),
);
await expect(getOwn(uuidv4())).rejects.toThrow('account lookup failed');
});
it('surfaces a failed profile read for an existing account', async () => {
const user = await makeUser();
vi.spyOn(server.stores.fsEntry, 'getEntryByPath').mockRejectedValue(
new Error('profile read failed'),
);
await expect(getOwn(user.uuid)).rejects.toThrow('profile read failed');
});
});
@@ -20,7 +20,10 @@
import type { Request, Response } from 'express';
import { Controller, Get, Post } from '../../core/http/decorators.js';
import { HttpError } from '../../core/http/HttpError.js';
import type { ProfileService } from '../../services/profile/ProfileService.js';
import type {
ProfileService,
UserProfile,
} from '../../services/profile/ProfileService.js';
import { PuterController } from '../types.js';
// Shape only; the store decides whether the account exists.
@@ -56,13 +59,21 @@ export class ProfileController extends PuterController {
const username = req.query.username;
let target;
let ownProfile: PromiseSettledResult<UserProfile> | undefined;
if (username === undefined) {
if (!caller?.uuid) {
throw new HttpError(401, 'Sign in, or pass a username', {
legacyCode: 'unauthorized',
});
}
target = await this.stores.user.getByUuid(caller.uuid);
// Read alongside the account; discarded if there's no account.
const [found, profile] = await Promise.allSettled([
this.stores.user.getByUuid(caller.uuid),
service.getProfile({ uuid: caller.uuid }),
]);
if (found.status === 'rejected') throw found.reason;
target = found.value;
ownProfile = profile;
} else {
if (
typeof username !== 'string' ||
@@ -80,6 +91,11 @@ export class ProfileController extends PuterController {
if (!isOwner && !(await service.isPubliclyVisible(target))) {
throw notFound();
}
if (ownProfile && isOwner) {
if (ownProfile.status === 'rejected') throw ownProfile.reason;
res.json(ownProfile.value);
return;
}
res.json(await service.getProfile(target));
}
@@ -30,7 +30,8 @@
*/
import crypto from 'node:crypto';
import { Readable } from 'node:stream';
import { Readable, Writable } from 'node:stream';
import { finished } from 'node:stream/promises';
import type { Request, Response } from 'express';
import {
afterAll,
@@ -1530,25 +1531,34 @@ describe('PuterAIController videoProxy upstream handling', () => {
} as unknown as Response);
const handler = captureGet('/puterai/video/proxy');
const { res, captured } = makeRes();
// The body is piped, so the response has to be a real stream.
const sink: Buffer[] = [];
(res as unknown as Record<string, unknown>).write = (
chunk: Buffer,
) => {
sink.push(Buffer.from(chunk));
return true;
};
(res as unknown as Record<string, unknown>).emit = () => true;
const headers: Record<string, string> = {};
const res = Object.assign(
new Writable({
write(chunk: Buffer, _enc, cb) {
sink.push(Buffer.from(chunk));
cb();
},
}),
{
setHeader: (k: string, v: string) => {
headers[k] = v;
},
},
) as unknown as Response;
await handler(makeReq({ query: signedQuery('vid1') }), res);
await settleStream();
await finished(res as unknown as Writable);
expect(fetchSpy).toHaveBeenCalledWith(
expect.stringContaining(
'vid1:download?alt=media&key=gemini-test-key',
),
{ signal: expect.any(AbortSignal) },
);
expect(captured.headers['Content-Type']).toBe('video/mp4');
expect(headers['Content-Type']).toBe('video/mp4');
expect(Buffer.concat(sink).toString('utf8')).toBe('video-bytes');
});
});
@@ -24,6 +24,7 @@ import { Context } from '../../core/context.js';
import { abortOnDisconnect } from '../../core/http/abortOnDisconnect.js';
import { HttpError } from '../../core/http/HttpError.js';
import { RouteOptions } from '../../core/http/index.js';
import { pipeToResponse } from '../../core/http/pipeToResponse.js';
import { computeNetworkFingerprint } from '../../core/http/middleware/rateLimit.js';
import type { PuterRouter } from '../../core/http/PuterRouter.js';
import {
@@ -337,8 +338,10 @@ export class PuterAIController extends PuterController {
return;
}
// A visitor who leaves stops the download, not just the reply.
const upstream = await fetch(
`${GEMINI_DOWNLOAD_BASE}/${fileId}:download?alt=media&key=${apiKey}`,
{ signal: abortOnDisconnect(res) },
);
if (!upstream.ok) {
res.status(upstream.status).send('Failed to fetch video');
@@ -351,9 +354,12 @@ export class PuterAIController extends PuterController {
res.status(500).send('Empty response body');
return;
}
Readable.fromWeb(
upstream.body as unknown as import('node:stream/web').ReadableStream,
).pipe(res);
void pipeToResponse(
Readable.fromWeb(
upstream.body as unknown as import('node:stream/web').ReadableStream,
),
res,
);
};
#listModels(driverKey: 'aiChat' | 'aiImage' | 'aiVideo') {
@@ -539,7 +545,8 @@ export class PuterAIController extends PuterController {
}
} else if (ev.type === 'usage') {
usageDetails = ev.usageDetails as
UsageDetails | undefined;
| UsageDetails
| undefined;
finishReason = ev.finish_reason as string | undefined;
} else if (ev.type === 'error') {
res.write(
@@ -687,7 +694,8 @@ export class PuterAIController extends PuterController {
sendChunk(ev.text);
} else if (ev.type === 'usage') {
usageDetails = ev.usageDetails as
UsageDetails | undefined;
| UsageDetails
| undefined;
finishReason = ev.finish_reason as string | undefined;
} else if (ev.type === 'error') {
res.write(
@@ -741,7 +749,8 @@ export class PuterAIController extends PuterController {
text: extractTextContent(
(
messageResult.message as
Record<string, unknown> | undefined
| Record<string, unknown>
| undefined
)?.content,
),
index: 0,
@@ -0,0 +1,249 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import crypto from 'node:crypto';
import http from 'node:http';
import {
afterAll,
afterEach,
beforeAll,
describe,
expect,
it,
vi,
} from 'vitest';
import type { PuterServer } from '../../server.js';
import { allocateEphemeralPort, setupTestServer } from '../../testUtil.js';
import type { IConfig } from '../../types.js';
/**
* The video proxy over a real socket: the provider download is the only thing
* stubbed, so what reaches the client and what a hang-up does upstream are the
* server's own behavior.
*/
describe('video proxy over HTTP', () => {
const DOWNLOAD_BASE = 'https://generativelanguage.googleapis.com/';
const SECRET = 'video-proxy-test-secret';
let server: PuterServer;
let port: number;
beforeAll(async () => {
port = await allocateEphemeralPort();
server = await setupTestServer(
{
port,
domain: 'puter.localhost',
origin: `http://puter.localhost:${port}`,
api_base_url: `http://api.puter.localhost:${port}`,
url_signature_secret: SECRET,
providers: {
'gemini-video-generation': { apiKey: 'test-key' },
},
} as unknown as IConfig,
{ listen: true },
);
}, 120_000);
afterAll(async () => {
await server?.shutdown();
});
afterEach(() => {
vi.restoreAllMocks();
});
const signedPath = (fileId: string) => {
const expires = String(Math.floor(Date.now() / 1000) + 60);
const signature = crypto
.createHash('sha256')
.update(`${fileId}/video-proxy/${SECRET}/${expires}`)
.digest('hex');
return `/puterai/video/proxy?${new URLSearchParams({
fileId,
expires,
signature,
provider: 'gemini',
})}`;
};
/** Serves `body` as the provider download and records what upstream sees. */
const stubDownload = (body: ReadableStream<Uint8Array>) => {
const seen: { signal?: AbortSignal } = {};
const realFetch = globalThis.fetch;
vi.spyOn(globalThis, 'fetch').mockImplementation(
async (input, init) => {
if (!String(input).startsWith(DOWNLOAD_BASE)) {
return realFetch(input, init);
}
seen.signal = init?.signal ?? undefined;
return new Response(body, {
status: 200,
headers: { 'content-type': 'video/mp4' },
});
},
);
return seen;
};
const get = (path: string) =>
http.request({
host: '127.0.0.1',
port,
path,
headers: { host: `api.puter.localhost:${port}` },
});
const waitFor = async (check: () => boolean) => {
for (let i = 0; i < 100 && !check(); i++) {
await new Promise((resolve) => setTimeout(resolve, 10));
}
return check();
};
it('streams the provider download to the client byte for byte', async () => {
const parts = ['video', '-', 'bytes'].map((s) =>
new TextEncoder().encode(s),
);
stubDownload(
new ReadableStream({
start(controller) {
for (const part of parts) controller.enqueue(part);
controller.close();
},
}),
);
const { status, contentType, body } = await new Promise<{
status?: number;
contentType?: string;
body: string;
}>((resolve, reject) => {
const req = get(signedPath('vid-ok'));
req.on('response', (res) => {
const chunks: Buffer[] = [];
res.on('data', (chunk: Buffer) => chunks.push(chunk));
res.on('end', () =>
resolve({
status: res.statusCode,
contentType: res.headers['content-type'],
body: Buffer.concat(chunks).toString('utf8'),
}),
);
res.on('error', reject);
});
req.on('error', reject);
req.end();
});
expect(status).toBe(200);
expect(contentType).toBe('video/mp4');
expect(body).toBe('video-bytes');
});
it('streams without a listener-leak warning on the response', async () => {
stubDownload(
new ReadableStream({
start(controller) {
controller.enqueue(new TextEncoder().encode('video'));
controller.close();
},
}),
);
const warnings: string[] = [];
const onWarning = (warning: Error) => warnings.push(warning.name);
process.on('warning', onWarning);
try {
await new Promise<void>((resolve, reject) => {
const req = get(signedPath('vid-listeners'));
req.on('response', (res) => {
res.on('data', () => {});
res.on('end', resolve);
res.on('error', reject);
});
req.on('error', reject);
req.end();
});
await new Promise((resolve) => setTimeout(resolve, 10));
} finally {
process.off('warning', onWarning);
}
expect(warnings).not.toContain('MaxListenersExceededWarning');
});
it('stops the provider download when the client hangs up', async () => {
let cancelled = false;
const seen = stubDownload(
new ReadableStream({
start(controller) {
controller.enqueue(new TextEncoder().encode('first'));
},
// Never finishes on its own: only a cancel ends it.
pull: () => new Promise(() => {}),
cancel() {
cancelled = true;
},
}),
);
await new Promise<void>((resolve, reject) => {
const req = get(signedPath('vid-leave'));
req.on('response', (res) => {
res.once('data', () => {
req.destroy();
resolve();
});
});
req.on('error', (err) => {
if (!req.destroyed) reject(err);
});
req.end();
});
expect(await waitFor(() => cancelled)).toBe(true);
expect(seen.signal?.aborted).toBe(true);
});
it('cuts the reply off when the provider download breaks mid-stream', async () => {
stubDownload(
new ReadableStream({
start(controller) {
controller.enqueue(new TextEncoder().encode('partial'));
setTimeout(
() => controller.error(new Error('upstream broke')),
10,
);
},
}),
);
const outcome = await new Promise<string>((resolve) => {
const req = get(signedPath('vid-broken'));
req.on('response', (res) => {
res.on('data', () => {});
res.on('end', () => resolve('ended'));
res.on('error', () => resolve('aborted'));
res.on('aborted', () => resolve('aborted'));
});
req.on('error', () => resolve('aborted'));
req.end();
});
expect(outcome).toBe('aborted');
});
});
@@ -341,6 +341,61 @@ describe('share endpoints over HTTP', () => {
expect(replayed.items).toEqual([]);
});
it('drops what a folder delegate re-shared once the item leaves the folder', async () => {
const owner = await makeUser();
const delegate = await makeUser();
const holder = await makeUser();
const dir = (await (
await post('/mkdir', owner.token, {
path: `/${owner.username}/Proj`,
})
).json()) as { uid: string };
await post('/touch', owner.token, {
path: `/${owner.username}/Proj/plan.txt`,
});
const file = (await (
await post('/stat', owner.token, {
path: `/${owner.username}/Proj/plan.txt`,
})
).json()) as { uid: string };
await post('/share', owner.token, {
recipients: [delegate.username],
items: [{ uid: dir.uid }],
mode: 'manage',
});
await post('/share', delegate.token, {
recipients: [holder.username],
items: [{ uid: file.uid }],
mode: 'read',
});
const listedTo = async (token: string) =>
(
(await (
await get('/share/shared-by-me', token, {})
).json()) as { items: Array<{ uid_entry: string }> }
).items.map((i) => i.uid_entry);
expect(await listedTo(delegate.token)).toContain(file.uid);
const moved = await post('/move', owner.token, {
source: file.uid,
destination: `/${owner.username}`,
});
expect(moved.status).toBe(200);
// A move doesn't bump readings; this stands in for the scan-cache TTL.
const delegateRow = await env.server.stores.user.getByUsername(
delegate.username,
);
await env.server.stores.permission.bumpCacheGeneration(
`user:${delegateRow!.uuid}`,
);
expect(await listedTo(delegate.token)).not.toContain(file.uid);
// The holder's grant still stands, so its owner still sees it.
expect(await listedTo(owner.token)).toContain(file.uid);
});
// A caller who believes they filtered must not silently receive
// everything: a duplicated param arrives as an array, and an empty
// string names nothing.
@@ -1,978 +0,0 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { makeActor } from '../../core/actor.js';
import type { IConfig } from '../../types';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
const randomHandle = () => `ws-${Math.random().toString(36).slice(2, 10)}`;
describe('team endpoints over HTTP', () => {
let env: PuterTestEnv;
beforeAll(async () => {
// The cap has its own suite; these tests need many teams.
env = await setupPuterTestEnv({
teams_enabled: true,
max_teams_per_user: 100,
} as IConfig);
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const call = (
method: string,
path: string,
token: string,
body?: unknown,
) =>
fetch(new URL(path, env.apiOrigin), {
method,
headers: {
'content-type': 'application/json',
authorization: `Bearer ${token}`,
},
...(body === undefined ? {} : { body: JSON.stringify(body) }),
});
/** An app token acting for `owner`, minted the way the dashboard does. */
const makeAppToken = async (owner: { username: string }) => {
const user = await env.server.stores.user.getByUsername(
owner.username,
);
const actor = makeActor({ user: user! });
const app = await env.server.stores.app.create(
{
name: `team-http-app-${crypto.randomUUID()}`,
title: 'Team app',
index_url: `https://team-${crypto.randomUUID()}.test/`,
},
{ ownerUserId: actor.user.id! },
);
return env.server.services.auth.getUserAppToken(actor, app.uid);
};
/** A scoped (non-full-access) access token for `owner`, carrying no app. */
const makeScopedToken = async (owner: {
username: string;
token: string;
}) => {
const user = await env.server.stores.user.getByUsername(
owner.username,
);
const anchor = `/${owner.username}/${crypto.randomUUID()}`;
await env.server.services.fs.mkdir(user!.id, {
path: anchor,
createMissingParents: true,
});
const entry = await env.server.stores.fsEntry.getEntryByPath(anchor);
const { actor } = await env.server.services.auth.authenticate(
owner.token,
);
return env.server.services.auth.createAccessToken(
actor!,
[[`fs:${entry!.uid}:list`]],
{ label: 'teams-scope' },
);
};
/** A team owned by `user`, with one provisioned member. */
const makeTeam = async () => {
const res = await call('POST', '/teams', env.users.user.token, {
name: 'Acme',
handle: randomHandle(),
});
// Unasserted, a broken create means tests run on `/teams/undefined`.
expect(res.status).toBe(200);
const team = (await res.json()) as { uid: string };
expect(team.uid).toBeTruthy();
const username = `http_${Math.random().toString(36).slice(2, 9)}`;
const member = await call(
'POST',
`/teams/${team.uid}/members`,
env.users.user.token,
{ username, email: `${username}@test.local` },
);
expect(member.status).toBe(200);
return { team, memberUsername: username };
};
// -- the owner-account gate --------------------------------------
it('creates a team and reports the caller as its owner', async () => {
const res = await call('POST', '/teams', env.users.user.token, {
name: 'Acme Design',
handle: randomHandle(),
});
expect(res.status).toBe(200);
const body = (await res.json()) as Record<string, unknown>;
expect(body).toMatchObject({ name: 'Acme Design', is_owner: true });
expect(body.uid).toMatch(/^[0-9a-f-]{36}$/u);
// `id` is internal and must not reach the wire.
expect(body).not.toHaveProperty('id');
});
it('returns 404, not 403, to a non-member', async () => {
const { team } = await makeTeam();
const res = await call(
'GET',
`/teams/${team.uid}`,
env.users.other.token,
);
// 403 would confirm the team exists.
expect(res.status).toBe(404);
});
it('refuses every administrative route to a member who is not the owner', async () => {
const { team, memberUsername } = await makeTeam();
// An activated seat: provisioning leaves the email unconfirmed, which
// `requireVerified` rejects, so an unactivated one cannot call at all.
const provisioned = await env.server.stores.user.getByUsername(
memberUsername,
);
await env.server.stores.user.update(provisioned!.id, {
email_confirmed: 1,
requires_email_confirmation: 0,
requires_password_change: 0,
});
const seat = await env.server.stores.user.getByUsername(memberUsername);
const { token } = await env.server.services.auth.createSessionToken(
seat!,
);
for (const [method, path] of [
['PUT', `/teams/${team.uid}`],
['DELETE', `/teams/${team.uid}`],
['POST', `/teams/${team.uid}/members`],
['GET', `/teams/${team.uid}/audit`],
] as const) {
const res = await call(
method,
path,
token,
method === 'GET' ? undefined : { name: 'nope' },
);
expect(res.status, `${method} ${path}`).toBe(403);
}
// Still a member, so reads it is entitled to still work.
const readable = await call('GET', `/teams/${team.uid}`, token);
expect(readable.status).toBe(200);
});
// -- the org_owned guard ------------------------------------------
it('refuses the team owner as the target of a member route', async () => {
const { team } = await makeTeam();
const res = await call(
'POST',
`/teams/${team.uid}/members/${env.users.user.username}/disable`,
env.users.user.token,
);
expect(res.status).toBe(404);
});
it('deletes a seat over HTTP only once it is disabled', async () => {
const { team, memberUsername } = await makeTeam();
const path = `/teams/${team.uid}/members/${memberUsername}`;
const live = await call('DELETE', path, env.users.user.token);
expect(live.status).toBe(409);
expect(await live.json()).toMatchObject({
code: 'account_must_be_disabled_first',
});
await call('POST', `${path}/disable`, env.users.user.token);
const gone = await call('DELETE', path, env.users.user.token);
expect(gone.status).toBe(200);
expect(
await env.server.stores.user.getByUsername(memberUsername),
).toBeFalsy();
});
// -- the directory ------------------------------------------------
it('keeps the directory shut until the team opts in', async () => {
const { team } = await makeTeam();
const before = await call(
'GET',
`/teams/${team.uid}/directory`,
env.users.user.token,
);
// 404, not 403: whether this is on is not an app's to probe for.
expect(before.status).toBe(404);
const on = await call('PUT', `/teams/${team.uid}`, env.users.user.token, {
directory_enabled: true,
});
expect(on.status).toBe(200);
expect(await on.json()).toMatchObject({ directory_enabled: true });
const after = await call(
'GET',
`/teams/${team.uid}/directory`,
env.users.user.token,
);
expect(after.status).toBe(200);
const body = (await after.json()) as { items: { username: string }[] };
expect(body.items.some((m) => m.username === env.users.user.username)).toBe(true);
// A scoped token was minted for one thing; a roster is not it.
const scoped = await makeScopedToken(env.users.user);
const narrow = await call(
'GET',
`/teams/${team.uid}/directory`,
typeof scoped === 'string' ? scoped : scoped.token,
);
expect(narrow.status).toBe(403);
});
it('refuses the directory to someone outside the team', async () => {
const { team } = await makeTeam();
await call('PUT', `/teams/${team.uid}`, env.users.user.token, {
directory_enabled: true,
});
const res = await call(
'GET',
`/teams/${team.uid}/directory`,
env.users.other.token,
);
expect(res.status).toBe(404);
});
// -- app actors, gated by the same opt-in as the directory --------
//
// Built through the service, not the wire: these routes share a
// rate-limit budget with the mutation tests above, and every HTTP
// create/update here would eat into that same window.
/** A team owned by `user`, created and toggled through the service. */
const makeServiceTeam = async () => {
const owner = (await env.server.stores.user.getByUsername(
env.users.user.username,
))!;
const team = await env.server.services.team.createTeam(owner.id, {
name: 'Acme',
handle: randomHandle(),
});
return { team, ownerId: owner.id };
};
it('omits a team from an app token until the directory opts in', async () => {
const { team, ownerId } = await makeServiceTeam();
const appToken = await makeAppToken(env.users.user);
const before = await call('GET', '/teams', appToken);
expect(before.status).toBe(200);
const beforeBody = (await before.json()) as { items: { uid: string }[] };
expect(beforeBody.items.some((t) => t.uid === team.uid)).toBe(false);
await env.server.services.team.updateTeam(team.uid, ownerId, {
directoryEnabled: true,
});
const after = await call('GET', '/teams', appToken);
expect(after.status).toBe(200);
const afterBody = (await after.json()) as { items: { uid: string }[] };
expect(afterBody.items.some((t) => t.uid === team.uid)).toBe(true);
});
it('does not count a team it hides from an app token', async () => {
const { team } = await makeServiceTeam();
const appToken = await makeAppToken(env.users.user);
const res = await call('GET', '/teams?includeTotal=true', appToken);
expect(res.status).toBe(200);
const body = (await res.json()) as {
items: { uid: string }[];
total: number;
};
expect(body.items.some((t) => t.uid === team.uid)).toBe(false);
// A count over rows the page withholds reports the hidden team.
expect(body.total).toBe(body.items.length);
});
it('still lists a team to its owner session with the directory off', async () => {
const { team } = await makeServiceTeam();
const res = await call('GET', '/teams', env.users.user.token);
expect(res.status).toBe(200);
const body = (await res.json()) as { items: { uid: string }[] };
expect(body.items.some((t) => t.uid === team.uid)).toBe(true);
});
it('lists a team to a full-access API token with the directory off', async () => {
const { team } = await makeServiceTeam();
const res = await call('GET', '/teams', env.users.user.apiToken);
expect(res.status).toBe(200);
const body = (await res.json()) as { items: { uid: string }[] };
expect(body.items.some((t) => t.uid === team.uid)).toBe(true);
});
it('404s /members to an app token until the directory opts in, then gives usernames only', async () => {
const { team, ownerId } = await makeServiceTeam();
const username = `svc_${Math.random().toString(36).slice(2, 9)}`;
await env.server.services.team.provisionAccount(team.uid, ownerId, {
username,
});
const appToken = await makeAppToken(env.users.user);
const before = await call(
'GET',
`/teams/${team.uid}/members`,
appToken,
);
expect(before.status).toBe(404);
expect(await before.json()).toMatchObject({ code: 'team_not_found' });
await env.server.services.team.updateTeam(team.uid, ownerId, {
directoryEnabled: true,
});
const after = await call(
'GET',
`/teams/${team.uid}/members`,
appToken,
);
expect(after.status).toBe(200);
const body = (await after.json()) as {
items: Record<string, unknown>[];
};
expect(body.items.length).toBeGreaterThan(0);
// An app gets username only — no org_owned, created_at, or uuid.
for (const m of body.items) expect(Object.keys(m)).toEqual(['username']);
});
it('still refuses an app token on GET /teams/:uid', async () => {
const { team } = await makeServiceTeam();
const appToken = await makeAppToken(env.users.user);
const res = await call('GET', `/teams/${team.uid}`, appToken);
expect(res.status).toBe(403);
expect(await res.json()).toMatchObject({ code: 'forbidden' });
});
it('excludes a never-activated or suspended seat from an app`s view of /members', async () => {
const { team, ownerId } = await makeServiceTeam();
const neverActivated = `svc_na_${Math.random().toString(36).slice(2, 9)}`;
await env.server.services.team.provisionAccount(team.uid, ownerId, {
username: neverActivated,
});
const suspendedUsername = `svc_su_${Math.random().toString(36).slice(2, 9)}`;
const suspended = await env.server.services.team.provisionAccount(
team.uid,
ownerId,
{ username: suspendedUsername },
);
await env.server.services.team.disableMember(
team.uid,
ownerId,
suspended.userId,
);
await env.server.services.team.updateTeam(team.uid, ownerId, {
directoryEnabled: true,
});
const appToken = await makeAppToken(env.users.user);
const appView = await call(
'GET',
`/teams/${team.uid}/members`,
appToken,
);
expect(appView.status).toBe(200);
const appItems = (
(await appView.json()) as { items: Record<string, unknown>[] }
).items;
for (const m of appItems) expect(Object.keys(m)).toEqual(['username']);
const appUsernames = appItems.map((m) => m.username);
expect(appUsernames).not.toContain(neverActivated);
expect(appUsernames).not.toContain(suspendedUsername);
// The owner's own seat is active, so it still shows.
expect(appUsernames).toContain(env.users.user.username);
// The owner's own session sees every seat regardless of status.
const ownerView = await call(
'GET',
`/teams/${team.uid}/members`,
env.users.user.token,
);
expect(ownerView.status).toBe(200);
const ownerUsernames = (
(await ownerView.json()) as { items: { username: string }[] }
).items.map((m) => m.username);
expect(ownerUsernames).toContain(neverActivated);
expect(ownerUsernames).toContain(suspendedUsername);
});
it('gives a full-access API token the owner`s uuid on /members', async () => {
const { team } = await makeServiceTeam();
const res = await call(
'GET',
`/teams/${team.uid}/members`,
env.users.user.apiToken,
);
expect(res.status).toBe(200);
const body = (await res.json()) as {
items: { username: string; uuid?: string }[];
};
const owner = body.items.find(
(m) => m.username === env.users.user.username,
);
expect(owner?.uuid).toEqual(expect.any(String));
});
it('refuses a scoped access token on GET /teams and /members', async () => {
const { team } = await makeServiceTeam();
const scoped = await makeScopedToken(env.users.user);
const teamsRes = await call('GET', '/teams', scoped);
expect(teamsRes.status).toBe(403);
expect(await teamsRes.json()).toMatchObject({ code: 'forbidden' });
const membersRes = await call(
'GET',
`/teams/${team.uid}/members`,
scoped,
);
expect(membersRes.status).toBe(403);
expect(await membersRes.json()).toMatchObject({ code: 'forbidden' });
});
it('lists members with org_owned distinguishing the owner', async () => {
const { team, memberUsername } = await makeTeam();
const res = await call(
'GET',
`/teams/${team.uid}/members`,
env.users.user.token,
);
expect(res.status).toBe(200);
const body = (await res.json()) as {
items: { username: string; org_owned: boolean }[];
};
const owner = body.items.find(
(m) => m.username === env.users.user.username,
);
const member = body.items.find((m) => m.username === memberUsername);
expect(owner?.org_owned).toBe(false);
expect(member?.org_owned).toBe(true);
});
it('gives seat uuids to the owner and to nobody else', async () => {
// Built through the service, not the wire: provisioning shares a
// rate-limit budget with the seat tests below, and one more HTTP
// provision here 429s them.
const owner = (await env.server.stores.user.getByUsername(
env.users.user.username,
))!;
const slug0 = Math.random().toString(36).slice(2, 9);
const team = await env.server.services.team.createTeam(owner.id, {
name: 'Acme',
handle: `uuidt-${slug0}`,
});
const memberUsername = `seat_${slug0}`;
await env.server.services.team.provisionAccount(team.uid, owner.id, {
username: memberUsername,
});
const owned = (await (
await call('GET', `/teams/${team.uid}/members`, env.users.user.token)
).json()) as { items: { username: string; uuid?: string }[] };
const seat = owned.items.find((m) => m.username === memberUsername);
// Billing keys a seat's plan on this, so the owner cannot act without it.
expect(seat?.uuid).toEqual(expect.any(String));
// A throwaway member, not the shared fixture: joining a team is
// permanent and would follow `other` into every later test.
const slug = Math.random().toString(36).slice(2, 9);
const joiner = await env.server.stores.user.create({
username: `joiner_${slug}`,
uuid: crypto.randomUUID(),
password: 'hashed',
email: `joiner_${slug}@test.local`,
});
await env.server.stores.team.addMember(team.uid, joiner.id, {
orgOwned: false,
});
const { token } =
await env.server.services.auth.createSessionToken(joiner);
const seen = (await (
await call('GET', `/teams/${team.uid}/members`, token)
).json()) as { items: { uuid?: string }[] };
expect(seen.items.length).toBeGreaterThan(0);
for (const m of seen.items) expect(m.uuid).toBeUndefined();
});
// -- provisioning over the wire -----------------------------------
it('never returns the activation link to the administrator', async () => {
const { team } = await makeTeam();
const username = `secret_${Math.random().toString(36).slice(2, 9)}`;
const res = await call(
'POST',
`/teams/${team.uid}/members`,
env.users.user.token,
{ username, email: `${username}@test.local` },
);
expect(res.status).toBe(200);
const raw = JSON.stringify(await res.json());
// Emailed to the member; returning it would let an admin use it.
expect(raw).not.toContain('set-new-password');
expect(raw).not.toContain('token');
});
it('refuses a taken username and offers alternatives', async () => {
const { team } = await makeTeam();
const res = await call(
'POST',
`/teams/${team.uid}/members`,
env.users.user.token,
{
username: env.users.other.username,
email: '[email protected]',
},
);
expect(res.status).toBe(409);
const body = (await res.json()) as {
suggestions?: string[];
fields?: { suggestions?: string[] };
};
const suggestions = body.suggestions ?? body.fields?.suggestions ?? [];
expect(suggestions.length).toBeGreaterThan(0);
expect(suggestions).not.toContain(env.users.other.username);
});
// -- audit --------------------------------------------------------
it('exposes the team audit to the team owner only', async () => {
const { team } = await makeTeam();
const mine = await call(
'GET',
`/teams/${team.uid}/audit`,
env.users.user.token,
);
expect(mine.status).toBe(200);
const body = (await mine.json()) as { items: { action: string }[] };
expect(body.items.map((e) => e.action)).toContain('provision');
const theirs = await call(
'GET',
`/teams/${team.uid}/audit`,
env.users.other.token,
);
expect(theirs.status).toBe(404);
});
// -- sharing with a team -------------------------------------
it('names the team it shared with instead of an empty recipient', async () => {
const { team } = await makeTeam();
const file = `/${env.users.user.username}/Documents/label-${Math.random()
.toString(36)
.slice(2, 8)}.txt`;
const written = await call('POST', '/fs/write', env.users.user.token, {
fileMetadata: { path: file, size: 3, contentType: 'text/plain' },
fileContent: 'abc',
});
expect(written.status).toBe(200);
const shared = await call('POST', '/share', env.users.user.token, {
recipients: [{ team: team.uid }],
items: [file],
mode: 'read',
});
expect(shared.status).toBe(200);
const body = (await shared.json()) as {
results: { recipient: string; status: string }[];
};
expect(body.results[0].status).toBe('success');
// Echoes the identifier the caller named; '' would leave a client with
// nothing to render or to match its request against.
expect(body.results[0].recipient).toBe(team.uid);
});
it('shares with two teams rather than collapsing them into one', async () => {
const second = await call('POST', '/teams', env.users.user.token, {
name: 'Second',
handle: randomHandle(),
});
expect(second.status).toBe(200);
const teamB = (await second.json()) as { uid: string };
const { team: teamA } = await makeTeam();
const file = `/${env.users.user.username}/Documents/two-${Math.random()
.toString(36)
.slice(2, 8)}.txt`;
const written = await call('POST', '/fs/write', env.users.user.token, {
fileMetadata: { path: file, size: 3, contentType: 'text/plain' },
fileContent: 'abc',
});
const fileUid = ((await written.json()) as { fsEntry: { uid: string } })
.fsEntry.uid;
const shared = await call('POST', '/share', env.users.user.token, {
recipients: [{ team: teamA.uid }, { team: teamB.uid }],
items: [file],
mode: 'read',
});
expect(shared.status).toBe(200);
const body = (await shared.json()) as {
results: { recipient: string; status: string }[];
};
expect(body.results).toHaveLength(2);
expect(body.results.every((r) => r.status === 'success')).toBe(true);
// The response shape alone proves nothing: `results` is built from the
// request's pairs, so a collapsed pair still reports two successes with
// the right labels. Only the shares that exist afterwards show it.
const mine = await call(
'GET',
'/share/shared-by-me?limit=100',
env.users.user.token,
);
expect(mine.status).toBe(200);
const listing = (await mine.json()) as {
items: { uid_entry?: string; holder_team?: { uid: string } }[];
};
const holders = listing.items
.filter((i) => i.uid_entry === fileUid)
.map((i) => i.holder_team?.uid)
.filter(Boolean)
.sort();
expect(holders).toEqual([teamA.uid, teamB.uid].sort());
});
it('names the team by handle when the caller shared by handle', async () => {
const handle = randomHandle();
const res = await call('POST', '/teams', env.users.user.token, {
name: 'Byhandle',
handle,
});
expect(res.status).toBe(200);
const file = `/${env.users.user.username}/Documents/handle-${Math.random()
.toString(36)
.slice(2, 8)}.txt`;
await call('POST', '/fs/write', env.users.user.token, {
fileMetadata: { path: file, size: 3, contentType: 'text/plain' },
fileContent: 'abc',
});
const shared = await call('POST', '/share', env.users.user.token, {
recipients: [{ teamHandle: handle }],
items: [file],
mode: 'read',
});
expect(shared.status).toBe(200);
const body = (await shared.json()) as {
results: { recipient: string }[];
};
expect(body.results[0].recipient).toBe(handle);
});
it('gives a non-member 404 on the member view, not an empty page', async () => {
const { team } = await makeTeam();
const res = await call(
'GET',
`/teams/${team.uid}/audit/me`,
env.users.other.token,
);
expect(res.status).toBe(404);
});
// -- the forced-change gate ---------------------------------------
/**
* A seat that has signed in on its temporary password. Email confirmation
* is cleared first so the gate under test is the one that answers.
*/
const signedInSeat = async (teamUid: string) => {
const username = `seat_${Math.random().toString(36).slice(2, 9)}`;
const res = await call(
'POST',
`/teams/${teamUid}/members`,
env.users.user.token,
{ username, email: `${username}@test.local` },
);
expect(res.status).toBe(200);
const { temporary_password: password } = (await res.json()) as {
temporary_password: string;
};
const row = (await env.server.stores.user.getByUsername(username))!;
await env.server.stores.user.update(row.id, {
requires_email_confirmation: false,
email_confirmed: true,
});
await env.server.stores.user.invalidateById(row.id);
const fresh = (await env.server.stores.user.getById(row.id))!;
const { token } = await env.server.services.auth.createSessionToken(
fresh,
{ user_agent: 'puter-test-seat' },
);
return { username, password, userId: row.id, token };
};
/**
* Cookie-credentialed on the GUI origin, as the user-protected gate insists.
* Sent with the seat's own user agent: the session touch otherwise rewrites
* it to Node's once a second boundary passes.
*/
const changePassword = (token: string, password: string, next: string) =>
fetch(new URL('/user-protected/change-password', env.origin), {
method: 'POST',
headers: {
'content-type': 'application/json',
'user-agent': 'puter-test-seat',
authorization: `Bearer ${token}`,
cookie: `puter_auth_token=${token}`,
},
body: JSON.stringify({ password, new_pass: next }),
});
it('signs the seat in but refuses every authenticated route after that', async () => {
const { team } = await makeTeam();
const seat = await signedInSeat(team.uid);
const res = await call('GET', '/teams', seat.token);
expect(res.status).toBe(403);
expect(await res.text()).toContain('password_change_required');
});
it('leaves change-password reachable, since it is what clears the gate', async () => {
const { team } = await makeTeam();
const seat = await signedInSeat(team.uid);
const changed = await changePassword(
seat.token,
seat.password,
'chosen-by-the-member',
);
expect(changed.status).toBe(200);
const row = await env.server.stores.user.getByProperty(
'id',
seat.userId,
{ force: true },
);
expect(Number(row?.requires_password_change)).toBe(0);
expect(row?.temp_password_expires_at ?? null).toBeNull();
});
it('admits the seat to the product once it has chosen a password', async () => {
const { team } = await makeTeam();
const seat = await signedInSeat(team.uid);
expect(
(await changePassword(seat.token, seat.password, 'my-own-password'))
.status,
).toBe(200);
// The session that changed the password is the one that survives.
const res = await call('GET', '/teams', seat.token);
expect(res.status).toBe(200);
});
it('closes re-issue behind a seat that has activated', async () => {
const { team } = await makeTeam();
const seat = await signedInSeat(team.uid);
await changePassword(seat.token, seat.password, 'a-password-of-mine');
const res = await call(
'POST',
`/teams/${team.uid}/members/${seat.username}/activation`,
env.users.user.token,
);
expect(res.status).toBe(409);
});
it('shows the member the reset and their own sign-in, and nothing else', async () => {
const { team } = await makeTeam();
const seat = await signedInSeat(team.uid);
await changePassword(seat.token, seat.password, 'chosen-once-already');
const reset = await call(
'POST',
`/teams/${team.uid}/members/${seat.username}/password-reset`,
env.users.user.token,
);
expect(reset.status).toBe(200);
const { temporary_password: issued } = (await reset.json()) as {
temporary_password: string;
};
const { token } = await env.server.services.auth.createSessionToken(
(await env.server.stores.user.getById(seat.userId))!,
{ user_agent: 'puter-test-seat' },
);
const mine = await call('GET', `/teams/${team.uid}/audit/me`, token);
// The seat owes a password change again, so its own view is all it reaches.
expect(mine.status).toBe(403);
const admin = await call(
'GET',
`/teams/${team.uid}/audit`,
env.users.user.token,
);
const body = (await admin.json()) as { items: { action: string }[] };
expect(body.items.map((e) => e.action)).toContain(
'reset_member_password',
);
expect(JSON.stringify(body)).not.toContain(issued);
});
it('shows an activated seat its own reset and sign-in over the wire', async () => {
const { team } = await makeTeam();
const seat = await signedInSeat(team.uid);
await changePassword(seat.token, seat.password, 'the-one-i-picked');
const res = await call('GET', `/teams/${team.uid}/audit/me`, seat.token);
expect(res.status).toBe(200);
const body = (await res.json()) as {
items: {
action: string;
username: string | null;
ip: string | null;
user_agent: string | null;
}[];
};
expect(body.items.map((e) => e.action)).toContain('activate');
const tell = body.items.find((e) => e.action === 'sign_in');
expect(tell?.username).toBe(seat.username);
expect(tell?.user_agent).toBe('puter-test-seat');
// Only their own; the team owner's entries are not theirs to read.
expect(
body.items.every((e) => e.username === seat.username),
).toBe(true);
});
it('refuses a temporary password that was never used in time', async () => {
const { team } = await makeTeam();
const seat = await signedInSeat(team.uid);
await env.server.stores.user.update(seat.userId, {
temp_password_expires_at: Math.floor(Date.now() / 1000) - 1,
});
await env.server.stores.user.invalidateById(seat.userId);
const res = await fetch(new URL('/login', env.origin), {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({
username: seat.username,
password: seat.password,
}),
});
expect(res.status).toBe(401);
expect(await res.text()).toContain('temporary_password_expired');
});
});
describe('team endpoints with teams_enabled off', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv({ teams_enabled: false } as IConfig);
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
it('404s every team route, so no team code is reachable', async () => {
// Real team: 200 with the flag on, so a 404 means no route.
const owner = await env.server.stores.user.getByUsername(
env.users.user.username,
);
const team = await env.server.stores.team.create({
ownerUserId: owner!.id,
name: 'Unreachable',
handle: `off-${Math.random().toString(36).slice(2, 8)}`,
});
await env.server.stores.team.addMember(team.uid, owner!.id, {
orgOwned: false,
});
const paths: [string, string][] = [
['POST', '/teams'],
['GET', '/teams'],
['GET', `/teams/${team.uid}`],
['PUT', `/teams/${team.uid}`],
['DELETE', `/teams/${team.uid}`],
['GET', `/teams/${team.uid}/members`],
['POST', `/teams/${team.uid}/members`],
['GET', `/teams/${team.uid}/audit`],
['GET', `/teams/${team.uid}/audit/me`],
];
for (const [method, path] of paths) {
const res = await fetch(new URL(path, env.apiOrigin), {
method,
headers: {
'content-type': 'application/json',
authorization: `Bearer ${env.users.user.token}`,
},
...(method === 'POST' || method === 'PUT'
? { body: JSON.stringify({ name: 'x' }) }
: {}),
});
expect(res.status, `${method} ${path}`).toBe(404);
// Ours would carry a team legacyCode; the framework's does not.
const body = await res.text();
expect(body, `${method} ${path}`).not.toContain('team_not_found');
expect(body, `${method} ${path}`).not.toContain(
'not_the_team_owner',
);
}
});
it('registers no team routes at all with the flag off', async () => {
// The controller exists and is constructed; only its routes are absent.
const controller = env.server.controllers.team as unknown as {
isEnabled?: () => boolean;
};
expect(controller).toBeTruthy();
expect(controller.isEnabled?.()).toBe(false);
});
it('leaves the schema inert rather than absent', async () => {
// The flag gates reachability, not DDL.
await expect(
env.server.stores.team.getByHandle('no-such-team'),
).resolves.toBeNull();
});
});
@@ -1,559 +0,0 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, Response } from 'express';
import {
Controller,
Delete,
Get,
Post,
Put,
} from '../../core/http/decorators.js';
import { isAccountContext } from '../../core/actor.js';
import { HttpError } from '../../core/http/HttpError.js';
import type { TeamRow } from '../../stores/team/TeamStore.js';
import { PuterController } from '../types.js';
/** Mirrors ShareController's dual-window shape. */
const TEAM_LIMIT = [
{ scope: 'team:mutate', limit: 60, window: 60_000, key: 'user' as const },
{
scope: 'team:mutate-daily',
limit: 500,
window: 24 * 60 * 60_000,
key: 'user' as const,
},
];
/** Tighter than TEAM_LIMIT: a reset hands the admin a working credential. */
const TEAM_RESET_LIMIT = [
...TEAM_LIMIT,
{
scope: 'team:reset-password',
limit: 20,
window: 24 * 60 * 60_000,
key: 'user' as const,
},
];
const TEAM_READ_LIMIT = {
scope: 'team:read',
limit: 600,
window: 60_000,
key: 'user' as const,
};
/** What a team looks like on the wire. `id` stays internal. */
const toClientTeam = (team: TeamRow, isOwner: boolean) => ({
uid: team.uid,
name: team.name,
handle: team.handle,
is_owner: isOwner,
created_at: team.created_at,
directory_enabled: Number(team.directory_enabled) === 1,
require_2fa: Number(team.require_2fa) === 1,
});
@Controller('/teams')
export class TeamController extends PuterController {
// Most routes gate with `requireUserActor`; `listTeams` and `listMembers`
// use `requireAuth` so an app can read a team whose owner opted into the
// directory (see `directory_enabled`).
/** Off means `/teams` 404s and no team route is registered at all. */
isEnabled(): boolean {
return this.config.teams_enabled === true;
}
@Post('', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_LIMIT,
})
async createTeam(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
await this.#requireTeamsAvailable(req, userId);
const body = this.#body(req);
const team = await this.services.team.createTeam(userId, {
name: this.#requireString(body.name, 'name'),
handle:
body.handle === undefined || body.handle === null
? null
: this.#requireString(body.handle, 'handle'),
});
res.json(toClientTeam(team, true));
}
@Get('', {
subdomain: 'api',
requireAuth: true,
requireVerified: true,
rateLimit: TEAM_READ_LIMIT,
})
async listTeams(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
await this.#requireTeamsAvailable(req, userId);
// In SQL, or cursor and total describe teams the app cannot see.
const page = await this.stores.team.listTeamsForUser(userId, {
limit: req.query.limit,
cursor:
typeof req.query.cursor === 'string'
? req.query.cursor
: undefined,
includeTotal: req.query.includeTotal === 'true',
openOnly: !isAccountContext(req.actor),
});
res.json({
items: page.items.map((t) =>
toClientTeam(t, t.owner_user_id === userId),
),
...(page.cursor ? { cursor: page.cursor } : {}),
...(page.total === undefined ? {} : { total: page.total }),
});
}
@Get('/:uid', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_READ_LIMIT,
})
async getTeam(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
const team = await this.services.team.requireMembership(
this.#param(req, 'uid'),
userId,
);
res.json(toClientTeam(team, team.owner_user_id === userId));
}
@Put('/:uid', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_LIMIT,
})
async updateTeam(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
await this.services.team.requireOwner(this.#param(req, 'uid'), userId);
const body = this.#body(req);
const changes: {
name?: string;
handle?: string | null;
directoryEnabled?: boolean;
require2fa?: boolean;
} = {};
if (body.name !== undefined)
changes.name = this.#requireString(body.name, 'name');
if (body.handle !== undefined)
changes.handle = body.handle === null ? null : String(body.handle);
if (body.directory_enabled !== undefined)
changes.directoryEnabled = body.directory_enabled === true;
if (body.require_2fa !== undefined)
changes.require2fa = body.require_2fa === true;
// Through the service: turning the directory on writes an audit row.
const team = await this.services.team.updateTeam(
this.#param(req, 'uid'),
userId,
changes,
);
res.json(toClientTeam(team, true));
}
@Delete('/:uid', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_LIMIT,
})
async deleteTeam(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
await this.services.team.deleteTeam(this.#param(req, 'uid'), userId);
res.json({ success: true });
}
// -- Members ------------------------------------------------------
@Get('/:uid/members', {
subdomain: 'api',
requireAuth: true,
requireVerified: true,
rateLimit: TEAM_READ_LIMIT,
})
async listMembers(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
const team = await this.services.team.requireMembership(
this.#param(req, 'uid'),
userId,
);
const accountContext = isAccountContext(req.actor);
// Same consent as the directory: an app only reads a team that opted in.
if (!accountContext) this.services.team.assertDirectoryOpen(team);
// uuid only when the team owner calls with their own session or API
// token; an app needing a stable id uses `listDirectory()` instead.
const showUuid = accountContext && team.owner_user_id === userId;
const page = await this.stores.team.listMembers(
this.#param(req, 'uid'),
{
limit: req.query.limit,
cursor:
typeof req.query.cursor === 'string'
? req.query.cursor
: undefined,
// An app's view matches the directory's: active seats only.
activeOnly: !accountContext,
},
);
// An app only ever gets a username; org_owned/created_at are account-only.
res.json({
items: page.items.map((m) =>
accountContext
? {
username: m.username,
org_owned: Number(m.org_owned) === 1,
created_at: m.created_at,
...(showUuid ? { uuid: m.uuid } : {}),
}
: { username: m.username },
),
...(page.cursor ? { cursor: page.cursor } : {}),
});
}
/** App actors too, once opted in; carries the `uuid` `/members` hides. */
@Get('/:uid/directory', {
subdomain: 'api',
requireVerified: true,
rateLimit: TEAM_READ_LIMIT,
})
async listDirectory(req: Request, res: Response): Promise<void> {
// The membership is the person's, never the app's.
const userId = this.#requireUserId(req);
// A scoped token holds only what it was minted for; a roster is not that.
const token = req.actor?.accessToken;
if (token && token.fullAccess !== true) {
throw new HttpError(
403,
'This endpoint is not available to scoped access tokens',
{ legacyCode: 'forbidden' },
);
}
const page = await this.services.team.listDirectory(
this.#param(req, 'uid'),
userId,
{
limit: req.query.limit,
cursor:
typeof req.query.cursor === 'string'
? req.query.cursor
: undefined,
},
);
res.json(page);
}
@Post('/:uid/members', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_LIMIT,
})
async createMember(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
const uid = this.#param(req, 'uid');
// Authority before shape, or a stranger learns if their body parsed.
await this.services.team.requireOwner(uid, userId);
const body = this.#body(req);
const result = await this.services.team.provisionAccount(uid, userId, {
username: this.#requireString(body.username, 'username'),
email: this.#optionalString(body.email, 'email'),
});
// Shown once; the admin delivers it out of band.
res.json({
username: result.username,
temporary_password: result.temporaryPassword,
});
}
// Same budget as a reset: both hand the administrator a working credential,
// and a reset re-arms the flag this route needs.
@Post('/:uid/members/:username/activation', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_RESET_LIMIT,
})
async reissueCredential(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
const uid = this.#param(req, 'uid');
// Authority first, or resolving `:username` is an existence oracle.
await this.services.team.requireOwner(uid, userId);
const target = await this.#requireTargetUserId(req);
const { temporaryPassword } =
await this.services.team.reissueCredential(uid, userId, target);
// Shown once for the admin to deliver out of band.
res.json({ temporary_password: temporaryPassword });
}
@Post('/:uid/members/:username/password-reset', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_RESET_LIMIT,
})
async resetMemberPassword(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
const uid = this.#param(req, 'uid');
// Authority first, or resolving `:username` is an existence oracle.
await this.services.team.requireOwner(uid, userId);
const target = await this.#requireTargetUserId(req);
const { temporaryPassword } =
await this.services.team.resetMemberPassword(uid, userId, target);
// Shown once; it is not retrievable afterwards.
res.json({ temporary_password: temporaryPassword });
}
// The gate closes the seat's own route to fix it, so the owner must.
@Post('/:uid/members/:username/email', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_LIMIT,
})
async updateSeatEmail(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
const uid = this.#param(req, 'uid');
// Authority first, or resolving `:username` is an existence oracle.
await this.services.team.requireOwner(uid, userId);
const target = await this.#requireTargetUserId(req);
await this.services.team.updateSeatEmail(
uid,
userId,
target,
(req.body as { email?: unknown } | undefined)?.email ?? null,
);
res.json({ success: true });
}
// Same budget as a password reset: both are a step toward the account.
@Post('/:uid/members/:username/2fa-reset', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_RESET_LIMIT,
})
async resetMemberTwoFactor(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
const uid = this.#param(req, 'uid');
// Authority first, or resolving `:username` is an existence oracle.
await this.services.team.requireOwner(uid, userId);
const target = await this.#requireTargetUserId(req);
await this.services.team.resetMemberTwoFactor(uid, userId, target);
res.json({ success: true });
}
@Post('/:uid/members/:username/disable', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_LIMIT,
})
async disableMember(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
const uid = this.#param(req, 'uid');
// Authority first, or resolving `:username` is an existence oracle.
await this.services.team.requireOwner(uid, userId);
const target = await this.#requireTargetUserId(req);
await this.services.team.disableMember(
this.#param(req, 'uid'),
userId,
target,
);
res.json({ success: true });
}
@Post('/:uid/members/:username/enable', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_LIMIT,
})
async enableMember(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
const uid = this.#param(req, 'uid');
// Authority first, or resolving `:username` is an existence oracle.
await this.services.team.requireOwner(uid, userId);
const target = await this.#requireTargetUserId(req);
await this.services.team.enableMember(
this.#param(req, 'uid'),
userId,
target,
);
res.json({ success: true });
}
@Delete('/:uid/members/:username', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_LIMIT,
})
async deleteMember(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
const uid = this.#param(req, 'uid');
// Authority first (anti-oracle); deleted team included, see the service.
await this.services.team.requireOwnedTeam(uid, userId);
const target = await this.#requireTargetUserId(req);
await this.services.team.deleteMember(uid, userId, target);
res.json({ success: true });
}
// -- Audit --------------------------------------------------------
@Get('/:uid/audit', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_READ_LIMIT,
})
async listAudit(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
res.json(
await this.services.team.listAudit(
this.#param(req, 'uid'),
userId,
this.#pageOpts(req),
),
);
}
@Get('/:uid/audit/me', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_READ_LIMIT,
})
async listOwnAudit(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
res.json(
await this.services.team.listOwnAudit(
this.#param(req, 'uid'),
userId,
this.#pageOpts(req),
),
);
}
// -- Helpers ------------------------------------------------------
/** `limit` and `cursor` as doc/pagination.md defines them. */
#pageOpts(req: Request): { limit?: unknown; cursor?: string } {
return {
limit: req.query.limit,
cursor:
typeof req.query.cursor === 'string'
? req.query.cursor
: undefined,
};
}
/**
* The domain-allowlist gate, on the two routes that enter the feature. Same
* 404 as a teams-off deployment; other routes bound by membership.
*/
async #requireTeamsAvailable(req: Request, userId: number): Promise<void> {
const email = (
req.actor as { user?: { email?: string | null } } | undefined
)?.user?.email;
if (await this.services.team.teamsAvailableTo(userId, email)) return;
throw new HttpError(404, 'Not found', { legacyCode: 'not_found' });
}
#requireUserId(req: Request): number {
const id = (req.actor as { user?: { id?: number } } | undefined)?.user
?.id;
if (!id)
throw new HttpError(401, 'User required', {
legacyCode: 'unauthorized',
});
return id;
}
/** Express types a param as `string | string[]`; routes here take one. */
#param(req: Request, name: string): string {
const value = req.params[name];
return Array.isArray(value) ? (value[0] ?? '') : (value ?? '');
}
#body(req: Request): Record<string, unknown> {
return (req.body ?? {}) as Record<string, unknown>;
}
/** Absent or empty means "not given"; a wrong type is still a 400. */
#optionalString(value: unknown, field: string): string | null {
if (value === undefined || value === null) return null;
if (typeof value !== 'string') {
throw new HttpError(400, `${field} must be a string`, {
legacyCode: 'bad_request',
});
}
return value.trim() === '' ? null : value;
}
#requireString(value: unknown, field: string): string {
if (typeof value !== 'string' || value.trim() === '') {
throw new HttpError(400, `${field} is required`, {
legacyCode: 'bad_request',
});
}
return value;
}
#notFound(): HttpError {
return new HttpError(404, 'Team not found', {
legacyCode: 'team_not_found',
});
}
/** Resolves `:username` to an id; the service decides whether it may act. */
async #requireTargetUserId(req: Request): Promise<number> {
const user = await this.stores.user.getByUsername(
this.#param(req, 'username'),
);
if (!user)
throw new HttpError(404, 'Not an account of this team', {
legacyCode: 'not_an_org_account',
});
return user.id;
}
}
@@ -1,220 +0,0 @@
/**
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* A team manages accounts and cannot read them. Nobody writes that grant
* deliberately, but a new implicator or a widened actor would create it and no
* other test would fail. Asserts outcomes, never that an implicator is absent.
*/
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { makeActor } from '../../core/actor.js';
import type { IConfig } from '../../types';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
describe('a team cannot read its members data', () => {
let env: PuterTestEnv;
let teamUid: string;
let memberUsername: string;
let memberFile: string;
let ownerFile: string;
let memberToken: string;
/** Written directly: these tests are about reading, not about writing. */
const makeFile = async (username: string, label: string) => {
const uid = crypto.randomUUID();
const name = `${label}-${uid.slice(0, 8)}.txt`;
const path = `/${username}/${name}`;
const user = await env.server.stores.user.getByUsername(username);
await env.server.clients.db.write(
'INSERT INTO `fsentries` (`uuid`, `name`, `path`, `user_id`, `is_dir`, `modified`) ' +
'VALUES (?, ?, ?, ?, ?, ?)',
[
uid,
name,
path,
user!.id,
// `is_dir` is a real boolean on postgres.
env.server.clients.db.booleanValue(false),
Math.floor(Date.now() / 1000),
],
);
return path;
};
const stat = (path: string, token: string) =>
fetch(new URL('/stat', env.apiOrigin), {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${token}`,
},
body: JSON.stringify({ path }),
});
const readdir = (path: string, token: string) =>
fetch(new URL('/readdir', env.apiOrigin), {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${token}`,
},
body: JSON.stringify({ path }),
});
beforeAll(async () => {
// The cap has its own suite; these tests need many teams.
env = await setupPuterTestEnv({
teams_enabled: true,
max_teams_per_user: 100,
} as IConfig);
// A provisioned account cannot authenticate until it activates.
const res = await fetch(new URL('/teams', env.apiOrigin), {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${env.users.user.token}`,
},
body: JSON.stringify({ name: 'Isolation', handle: `iso-${Date.now()}` }),
});
teamUid = ((await res.json()) as { uid: string }).uid;
// A real provisioned seat, activated so it can authenticate.
memberUsername = `iso_${Math.random().toString(36).slice(2, 9)}`;
await env.server.services.team.provisionAccount(
teamUid,
(await env.server.stores.user.getByUsername(
env.users.user.username,
))!.id,
{ username: memberUsername, email: `${memberUsername}@test.local` },
);
const member = await env.server.stores.user.getByUsername(memberUsername);
await env.server.stores.user.update(member!.id, {
email_confirmed: 1,
requires_email_confirmation: 0,
requires_password_change: 0,
});
memberToken = (
await env.server.services.auth.createSessionToken(
(await env.server.stores.user.getByUsername(memberUsername))!,
)
).token;
memberFile = await makeFile(memberUsername, 'member');
ownerFile = await makeFile(env.users.user.username, 'owner');
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
// -- refused ------------------------------------------------------
it('refuses the team owner a members file', async () => {
const res = await stat(memberFile, env.users.user.token);
// Administering, paying for, and reading an account are three things.
expect(res.status).toBeGreaterThanOrEqual(400);
});
it('refuses the team owner a members home directory', async () => {
const res = await readdir(`/${memberUsername}`, env.users.user.token);
expect(res.status).toBeGreaterThanOrEqual(400);
});
it('refuses a full-access token no less than a session', async () => {
// `#scanAccessToken` re-scans the issuer as a plain user actor.
const res = await stat(memberFile, env.users.user.apiToken);
expect(res.status).toBeGreaterThanOrEqual(400);
});
it('grants no team-wide reach through the team routes', async () => {
// There is no route to a member's files or KV, and none should appear.
for (const path of [
`/teams/${teamUid}/files`,
`/teams/${teamUid}/members/${memberUsername}/files`,
`/teams/${teamUid}/kv`,
]) {
const res = await fetch(new URL(path, env.apiOrigin), {
headers: { authorization: `Bearer ${env.users.user.token}` },
});
expect(res.status, path).toBe(404);
}
});
// -- allowed ------------------------------------------------------
it('lets the team owner read its own files', async () => {
const res = await stat(ownerFile, env.users.user.token);
expect(res.status).toBe(200);
});
it('lets a member read their own files', async () => {
const res = await stat(memberFile, memberToken);
expect(res.status).toBe(200);
});
it('lets an explicit grant through, as an ordinary share', async () => {
const member = await env.server.stores.user.getByUsername(
memberUsername,
);
const [entry] = (await env.server.clients.db.read(
'SELECT `uuid` FROM `fsentries` WHERE `path` = ?',
[memberFile],
)) as { uuid: string }[];
// Only the member's own grant changes, never team authority.
const before = await stat(memberFile, env.users.user.token);
expect(before.status).toBeGreaterThanOrEqual(400);
await env.server.services.permission.grantUserUserPermission(
makeActor({ user: member!, app: null, accessToken: null } as never),
env.users.user.username,
`fs:${entry.uuid}:read`,
);
const after = await stat(memberFile, env.users.user.token);
expect(after.status).toBe(200);
});
it('disabling a member neither grants nor transfers their files', async () => {
const owner = await env.server.stores.user.getByUsername(
env.users.user.username,
);
const member = await env.server.stores.user.getByUsername(
memberUsername,
);
// A fresh file: the earlier test granted the owner read on `memberFile`.
const untouched = await makeFile(memberUsername, 'untouched');
await env.server.services.team.disableMember(
teamUid,
owner!.id,
member!.id,
);
// The member loses access to their own files...
const asMember = await stat(untouched, memberToken);
expect(asMember.status).toBeGreaterThanOrEqual(400);
// ...and the team still does not gain it.
const asOwner = await stat(untouched, env.users.user.token);
expect(asOwner.status).toBeGreaterThanOrEqual(400);
});
});
@@ -1,11 +1,13 @@
// This suite tests basic features of puter webdav. it is not a comprehensive webdav test suite unlike litmus
// but rather it performs some common sense checks to ensure that WebDAV support isn't irrevocably broken in puter
import { installSeatStub } from '../../testFixtures/seatStub.js';
import type { Request, RequestHandler, Response } from 'express';
import { Readable, Writable } from 'node:stream';
import { GetObjectCommand } from '@aws-sdk/client-s3';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import { hash as bcryptHash } from 'bcrypt';
import { TOTP } from 'otpauth';
import { PuterRouter } from '../../core/http/PuterRouter.js';
import type { RouteDescriptor } from '../../core/http/types.js';
import { PuterServer } from '../../server.js';
@@ -14,6 +16,7 @@ import { makeActor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { FULL_API_ACCESS } from '../../services/permission/consts.js';
import { generateDefaultFsentries } from '../../util/userProvisioning.js';
import { createSecret as otpCreateSecret } from '../../services/auth/OTPUtil.js';
import type { WebDAVController } from './WebDAVController.js';
let server: PuterServer;
@@ -670,6 +673,92 @@ describe('WebDAVController', () => {
});
});
describe('Basic-auth account parity with /login', () => {
const makeBasicUser = async (
prefix: string,
fields: Record<string, unknown>,
) => {
const username = `${prefix}-${Math.random().toString(36).slice(2, 10)}`;
const created = await server.stores.user.create({
username,
uuid: uuidv4(),
password: await bcryptHash('correct-horse', 4),
email: `${username}@test.local`,
free_storage: 100 * 1024 * 1024,
requires_email_confirmation: false,
});
await server.stores.user.update(created.id, fields);
return { id: created.id, username };
};
const propfindAs = async (username: string, password: string) => {
const { res, captured } = makeRes();
await dispatchMiddleware(
makeReq({
method: 'PROPFIND',
headers: { authorization: basicAuth(username, password) },
}),
res,
noop,
);
return captured.statusCode;
};
const nowSeconds = () => Math.floor(Date.now() / 1000);
it('rejects an expired temporary password with 401', async () => {
const { username } = await makeBasicUser('webdav-temp-expired', {
temp_password_expires_at: nowSeconds() - 60,
});
expect(await propfindAs(username, 'correct-horse')).toBe(401);
});
it('rejects a seat that has not replaced its temporary password with 403', async () => {
const { username } = await makeBasicUser('webdav-temp-live', {
temp_password_expires_at: nowSeconds() + 3600,
requires_password_change: 1,
});
expect(await propfindAs(username, 'correct-horse')).toBe(403);
});
it('lets a live temporary password through once the change flag is clear', async () => {
const { username } = await makeBasicUser('webdav-temp-ok', {
temp_password_expires_at: nowSeconds() + 3600,
requires_password_change: 0,
});
const status = await propfindAs(username, 'correct-horse');
expect([401, 403]).not.toContain(status);
});
it('lets a 2FA member of a team that requires 2FA through', async () => {
const { id, username } = await makeBasicUser('webdav-2fa', {});
const secret = otpCreateSecret(username).secret;
await server.stores.user.update(id, {
otp_enabled: 1,
otp_secret: secret,
});
// Teams is optional, so the gate's input is stood up directly.
installSeatStub(server as never, [id], { require2fa: true });
const seat = vi.spyOn(server.stores.team as never, 'getOrgSeat');
try {
const code = new TOTP({
issuer: 'puter.com',
label: username,
algorithm: 'SHA1',
digits: 6,
secret,
}).generate();
const status = await propfindAs(
username,
`correct-horse${code}`,
);
expect([401, 403]).not.toContain(status);
} finally {
seat.mockRestore();
}
});
});
describe('unsupported methods', () => {
it('returns 405 for unknown HTTP methods', async () => {
const { res, captured } = makeRes();
@@ -918,7 +1007,7 @@ describe('WebDAVController', () => {
});
describe('DELETE', () => {
it('rejects delete when ACL denies write access', async () => {
it('refuses a delete outside the caller’s tree as not found', async () => {
const { res, captured } = makeRes();
await dispatchMiddleware(
makeReq({
@@ -935,7 +1024,7 @@ describe('WebDAVController', () => {
res,
noop,
);
expect(captured.statusCode).toBe(403);
expect(captured.statusCode).toBe(404);
});
});
@@ -1505,7 +1594,7 @@ describe('WebDAVController verbs', () => {
expect(captured.body).toBe('Cannot GET a directory');
});
it("denies reading another user's file with 403", async () => {
it("hides another user's file as not found", async () => {
const owner = await makeUser();
const stranger = await makeUser();
const path = `/${owner.username}/Documents/private.txt`;
@@ -1516,8 +1605,8 @@ describe('WebDAVController verbs', () => {
path,
actor: stranger.actor,
});
expect(captured.statusCode).toBe(403);
expect(captured.body).toBe('Permission denied');
expect(captured.statusCode).toBe(404);
expect(captured.body).toBe('Not Found');
});
});
@@ -1805,6 +1894,18 @@ describe('WebDAVController verbs', () => {
expect(captured.statusCode).toBe(207);
expect(captured.body as string).toContain('HTTP/1.1 200 OK');
});
it("refuses a property update on another user's path", async () => {
const owner = await makeUser();
const other = await makeUser();
const captured = await dispatch({
method: 'PROPPATCH',
path: `/${owner.username}/Documents`,
actor: other.actor,
});
// Out of sight, so refused like a path with nothing at it.
expect(captured.statusCode).toBe(404);
});
});
describe('MKCOL', () => {
@@ -1837,7 +1938,8 @@ describe('WebDAVController verbs', () => {
path: `/${owner.username}/Documents/intruder`,
actor: stranger.actor,
});
expect(captured.statusCode).toBe(403);
// The parent is out of sight, so the refusal doesn't confirm it.
expect(captured.statusCode).toBe(404);
});
});
@@ -2048,7 +2150,7 @@ describe('WebDAVController verbs', () => {
},
});
expect(captured.statusCode).toBe(404);
expect(captured.body).toBe('Source not found');
expect(captured.body).toBe('Not Found');
});
it('returns 409 when the destination parent is not a directory', async () => {
@@ -2174,7 +2276,7 @@ describe('WebDAVController verbs', () => {
},
});
expect(captured.statusCode).toBe(404);
expect(captured.body).toBe('Source not found');
expect(captured.body).toBe('Not Found');
});
it("denies moving into another user's home", async () => {
@@ -2192,7 +2294,7 @@ describe('WebDAVController verbs', () => {
host: 'dav.puter.localhost',
},
});
expect(captured.statusCode).toBe(403);
expect(captured.statusCode).toBe(404);
});
});
@@ -2363,6 +2465,7 @@ describe('WebDAVController verbs', () => {
const shareFolder = async (
owner: Awaited<ReturnType<typeof makeUser>>,
to: Awaited<ReturnType<typeof makeUser>>,
mode: 'read' | 'write' = 'read',
) => {
const dirUuid = uuidv4();
const dirPath = `/${owner.username}/Documents/Secrets`;
@@ -2400,32 +2503,34 @@ describe('WebDAVController verbs', () => {
server.services.share.share(owner.actor as never, {
uid: dirUuid,
recipient: { username: to.username! },
mode: 'read',
mode,
}),
);
return { dirUuid, dirPath };
};
/**
* PROPFIND under the ALS wrap a real request gets, pre-auth actor
* A request under the ALS wrap a real request gets, pre-auth actor
* empty.
*/
const propfind = async (path: string, actor: unknown, depth = '1') => {
const davRequest = async (
method: string,
path: string,
actor: unknown,
headers: Record<string, string> = {},
) => {
const { res, captured } = makeRes();
await runWithContext({ actor: undefined }, () =>
dispatchMiddleware(
makeReq({
method: 'PROPFIND',
path,
headers: { depth },
actor,
}),
makeReq({ method, path, headers, actor }),
res,
noop,
),
);
return captured;
};
const propfind = (path: string, actor: unknown, depth = '1') =>
davRequest('PROPFIND', path, actor, { depth });
// The DAV controller authenticates in-controller, after the request
// context snapshotted an empty `req.actor` — so it has to place the
@@ -2445,6 +2550,40 @@ describe('WebDAVController verbs', () => {
expect(xml).not.toContain('/Documents/Secrets');
});
it('refuses PROPPATCH on a share the recipient can only read', async () => {
const owner = await makeUser();
const recipient = await makeUser();
const { dirUuid } = await shareFolder(owner, recipient);
const masked = `/${owner.username}/${dirUuid}/Secrets`;
const captured = await davRequest(
'PROPPATCH',
masked,
recipient.actor,
);
expect(captured.statusCode).toBe(403);
expect(String(captured.body)).not.toContain('/Documents/Secrets');
});
it('answers PROPPATCH on a writable share under its masked path', async () => {
const owner = await makeUser();
const recipient = await makeUser();
const { dirUuid } = await shareFolder(owner, recipient, 'write');
const masked = `/${owner.username}/${dirUuid}/Secrets`;
const captured = await davRequest(
'PROPPATCH',
masked,
recipient.actor,
);
expect(captured.statusCode).toBe(207);
const xml = String(captured.body);
expect(xml).toContain(`<D:href>${masked}</D:href>`);
expect(xml).not.toContain('/Documents/Secrets');
});
it('lists each sharer as a collection under the root', async () => {
const owner = await makeUser();
const recipient = await makeUser();
@@ -2491,7 +2630,7 @@ describe('WebDAVController verbs', () => {
expect(xml).not.toContain('Documents');
});
it('keeps a sharer’s collection a 403 for a caller with no share', async () => {
it('keeps a sharer’s collection out of sight for a caller with no share', async () => {
const owner = await makeUser();
const stranger = await makeUser();
await shareFolder(owner, await makeUser());
@@ -2500,7 +2639,8 @@ describe('WebDAVController verbs', () => {
`/${owner.username}`,
stranger.actor,
);
expect(captured.statusCode).toBe(403);
// The same 404 as a name nobody holds, so it confirms no account.
expect(captured.statusCode).toBe(404);
});
});
@@ -2670,9 +2810,9 @@ describe('WebDAVController verbs', () => {
const root = await propfind('/', holder.actor);
expect(String(root.body)).not.toContain(`/${owner.username}/`);
// And the level it used to stand in for is a plain 403 again.
// And the level it used to stand in for is out of sight again.
const owned = await propfind(`/${owner.username}`, holder.actor);
expect(owned.statusCode).toBe(403);
expect(owned.statusCode).toBe(404);
});
it('refuses writes and reads aimed at the invented levels', async () => {
@@ -2721,4 +2861,257 @@ describe('WebDAVController verbs', () => {
).toBeFalsy();
});
});
// Every check runs on the path the entry is stored under, and a refusal of
// something the caller can't see reads exactly like a path with nothing
// at it.
describe('authorization against the stored entry', () => {
it('answers a stranger alike for a missing path and a private file', async () => {
const owner = await makeUser();
const stranger = await makeUser();
await putFile(
owner.actor,
`/${owner.username}/Documents/private.txt`,
'secret',
);
const answers = [];
for (const [method, path] of [
['GET', `/${owner.username}/Documents/private.txt`],
['GET', `/${owner.username}/Documents/missing.txt`],
// A directory too: refused before the "is a directory" 400.
['GET', `/${owner.username}/Documents`],
['PROPFIND', `/${owner.username}`],
['PROPFIND', `/no-such-user-${uuidv4()}`],
['DELETE', `/${owner.username}/Documents/private.txt`],
['DELETE', `/${owner.username}/Documents/missing.txt`],
] as const) {
const captured = await dispatch({
method,
path,
actor: stranger.actor,
});
answers.push([captured.statusCode, captured.body]);
}
for (const answer of answers) {
expect(answer).toEqual([404, 'Not Found']);
}
});
// MySQL collates `fsentries.path` case-insensitively, so a request for
// `/<owner>/Public/...` can resolve a private `/<owner>/public/...`.
// The test databases compare exactly, so the lookup is made to answer
// the way MySQL would; everything past it — the ancestor chain, the
// ACL — runs for real.
it('does not pass a case variant of Public off as the public folder', async () => {
const owner = await makeUser();
const stranger = await makeUser();
await server.stores.user.update(owner.userId, {
email_confirmed: 1,
});
const privateDir = `/${owner.username}/public`;
const mkcol = await dispatch({
method: 'MKCOL',
path: privateDir,
actor: owner.actor,
});
expect(mkcol.statusCode).toBe(201);
await putFile(owner.actor, `${privateDir}/secret.txt`, 'secret');
await putFile(
owner.actor,
`/${owner.username}/Public/open.txt`,
'open',
);
const acl = server.services.acl as unknown as {
config: { enable_public_folders?: boolean };
};
const wasEnabled = acl.config.enable_public_folders;
acl.config.enable_public_folders = true;
try {
// The real Public folder is open to the stranger.
const open = await dispatch({
method: 'GET',
path: `/${owner.username}/Public/open.txt`,
actor: stranger.actor,
});
expect(open.statusCode).toBe(200);
const store = server.stores.fsEntry;
const realLookup = store.getEntryByPath.bind(store);
const asked = `/${owner.username}/Public`;
vi.spyOn(store, 'getEntryByPath').mockImplementation(
(path: string) =>
realLookup(
path === asked || path.startsWith(`${asked}/`)
? privateDir + path.slice(asked.length)
: path,
),
);
const { res, captured } = makeStreamRes();
await dispatchMiddleware(
makeReq({
method: 'GET',
path: `${asked}/secret.txt`,
actor: stranger.actor,
}),
res,
noop,
);
await new Promise((resolve) => setTimeout(resolve, 0));
expect(captured.statusCode).toBe(404);
expect(captured.text()).toBe('');
const listing = await dispatch({
method: 'PROPFIND',
path: asked,
actor: stranger.actor,
});
expect(listing.statusCode).toBe(404);
expect(String(listing.body)).not.toContain('secret.txt');
const copied = await dispatch({
method: 'COPY',
path: `${asked}/secret.txt`,
actor: stranger.actor,
headers: {
destination: `/${stranger.username}/Documents/loot.txt`,
host: 'dav.puter.localhost',
},
});
expect(copied.statusCode).toBe(404);
} finally {
vi.restoreAllMocks();
acl.config.enable_public_folders = wasEnabled;
}
expect(
await server.stores.fsEntry.getEntryByPath(
`/${stranger.username}/Documents/loot.txt`,
),
).toBeFalsy();
});
});
// A scoped token writes over DAV wherever its grant reaches, and nowhere
// else — the same reach it has over the `/fs` write routes.
describe('scoped access tokens', () => {
const mintFor = async (userId: number, permission: string) => {
const user = (await server.stores.user.getById(userId))!;
return server.services.auth.createAccessToken(makeActor({ user }), [
[permission],
]);
};
// Own address, so the per-IP auth failure bucket can't refuse these.
const asToken = (token: string, init: Parameters<typeof dispatch>[0]) =>
dispatch({
...init,
ip: '203.0.113.58',
headers: {
...init.headers,
authorization: basicAuth('-token', token),
},
});
it('writes inside its grant and is refused outside it', async () => {
const { userId, username, actor } = await makeUser();
const documents = (await server.stores.fsEntry.getEntryByPath(
`/${username}/Documents`,
))!;
const token = await mintFor(userId, `fs:${documents.uuid}:write`);
const inside = `/${username}/Documents/tok`;
expect(
(await asToken(token, { method: 'MKCOL', path: inside }))
.statusCode,
).toBe(201);
expect(
(
await asToken(token, {
method: 'PUT',
path: `${inside}/a.txt`,
headers: { 'content-length': '3' },
content: 'abc',
})
).statusCode,
).toBe(201);
expect(
(
await asToken(token, {
method: 'MOVE',
path: `${inside}/a.txt`,
headers: {
destination: `${inside}/b.txt`,
host: 'dav.puter.localhost',
},
})
).statusCode,
).toBe(201);
expect(
(
await asToken(token, {
method: 'DELETE',
path: `${inside}/b.txt`,
})
).statusCode,
).toBe(204);
await putFile(actor, `/${username}/Desktop/kept.txt`, 'kept');
for (const init of [
{ method: 'MKCOL', path: `/${username}/Desktop/intruder` },
{
method: 'PUT',
path: `/${username}/Desktop/kept.txt`,
headers: { 'content-length': '1' },
content: 'x',
},
{ method: 'DELETE', path: `/${username}/Desktop/kept.txt` },
{
method: 'COPY',
path: `/${username}/Desktop/kept.txt`,
headers: {
destination: `${inside}/kept.txt`,
host: 'dav.puter.localhost',
},
},
{ method: 'DELETE', path: `/${username}` },
]) {
const captured = await asToken(token, init);
// Outside the grant is out of sight, so it reads as missing.
expect(captured.statusCode, init.method).toBe(404);
}
expect(
await server.stores.fsEntry.getEntryByPath(
`/${username}/Desktop/intruder`,
),
).toBeFalsy();
expect(
await server.stores.fsEntry.getEntryByPath(
`/${username}/Desktop/kept.txt`,
),
).toMatchObject({ size: 4 });
});
it('refuses every write to a read-only token', async () => {
const { userId, username } = await makeUser();
const documents = (await server.stores.fsEntry.getEntryByPath(
`/${username}/Documents`,
))!;
const token = await mintFor(userId, `fs:${documents.uuid}:read`);
for (const init of [
{ method: 'MKCOL', path: `/${username}/Documents/ro` },
{
method: 'PUT',
path: `/${username}/Documents/ro.txt`,
headers: { 'content-length': '1' },
content: 'x',
},
{ method: 'DELETE', path: `/${username}/Documents` },
]) {
const captured = await asToken(token, init);
expect(captured.statusCode, init.method).toBe(403);
}
});
});
});
@@ -17,11 +17,12 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Team2faLookup } from '../../core/http/middleware/gates.js';
import { compare as bcryptCompare } from 'bcrypt';
import type { Request, Response } from 'express';
import { posix as pathPosix } from 'node:path';
import { pipeline } from 'node:stream/promises';
import { makeActor, type Actor } from '../../core/actor.js';
import type { Actor } from '../../core/actor.js';
import { HttpError } from '../../core/http/HttpError.js';
import {
assertNotSuspended,
@@ -44,12 +45,13 @@ import {
Put,
Unlock,
} from '../../core/http/decorators.js';
import type { RouteOptions } from '../../core/http/types.js';
import { PREFLIGHT_MAX_AGE, type RouteOptions } from '../../core/http/types.js';
import { verify as verifyOtp } from '../../services/auth/OTPUtil.js';
import { expandTildePath } from '../../services/fs/resolveNode.js';
import { rootShowsHome } from '../../services/fs/rootListing.js';
import {
maskEntryPath,
maskPathForRequest,
parseMaskedSharePath,
resolveSharePath,
} from '../../services/fs/sharePathMask.js';
@@ -81,6 +83,7 @@ import {
peekRateLimit,
} from '../../core/http/middleware/rateLimit.js';
import { assertActorHasCredits } from '../../services/metering/enforcement.js';
import { isTemporaryPasswordExpired } from '../../util/temporaryPassword.js';
import type { ResolvedShare } from '../../services/share/ShareService.js';
const DAV_HEADERS = {
@@ -116,9 +119,6 @@ const DAV_ROUTE_OPTIONS: RouteOptions = {
concurrent: DAV_CONCURRENT,
};
/** How long a browser may cache a DAV preflight. */
const PREFLIGHT_MAX_AGE = '86400';
/** How many shared items the virtual share collections list. */
const SHARE_LISTING_CAP = 200;
@@ -289,7 +289,7 @@ export class WebDAVController extends PuterController {
// filesystem over the `dav` subdomain, bypassing the gates.
assertNotSuspended(actor.user);
assertVerifiedAccount(actor.user);
await assertTeam2fa(actor.user, this.stores.team);
await assertTeam2fa(actor.user, this.stores.team as Team2faLookup);
// The actor was absent when the auth probe ran, so anything that reads
// it off the request or the context — shared-path masking, egress
@@ -475,28 +475,19 @@ export class WebDAVController extends PuterController {
return null;
}
// Build a session-less actor for the user
return makeActor({
user: {
id: user.id,
uuid: user.uuid,
username: user.username,
email: user.email ?? null,
suspended: user.suspended ?? false,
email_confirmed: user.email_confirmed ?? false,
requires_email_confirmation:
user.requires_email_confirmation ?? false,
// Carry the signup-time verification flags so the
// assertVerifiedAccount() gate in #dispatch can see them on
// the Basic-auth path too (the cookie / `-token` paths get
// them from AuthService#actorUserFromRow). Omitting these is
// what let phone/card-gated accounts through WebDAV.
requires_phone_verification:
user.requires_phone_verification ?? false,
requires_card_verification:
user.requires_card_verification ?? false,
},
});
// An administrator-issued temporary password that was never replaced
// dies here as it does at /login, rather than living on over DAV.
if (isTemporaryPasswordExpired(user)) {
res.status(401)
.set('WWW-Authenticate', 'Basic realm="WebDAV"')
.send('Temporary password expired');
return null;
}
// The same row→actor mapping the cookie / `-token` paths use, so the
// gates in #context see every flag (requires_password_change,
// otp_enabled, the verification flags) rather than a hand-picked few.
return this.services.auth.buildPasswordActor(user);
}
// -- OPTIONS ------------------------------------------------------
@@ -522,15 +513,15 @@ export class WebDAVController extends PuterController {
headOnly: boolean,
): Promise<void> {
const entry = await this.stores.fsEntry.getEntryByPath(davPath);
if (!entry)
throw new HttpError(404, 'Not Found', { legacyCode: 'not_found' });
if (!entry) throw davNotFound();
// Authorized before the directory check, so a caller who can't see
// the entry can't learn that it is one.
await this.#assertAccess(actor, entry.path, 'read');
if (entry.isDir)
throw new HttpError(400, 'Cannot GET a directory', {
legacyCode: 'bad_request',
});
await this.#assertRead(actor, davPath);
const modified = entry.modified ?? entry.created ?? 0;
const etag = entryEtag(entry.uuid, modified);
const size = entry.size ?? 0;
@@ -576,7 +567,7 @@ export class WebDAVController extends PuterController {
const depth = req.headers.depth ?? '1';
if (davPath === '/') {
await this.#assertRead(actor, davPath);
await this.#assertAccess(actor, davPath, 'read');
this.#sendMultistatus(res, await this.#rootPropfind(actor, depth));
return;
}
@@ -598,11 +589,9 @@ export class WebDAVController extends PuterController {
return;
}
if (!entry) {
throw new HttpError(404, 'Not Found', { legacyCode: 'not_found' });
}
if (!entry) throw davNotFound();
await this.#assertRead(actor, davPath);
await this.#assertAccess(actor, entry.path, 'read');
// `davPath` is the resolved real path; the href has to be the masked
// one, like every child below it, or the self-entry names the owner's
@@ -704,7 +693,7 @@ export class WebDAVController extends PuterController {
if (!entry?.isDir) return null;
// An owner who shared their whole home directory has a real listing
// here, and it beats the virtual one.
if (await this.#canRead(actor, davPath)) return null;
if (await this.#canAccess(actor, entry.path, 'read')) return null;
const shares = (await this.#sharedWithMe(actor)).filter(
(share) => share.owner?.username === owner,
@@ -756,17 +745,20 @@ export class WebDAVController extends PuterController {
async #proppatch(
res: Response,
{ davPath, lockToken }: DavContext,
{ actor, davPath, lockToken }: DavContext,
): Promise<void> {
if (
!(await hasWritePermission(this.clients.redis, davPath, lockToken))
) {
throw new HttpError(423, 'Locked', { legacyCode: 'conflict' });
}
const target = await this.stores.fsEntry.getEntryByPath(davPath);
if (target) await this.#assertAccess(actor, target.path, 'write');
else await this.#assertCreate(actor, davPath);
res.status(207)
.set({ 'Content-Type': 'application/xml; charset=utf-8' })
.send(
`<?xml version="1.0" encoding="utf-8"?>\n<D:multistatus xmlns:D="DAV:"><D:response><D:href>${escapeXml(encodeURI(davPath))}</D:href><D:propstat><D:prop/><D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response></D:multistatus>`,
`<?xml version="1.0" encoding="utf-8"?>\n<D:multistatus xmlns:D="DAV:"><D:response><D:href>${escapeXml(encodeURI(maskPathForRequest(davPath)))}</D:href><D:propstat><D:prop/><D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response></D:multistatus>`,
);
}
@@ -795,8 +787,7 @@ export class WebDAVController extends PuterController {
throw new HttpError(423, 'Locked', { legacyCode: 'conflict' });
}
const userId = actor.user!.id as number;
const parentPath = pathPosix.dirname(davPath);
await this.#assertWrite(actor, parentPath);
await this.#assertCreate(actor, davPath);
const existing = await this.stores.fsEntry.getEntryByPath(davPath);
if (existing)
@@ -832,8 +823,11 @@ export class WebDAVController extends PuterController {
}
const userId = actor.user!.id as number;
const parentPath = pathPosix.dirname(davPath);
await this.#assertWrite(actor, parentPath);
// Overwriting takes write on what is there, as `/fs/write` does;
// creating takes write on the parent.
const existing = await this.stores.fsEntry.getEntryByPath(davPath);
if (existing) await this.#assertAccess(actor, existing.path, 'write');
else await this.#assertCreate(actor, davPath);
const contentLength = Number(
req.headers['content-length'] ??
@@ -845,9 +839,6 @@ export class WebDAVController extends PuterController {
legacyCode: 'bad_request',
});
// Check if overwrite
const existing = await this.stores.fsEntry.getEntryByPath(davPath);
// Expect: 100-continue
if (req.headers.expect === '100-continue') {
(req.socket as { write?: (s: string) => void }).write?.(
@@ -892,11 +883,9 @@ export class WebDAVController extends PuterController {
throw new HttpError(423, 'Locked', { legacyCode: 'conflict' });
}
const userId = actor.user!.id as number;
await this.#assertWrite(actor, davPath);
const entry = await this.stores.fsEntry.getEntryByPath(davPath);
if (!entry)
throw new HttpError(404, 'Not Found', { legacyCode: 'not_found' });
if (!entry) throw davNotFound();
await this.#assertAccess(actor, entry.path, 'write');
await this.services.fs.remove(userId, { entry, recursive: true });
this.#emitGuiEvent('outer.gui.item.removed', entry);
@@ -918,14 +907,10 @@ export class WebDAVController extends PuterController {
}
const userId = actor.user!.id as number;
await this.#assertRead(actor, davPath);
await this.#assertWrite(actor, pathPosix.dirname(destPath));
const source = await this.stores.fsEntry.getEntryByPath(davPath);
if (!source)
throw new HttpError(404, 'Source not found', {
legacyCode: 'not_found',
});
if (!source) throw davNotFound();
await this.#assertAccess(actor, source.path, 'read');
await this.#assertCreate(actor, destPath);
const overwrite = req.headers.overwrite !== 'F';
const destExists = await this.stores.fsEntry.getEntryByPath(destPath);
@@ -970,14 +955,10 @@ export class WebDAVController extends PuterController {
throw new HttpError(423, 'Locked', { legacyCode: 'conflict' });
const userId = actor.user!.id as number;
await this.#assertWrite(actor, davPath);
await this.#assertWrite(actor, pathPosix.dirname(destPath));
const source = await this.stores.fsEntry.getEntryByPath(davPath);
if (!source)
throw new HttpError(404, 'Source not found', {
legacyCode: 'not_found',
});
if (!source) throw davNotFound();
await this.#assertAccess(actor, source.path, 'write');
await this.#assertCreate(actor, destPath);
const overwrite = req.headers.overwrite !== 'F';
const destExists = await this.stores.fsEntry.getEntryByPath(destPath);
@@ -1022,8 +1003,11 @@ export class WebDAVController extends PuterController {
// ACL must succeed before any lock state is touched — otherwise
// an authenticated user could lock paths they don't own (e.g. `/`)
// and block writes for everyone else.
await this.#assertWrite(actor, davPath);
// and block writes for everyone else. A path with nothing at it yet
// is a lock-null resource, locked ahead of its create.
const target = await this.stores.fsEntry.getEntryByPath(davPath);
if (target) await this.#assertAccess(actor, target.path, 'write');
else await this.#assertCreate(actor, davPath);
const timeoutSeconds = parseLockTimeout(
req.headers.timeout as string | undefined,
@@ -1156,25 +1140,46 @@ export class WebDAVController extends PuterController {
// -- ACL helpers -------------------------------------------------
async #canRead(actor: Actor, path: string): Promise<boolean> {
// Checks take the path an entry is stored under, never the request's
// spelling of it: the path lookup can match a differently-cased row (MySQL
// collates `fsentries.path` case-insensitively), while the ACL's path rules
// — home, Public — compare exactly. Only a create, whose target has no row
// yet, falls back to a requested path.
async #canAccess(
actor: Actor,
path: string,
mode: 'read' | 'write',
): Promise<boolean> {
const acl = this.services.acl;
return await acl.check(actor, acl.fsDescriptor(path), 'read');
return await acl.check(actor, acl.fsDescriptor(path), mode);
}
async #assertRead(actor: Actor, path: string): Promise<void> {
if (!(await this.#canRead(actor, path)))
throw new HttpError(403, 'Permission denied', {
legacyCode: 'permission_denied',
});
/**
* Refused the way `/fs` refuses: a caller who can't even see the target
* gets the same 404 as for a path with nothing at it, so a refusal never
* confirms that something exists.
*/
async #assertAccess(
actor: Actor,
path: string,
mode: 'read' | 'write',
): Promise<void> {
const acl = this.services.acl;
const descriptor = acl.fsDescriptor(path);
if (await acl.check(actor, descriptor, mode)) return;
const safe = await acl.getSafeAclError(actor, descriptor, mode);
if (safe.status === 404) throw davNotFound();
throw new HttpError(403, 'Permission denied', {
legacyCode: 'permission_denied',
});
}
async #assertWrite(actor: Actor, path: string): Promise<void> {
const acl = this.services.acl;
const ok = await acl.check(actor, acl.fsDescriptor(path), 'write');
if (!ok)
throw new HttpError(403, 'Permission denied', {
legacyCode: 'permission_denied',
});
/** Write on the parent of a path about to be created at `targetPath`. */
async #assertCreate(actor: Actor, targetPath: string): Promise<void> {
const parentPath = pathPosix.dirname(targetPath);
const parent = await this.stores.fsEntry.getEntryByPath(parentPath);
await this.#assertAccess(actor, parent?.path ?? parentPath, 'write');
}
// -- Event emission ----------------------------------------------
@@ -1268,6 +1273,11 @@ function sendDavError(res: Response, err: unknown): void {
res.status(500).send('Internal Server Error');
}
/** Nothing there — or nothing the caller may know is there; they read alike. */
function davNotFound(): HttpError {
return new HttpError(404, 'Not Found', { legacyCode: 'not_found' });
}
// -- XML helpers ------------------------------------------------------
function escapeXml(text: string): string {
@@ -4,7 +4,7 @@ import type { Request, Response } from 'express';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { createTestUser, setupTestServer } from '../../testUtil.js';
import type { WispController } from './WispController.js';
let server: PuterServer;
@@ -147,26 +147,114 @@ describe('WispController', () => {
});
describe('verify', () => {
it('verifies a valid authenticated-user token', async () => {
const { res: createRes, captured: createCaptured } = makeRes();
await createHandler(
makeReq({
actor: {
user: {
uuid: '00000000-0000-0000-0000-000000000001',
},
},
}),
createRes,
);
const token = (createCaptured.body as { token: string }).token;
const relayTokenFor = async (actor: unknown) => {
const { res, captured } = makeRes();
await createHandler(makeReq({ actor }), res);
return (captured.body as { token: string }).token;
};
const verifyStatus = async (token: string) => {
const { res, captured } = makeRes();
try {
await verifyHandler(makeReq({ body: { token } }), res);
return captured.statusCode;
} catch (e) {
return (e as { statusCode: number }).statusCode;
}
};
const freshUser = async () => {
const username = `wisp_${Math.random().toString(36).slice(2, 10)}`;
const creds = await createTestUser(server, {
username,
password: 'wisp-test-password',
});
const row = (await server.stores.user.getByUsername(username))!;
const actorFor = async (token: string) =>
(await server.services.auth.authenticateFromToken(token))!;
return {
row,
sessionActor: await actorFor(creds.token),
apiTokenActor: await actorFor(creds.apiToken),
apiToken: creds.apiToken,
};
};
it('verifies a token minted by a live session', async () => {
const { sessionActor } = await freshUser();
const token = await relayTokenFor(sessionActor);
const { res, captured } = makeRes();
await verifyHandler(makeReq({ body: { token } }), res);
expect(captured.statusCode).toBe(200);
const body = captured.body as { allow: boolean };
expect(body.allow).toBe(true);
expect((captured.body as { allow: boolean }).allow).toBe(true);
});
it('rejects a token once the account is suspended', async () => {
const { row, sessionActor } = await freshUser();
const token = await relayTokenFor(sessionActor);
await server.stores.user.update(row.id, { suspended: 1 });
expect(await verifyStatus(token)).toBe(403);
});
it('rejects a token once the account is pending verification', async () => {
const { row, sessionActor } = await freshUser();
const token = await relayTokenFor(sessionActor);
await server.stores.user.update(row.id, {
requires_email_confirmation: 1,
email_confirmed: 0,
});
expect(await verifyStatus(token)).toBe(403);
});
it('rejects a token whose user no longer exists', async () => {
const token = await relayTokenFor({
user: { uuid: '00000000-0000-0000-0000-000000000001' },
});
expect(await verifyStatus(token)).toBe(403);
});
it('rejects a token once its session is revoked', async () => {
const { sessionActor } = await freshUser();
const token = await relayTokenFor(sessionActor);
expect(await verifyStatus(token)).toBe(200);
await server.services.auth.revokeSession(sessionActor.session!.uid);
expect(await verifyStatus(token)).toBe(403);
});
it('binds a token minted by an access token to that token', async () => {
const { sessionActor, apiTokenActor, apiToken } = await freshUser();
const token = await relayTokenFor(apiTokenActor);
expect(await verifyStatus(token)).toBe(200);
await server.services.auth.revokeAccessToken(
sessionActor,
apiToken,
);
expect(await verifyStatus(token)).toBe(403);
});
it('applies the account checks to a token with no session claim', async () => {
const { row } = await freshUser();
const token = server.services.token.sign(
'wisp',
{ $: 'token:wisp', $v: '0.0.0', user_uid: row.uuid },
{ expiresIn: '1d' },
);
expect(await verifyStatus(token)).toBe(200);
await server.stores.user.update(row.id, { suspended: 1 });
expect(await verifyStatus(token)).toBe(403);
});
it('verifies a valid guest token', async () => {
+46 -5
View File
@@ -18,7 +18,12 @@
*/
import type { Request, Response } from 'express';
import type { Actor } from '../../core/actor.js';
import { HttpError } from '../../core/http/HttpError.js';
import {
assertNotSuspended,
assertVerifiedAccount,
} from '../../core/http/middleware/gates.js';
import type { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterController } from '../types.js';
import {
@@ -71,12 +76,14 @@ export class WispController extends PuterController {
const wispCfg = this.#wispConfig();
if (actor?.user?.uuid) {
const sessionUid = await this.#sessionUidFor(actor);
const token = this.services.token.sign(
'wisp',
{
$: 'token:wisp',
$v: '0.0.0',
user_uid: actor.user.uuid,
...(sessionUid ? { session_uid: sessionUid } : {}),
},
{ expiresIn: '1d' },
);
@@ -120,13 +127,10 @@ export class WispController extends PuterController {
});
}
// Build policy event — extensions can deny via extension.on('wisp.get-policy')
const isGuest = Boolean(decoded.guest);
let user: Record<string, unknown> | null = null;
if (!isGuest && decoded.user_uid) {
user = await this.stores.user.getByUuid(String(decoded.user_uid));
}
const user = isGuest ? null : await this.#liveUserFor(decoded);
// Build policy event — extensions can deny via extension.on('wisp.get-policy')
const event: Record<string, unknown> = {
allow: true,
policy: { allow: true },
@@ -145,6 +149,43 @@ export class WispController extends PuterController {
res.json(event.policy);
};
/**
* The session a relay token lives no longer than. An access token's is its
* own session row; the actor doesn't carry it.
*/
async #sessionUidFor(actor: Actor): Promise<string | undefined> {
if (actor.session?.uid) return actor.session.uid;
if (!actor.accessToken) return undefined;
const row = await this.stores.session.findActiveByAccessTokenUid(
actor.accessToken.uid,
);
return row?.uuid;
}
/**
* The token's user, held to the same account gates as minting. A token
* without `session_uid` (minted before the claim existed, or by a legacy
* access token with no session row) gets the account checks only.
*/
async #liveUserFor(
decoded: Record<string, unknown>,
): Promise<Record<string, unknown>> {
const forbidden = () =>
new HttpError(403, 'Forbidden', { legacyCode: 'forbidden' });
if (!decoded.user_uid) throw forbidden();
const user = await this.stores.user.getByUuid(String(decoded.user_uid));
if (!user) throw forbidden();
assertNotSuspended(user);
assertVerifiedAccount(user);
if (
decoded.session_uid &&
!(await this.stores.session.getByUuid(String(decoded.session_uid)))
) {
throw forbidden();
}
return user;
}
#wispConfig(): NonNullable<typeof this.config.wisp> {
return this.config.wisp ?? {};
}
+7
View File
@@ -67,6 +67,12 @@ export interface KnownContextFields {
* receive stops early and is never metered.
*/
abortSignal: AbortSignal;
/**
* Aborts the chat driver's upstream request. A client disconnect doesn't
* fire it: the upstream bills a request it has accepted either way, so the
* driver lets it finish and meters what it reports.
*/
upstreamSignal: AbortSignal;
/**
* When true, `ChatCompletionDriver.complete` treats a request-level
* upstream 4xx as final instead of falling back to another provider — set
@@ -84,6 +90,7 @@ const KNOWN_KEY_FLAGS: Record<keyof KnownContextFields, true> = {
requestId: true,
driverName: true,
abortSignal: true,
upstreamSignal: true,
strictUpstreamErrors: true,
};
const KNOWN_KEYS: ReadonlySet<string> = new Set(Object.keys(KNOWN_KEY_FLAGS));
+9
View File
@@ -81,6 +81,8 @@ export type LegacyErrorCodes =
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
*/
import type { PagerSeverity } from '../../types';
/** Options accepted by `HttpError`. All optional. */
export interface HttpErrorOptions {
/** Underlying error. Set as the standard `Error.cause`. */
@@ -111,6 +113,11 @@ export interface HttpErrorOptions {
* serialized to the client.
*/
noAlarm?: boolean;
/**
* Alarm at this severity even where the 4xx / `upstream_` rules would skip
* it. `noAlarm` still wins. Never serialized to the client.
*/
alarmSeverity?: PagerSeverity;
}
/**
@@ -140,6 +147,7 @@ export class HttpError extends Error {
readonly code?: string;
readonly fields?: Record<string, unknown>;
readonly noAlarm?: boolean;
readonly alarmSeverity?: PagerSeverity;
constructor(
statusCode: number,
@@ -156,6 +164,7 @@ export class HttpError extends Error {
this.code = options.code;
this.fields = options.fields;
this.noAlarm = options.noAlarm;
this.alarmSeverity = options.alarmSeverity;
}
}
+1 -1
View File
@@ -69,7 +69,7 @@ declare global {
rawBody?: Buffer;
/** Parsed user-agent, populated by the global UA-parsing middleware. */
/** User-agent, parsed from the header on first read. */
ua?: {
browser: { name?: string; version?: string; major?: string };
os: { name?: string; version?: string };
@@ -29,6 +29,7 @@ import {
createNativeAppStatic,
createUserSubdomainNotFound,
createWwwRedirect,
subdomainLabelOf,
type ResolvedClaim,
type SubdomainClaim,
} from './hostRedirects';
@@ -175,7 +176,7 @@ describe('createUserSubdomainNotFound', () => {
it('passes through reserved subdomains (api, js, native apps, etc.)', () => {
// `api`, `js`, `dav`, `docs`, `developer`, `editor`, `pdf`,
// `puter-app-icons`, `onlyoffice`, etc. all bypass.
// `puter-app-icons`, etc. all pass through.
for (const sub of ['api', 'js', 'docs', 'editor', 'puter-app-icons']) {
const { out, next } = run(
createUserSubdomainNotFound(config),
@@ -189,6 +190,17 @@ describe('createUserSubdomainNotFound', () => {
}
});
it('404s an extension-owned subdomain nobody has claimed', () => {
const { next } = run(
createUserSubdomainNotFound(config),
makeReq({
subdomains: ['com', 'puter', 'onlyoffice'],
host: 'onlyoffice.puter.com',
}),
);
expectNotFound(next);
});
it('passes through when no subdomain is present (root)', () => {
const { next } = run(
createUserSubdomainNotFound(config),
@@ -640,6 +652,21 @@ describe('createUserSubdomainNotFound with claimed subdomains', () => {
expect(args).toEqual([]);
});
it('hands a main-pipeline claim to the rest of the stack, like a reserved name', async () => {
const claims = lookup({ office: { mainPipeline: true } });
for (const [host, subdomains] of [
['office.puter.com', ['office']],
['office.puter.com:4100', ['office']],
['office.x.puter.com', ['x', 'office']],
] as const) {
const args = await runClaimed(
createUserSubdomainNotFound(config, claims),
hostReq(host, [...subdomains]),
);
expect(args, host).toEqual([]);
}
});
it('looks the claim up on every request', async () => {
const claims: Record<string, ResolvedClaim> = {};
const middleware = createUserSubdomainNotFound(config, lookup(claims));
@@ -650,6 +677,32 @@ describe('createUserSubdomainNotFound with claimed subdomains', () => {
});
});
describe('subdomainLabelOf', () => {
it('returns the single label under the domain, ignoring case and port', () => {
expect(subdomainLabelOf('shop.puter.com', 'puter.com')).toBe('shop');
expect(subdomainLabelOf('SHOP.Puter.com:4100', 'puter.com')).toBe(
'shop',
);
expect(subdomainLabelOf('shop.puter.com', 'Puter.com:443')).toBe(
'shop',
);
});
it('returns undefined for anything but exactly one label under the domain', () => {
for (const host of [
'puter.com',
'.puter.com',
'a.shop.puter.com',
'shop.puter.com.example.test',
'shopputer.com',
'shop.other.com',
]) {
expect(subdomainLabelOf(host, 'puter.com')).toBeUndefined();
}
expect(subdomainLabelOf('shop.puter.com', '')).toBeUndefined();
});
});
describe('addClaimedSubdomain', () => {
const claim: SubdomainClaim = { authorize: () => true };
@@ -677,7 +730,7 @@ describe('addClaimedSubdomain', () => {
});
it('rejects subdomains core already handles', () => {
for (const name of ['www', 'api', 'js', 'dav', 'docs', 'onlyoffice']) {
for (const name of ['www', 'api', 'js', 'dav', 'docs']) {
expect(() => addClaimedSubdomain(new Map(), name, claim)).toThrow(
/reserved/,
);
@@ -706,6 +759,26 @@ describe('addClaimedSubdomain', () => {
).toThrow(/invalid checkRoute/);
});
it('records a main-pipeline claim, and refuses one that also carries a gate', () => {
const claims = new Map<string, SubdomainClaim>();
addClaimedSubdomain(claims, 'onlyoffice', { mainPipeline: true });
expect(claims.get('onlyoffice')).toEqual({ mainPipeline: true });
expect(() =>
addClaimedSubdomain(claims, 'onlyoffice', { mainPipeline: true }),
).toThrow(/already claimed/);
for (const gate of [
{ authorize: () => true },
{ checkRoute: () => undefined },
]) {
expect(() =>
addClaimedSubdomain(new Map(), 'portal', {
mainPipeline: true,
...gate,
} as unknown as SubdomainClaim),
).toThrow(/main pipeline with a gate/);
}
});
it('rejects a claim without authorize', () => {
for (const bad of [undefined, {}, { authorize: true }]) {
expect(() =>
@@ -41,8 +41,9 @@ const NATIVE_APP_SUBDOMAINS = [
const NATIVE_APPS_WITH_DIST = new Set(['docs', 'developer']);
/**
* Subdomains that v2 serves itself. Anything NOT in this set that lives on the
* root domain is treated as a user-defined site and rejected with a 404.
* Subdomains that v2 serves itself. A root-domain subdomain neither in this set
* nor claimed by an extension is treated as a user-defined site and rejected
* with a 404.
*
* Kept as a plain Set so `has()` is O(1); order doesn't matter.
*/
@@ -55,8 +56,6 @@ const RESERVED_SUBDOMAINS = new Set<string>([
...NATIVE_APP_SUBDOMAINS,
// App-icon serving subdomain.
'puter-app-icons',
// Extension-owned subdomains.
'onlyoffice',
]);
/**
@@ -69,7 +68,8 @@ export type ClaimAuthorize = (
res: Response,
) => boolean | Promise<boolean>;
export interface SubdomainClaim {
/** A claim answered on its own router, only past `authorize`. */
export interface GatedSubdomainClaim {
authorize: ClaimAuthorize;
/**
* Vets each route declared for the claim as the server builds it; a throw
@@ -78,16 +78,39 @@ export interface SubdomainClaim {
checkRoute?: (options: RouteOptions, label: string) => void;
}
/** A claim as dispatch sees it: its gate, plus the routes served behind it. */
export interface ResolvedClaim {
authorize: ClaimAuthorize;
handler?: RequestHandler;
/** Served like core's own subdomains: CORS, auth and every route. */
export interface MainPipelineSubdomainClaim {
mainPipeline: true;
}
export type SubdomainClaim = GatedSubdomainClaim | MainPipelineSubdomainClaim;
/** A claim as dispatch sees it: a gate and the routes behind it, or neither. */
export type ResolvedClaim =
| { authorize: ClaimAuthorize; handler?: RequestHandler }
| MainPipelineSubdomainClaim;
export const isMainPipelineClaim = (
claim: SubdomainClaim | ResolvedClaim | undefined,
): claim is MainPipelineSubdomainClaim =>
(claim as { mainPipeline?: unknown } | undefined)?.mainPipeline === true;
export type ClaimLookup = (name: string) => ResolvedClaim | undefined;
const SUBDOMAIN_LABEL = /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/;
/** The label `x` when `host` is exactly `x.<domain>`, ignoring case and port. */
export const subdomainLabelOf = (
host: string,
domain: string,
): string | undefined => {
const hostName = host.toLowerCase().split(':')[0];
const base = domain.toLowerCase().split(':')[0];
if (!base || !hostName.endsWith(`.${base}`)) return undefined;
const label = hostName.slice(0, -(base.length + 1));
return label && !label.includes('.') ? label : undefined;
};
/**
* Record a claim on `<name>.<domain>`. Throws on a malformed or uppercase
* label, a subdomain core already handles, or a name that is already claimed.
@@ -106,6 +129,15 @@ export const addClaimedSubdomain = (
if (claims.has(name)) {
throw new Error(`Subdomain '${name}' is already claimed`);
}
if (isMainPipelineClaim(claim)) {
if ('authorize' in claim || 'checkRoute' in claim) {
throw new Error(
`Subdomain '${name}' claimed for the main pipeline with a gate`,
);
}
claims.set(name, { mainPipeline: true });
return;
}
if (typeof claim?.authorize !== 'function') {
throw new Error(`Subdomain '${name}' claimed without authorize`);
}
@@ -131,7 +163,7 @@ const subdomainNotFound = () =>
* unclaimed-subdomain 404, and errors reach the error handler.
*/
const dispatchClaimed = async (
claim: ResolvedClaim,
claim: Exclude<ResolvedClaim, MainPipelineSubdomainClaim>,
req: Request,
res: Response,
next: NextFunction,
@@ -190,7 +222,8 @@ export const createWwwRedirect = (config: IConfig): RequestHandler => {
* Passes through when:
*
* - No active subdomain (root)
* - Active subdomain is reserved (api, js, native apps, …)
* - Active subdomain is reserved (api, js, native apps, …) or claimed for the
* main pipeline
* - Host is on one of the hosting domains (they may nest under `config.domain`)
* - Host is a local worker host and the local worker server is on
* - Host isn't under `config.domain` (custom domains, other hosts)
@@ -223,6 +256,8 @@ export const createUserSubdomainNotFound = (
req.subdomains?.[req.subdomains.length - 1] ?? ''
).toLowerCase();
if (active === '' || RESERVED_SUBDOMAINS.has(active)) return next();
const claim = lookupClaim(active);
if (isMainPipelineClaim(claim)) return next();
const hostName = (req.headers.host ?? '').toLowerCase().split(':')[0];
if (
@@ -239,8 +274,9 @@ export const createUserSubdomainNotFound = (
// responder, body parsing, auth and every route, `subdomain: '*'` ones
// included. One label deep only: `<name>.x.<domain>` is not the claim.
const claimed =
req.subdomains?.length === 1 && hostName === `${active}.${domain}`
? lookupClaim(active)
req.subdomains?.length === 1 &&
subdomainLabelOf(hostName, domain) === active
? claim
: undefined;
if (claimed) {
void dispatchClaimed(claimed, req, res, next);
@@ -669,7 +669,7 @@ describe('resolveOwnedAppForHostedSite', () => {
const out = await resolveOwnedAppForHostedSite({
req: reqOf('beans.site.puter.localhost'),
site: { user_id: owner.id },
db: server.clients.db,
appStore: server.stores.app,
config: baseConfig(),
});
expect(out?.uid).toBe(app.uid);
@@ -684,7 +684,7 @@ describe('resolveOwnedAppForHostedSite', () => {
const out = await resolveOwnedAppForHostedSite({
req: reqOf('beans.app.puter.localhost'),
site: { user_id: owner.id },
db: server.clients.db,
appStore: server.stores.app,
config: baseConfig(),
});
expect(out?.uid).toBe(app.uid);
@@ -700,7 +700,7 @@ describe('resolveOwnedAppForHostedSite', () => {
const out = await resolveOwnedAppForHostedSite({
req: reqOf('beans.site.puter.localhost'),
site: { user_id: a.id },
db: server.clients.db,
appStore: server.stores.app,
config: baseConfig(),
});
expect(out).toBeNull();
@@ -716,7 +716,7 @@ describe('resolveOwnedAppForHostedSite', () => {
const out = await resolveOwnedAppForHostedSite({
req: reqOf('beans.site.puter.localhost'),
site: { user_id: owner.id },
db: server.clients.db,
appStore: server.stores.app,
config: baseConfig(),
});
expect(out?.uid).toBe(app.uid);
@@ -732,7 +732,7 @@ describe('resolveOwnedAppForHostedSite', () => {
const out = await resolveOwnedAppForHostedSite({
req: reqOf('beans.site.puter.localhost'),
site: { user_id: owner.id },
db: server.clients.db,
appStore: server.stores.app,
config: baseConfig(),
requirePrivate: true,
});
@@ -753,18 +753,54 @@ describe('resolveOwnedAppForHostedSite', () => {
const out = await resolveOwnedAppForHostedSite({
req: reqOf('beans.app.puter.localhost'),
site: { user_id: owner.id },
db: server.clients.db,
appStore: server.stores.app,
config: baseConfig(),
});
expect(out?.uid).toBe(app.uid);
expect(Boolean(out?.is_private)).toBe(true);
});
it('leaves columns the gate never reads, such as an inline icon, out of the match', async () => {
const owner = await makeUser();
const app = await createPrivateApp(
owner.id,
'http://beans.site.puter.localhost/',
);
await server.clients.db.write(
'UPDATE `apps` SET `icon` = ?, `description` = ? WHERE `uid` = ?',
[`data:image/png;base64,${'A'.repeat(512)}`, 'about', app.uid],
);
const out = await resolveOwnedAppForHostedSite({
req: reqOf('beans.site.puter.localhost'),
site: { user_id: owner.id },
appStore: server.stores.app,
config: baseConfig(),
});
expect(Object.keys(out!).sort()).toEqual([
'id',
'index_url',
'is_private',
'name',
'owner_user_id',
'title',
'uid',
]);
expect(out).toMatchObject({
id: app.id,
uid: app.uid,
name: `private-${app.uid}`,
title: `private-${app.uid}`,
owner_user_id: owner.id,
index_url: 'http://beans.site.puter.localhost/',
});
expect(Boolean(out!.is_private)).toBe(true);
});
it('returns null when the site has no owner', async () => {
const out = await resolveOwnedAppForHostedSite({
req: reqOf('beans.site.puter.localhost'),
site: { user_id: null },
db: server.clients.db,
appStore: server.stores.app,
config: baseConfig(),
});
expect(out).toBeNull();
@@ -774,7 +810,7 @@ describe('resolveOwnedAppForHostedSite', () => {
const out = await resolveOwnedAppForHostedSite({
req: reqOf(''),
site: { user_id: 1 },
db: server.clients.db,
appStore: server.stores.app,
config: baseConfig(),
});
expect(out).toBeNull();
@@ -18,12 +18,12 @@
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
*/
import type { AbstractDatabaseClient } from '@heyputer/backend/src/clients/database/DatabaseClient';
import type { Request } from 'express';
import {
HOSTED_ASSET_COOKIES,
type AuthService,
} from '../../../services/auth/AuthService';
import type { AppStore } from '../../../stores/app/AppStore.js';
import type { IConfig } from '../../../types';
import type { Actor } from '../../actor';
@@ -193,7 +193,7 @@ export function subdomainFromHost(
export async function resolveOwnedAppForHostedSite(opts: {
req: Request;
site: SubdomainLike;
db: AbstractDatabaseClient;
appStore: Pick<AppStore, 'listOwnedByIndexUrlCandidates'>;
config: PrivateHostingConfig;
requirePrivate?: boolean;
}): Promise<AppLike | null> {
@@ -247,15 +247,11 @@ export async function resolveOwnedAppForHostedSite(opts: {
const uniqueCandidates = [...new Set(urlCandidates)];
if (uniqueCandidates.length === 0) return null;
const privateFilter = opts.requirePrivate
? `AND \`is_private\` = ${opts.db.booleanLiteral(true)} `
: '';
const placeholders = uniqueCandidates.map(() => '?').join(', ');
// Ambiguity fails closed: a private row wins; `id` keeps it deterministic.
const orderClause = `ORDER BY CASE WHEN \`is_private\` = ${opts.db.booleanLiteral(true)} THEN 0 ELSE 1 END, \`id\``;
const rows = await opts.db.read(
`SELECT * FROM apps WHERE owner_user_id = ? ${privateFilter}AND index_url IN (${placeholders}) ${orderClause} LIMIT 2`,
[opts.site.user_id, ...uniqueCandidates],
const rows = await opts.appStore.listOwnedByIndexUrlCandidates(
opts.site.user_id,
uniqueCandidates,
{ requirePrivate: opts.requirePrivate },
);
if (rows.length === 0) return null;
if (rows.length > 1) {
@@ -17,7 +17,8 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { Writable } from 'node:stream';
import { GetObjectCommand } from '@aws-sdk/client-s3';
import { Readable, Writable } from 'node:stream';
import type { Request, Response } from 'express';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
@@ -654,6 +655,66 @@ describe('createPuterSiteMiddleware — file serving', () => {
expect(out.headers['Accept-Ranges']).toBe('bytes');
});
it('cuts the response off when storage fails mid-download', async () => {
const owner = await makeUserWithHome();
const homePath = `/${owner.username}`;
const homeEntry = await server.stores.fsEntry.getEntryByPath(homePath);
const sub = `cut-${Math.random().toString(36).slice(2, 8)}`;
await server.stores.subdomain.create({
userId: owner.id,
subdomain: sub,
rootDirId: homeEntry!.id,
});
const entry = await writeFile(
owner.id,
`${homePath}/movie.bin`,
Buffer.alloc(64, 0x42),
);
// One chunk, then the storage connection drops.
let reads = 0;
const body = new Readable({
read() {
reads += 1;
if (reads === 1) this.push('partial');
else this.destroy(new Error('aborted'));
},
});
const client = server.clients.s3.get(
server.stores.s3Object.resolveRegion(entry!.bucketRegion),
);
const send = client.send.bind(client);
const spy = vi
.spyOn(client, 'send')
.mockImplementation(((command: unknown) =>
command instanceof GetObjectCommand
? Promise.resolve({ Body: body, ContentLength: 64 })
: send(command as never)) as never);
const mw = buildMiddleware();
const { res } = makeRes();
// A real response is torn down through its socket and never emits
// 'error' itself; the stand-in Writable does, so absorb it.
res.on('error', () => undefined);
try {
await mw(
makeReq({
hostname: `${sub}.site.puter.localhost`,
path: '/movie.bin',
}),
res,
vi.fn(),
);
await new Promise<void>((resolve) => setImmediate(resolve));
} finally {
spy.mockRestore();
}
expect(res.destroyed).toBe(true);
expect(res.writableFinished).toBe(false);
expect(body.destroyed).toBe(true);
});
it("rewrites a trailing slash request to /index.html under the site's root", async () => {
const owner = await makeUserWithHome();
const homePath = `/${owner.username}`;
@@ -1217,6 +1278,50 @@ describe('createPuterSiteMiddleware — .puter_site_config', () => {
expect(piped!.equals(body)).toBe(true);
});
it('reads the config only for a path that 404s', async () => {
const { owner, homePath, sub } = await setupSiteWithConfig({
errors: { '404': { file: '/index.html', status: 200 } },
});
const body = Buffer.from('<html>spa-shell</html>');
await writeFile(owner.id, `${homePath}/index.html`, body, 'text/html');
const root = await server.stores.fsEntry.getEntryByPath(homePath);
// The loader caches whatever it reads, so an empty slot means it
// never ran.
const cached = () =>
server.clients.redis.get(`puter-site-config:${root!.id}`);
const settle = () => new Promise((resolve) => setTimeout(resolve, 10));
const mw = buildMiddleware();
const served = makeRes();
await mw(
makeReq({
hostname: `${sub}.site.puter.localhost`,
path: '/index.html',
}),
served.res,
vi.fn(),
);
await settle();
expect(served.out.statusCode).toBe(200);
expect(await cached()).toBeNull();
const missing = makeRes();
await mw(
makeReq({
hostname: `${sub}.site.puter.localhost`,
path: '/client-route',
}),
missing.res,
vi.fn(),
);
await settle();
expect(missing.out.statusCode).toBe(200);
expect((missing.out.body as Buffer | undefined)?.equals(body)).toBe(
true,
);
expect(await cached()).not.toBeNull();
});
it('serves the cached config on a second request, even when the on-disk file is removed (Redis cache holds within TTL)', async () => {
// First request seeds the cache with the parsed config. We
// then delete the underlying file and fire a second request:
@@ -1462,6 +1567,86 @@ describe('createPuterSiteMiddleware — __workers/ is never served', () => {
expect(Buffer.isBuffer(piped)).toBe(true);
expect(piped!.equals(shell)).toBe(true);
});
// SQLite compares paths bytewise; MySQL's utf8mb4_unicode_ci folds
// accents, fullwidth forms and zero-width chars. Emulate that fold.
const withCollationAlias = async (
alias: string,
real: string,
fn: () => Promise<void>,
) => {
const store = server.stores.fsEntry;
const original = store.getEntryByPath.bind(store);
const spy = vi
.spyOn(store, 'getEntryByPath')
.mockImplementation((path, options) =>
original(
path.startsWith(alias)
? real + path.slice(alias.length)
: path,
options,
),
);
try {
await fn();
} finally {
spy.mockRestore();
}
};
it('404s a collation alias of __workers/ that the database resolves to the real file', async () => {
const { homePath, sub } = await setupSiteWithWorker();
await withCollationAlias(
`${homePath}/__wörkers/matchmaking.worker.js`,
`${homePath}/__workers/matchmaking.worker.js`,
async () => {
const out = await request(
sub,
'/__w%C3%B6rkers/matchmaking.worker.js',
);
expect(out.statusCode).toBe(404);
expect(String(out.body)).not.toContain('secret');
},
);
});
it('404s a collation alias of a folder whose index.html sits in __workers/', async () => {
const { owner, homePath, sub } = await setupSiteWithWorker();
await writeFile(
owner.id,
`${homePath}/__workers/index.html`,
Buffer.from('<html>listing</html>'),
'text/html',
);
await withCollationAlias(
`${homePath}/__wörkers`,
`${homePath}/__workers`,
async () => {
const out = await request(sub, '/__w%C3%B6rkers');
expect(out.statusCode).toBe(404);
expect(String(out.body)).not.toContain('listing');
},
);
});
it('404s a collation alias of .puter_site_config', async () => {
const { owner, homePath, sub } = await setupSiteWithWorker();
await writeFile(
owner.id,
`${homePath}/.puter_site_config`,
Buffer.from(JSON.stringify({ errors: {}, marker: 'cfg-body' })),
'application/json',
);
await withCollationAlias(
`${homePath}/.püter_site_config`,
`${homePath}/.puter_site_config`,
async () => {
const out = await request(sub, '/.p%C3%BCter_site_config');
expect(out.statusCode).toBe(404);
expect(String(out.body)).not.toContain('cfg-body');
},
);
});
});
describe('createPuterSiteMiddleware — hosting CSP', () => {
@@ -1824,3 +2009,463 @@ describe('createPuterSiteMiddleware — private app sign-in token', () => {
}
});
});
// ── Owner, app and root lookups ─────────────────────────────────────
describe('createPuterSiteMiddleware — site lookups', () => {
const failure = (what: string) => new Error(`${what} lookup failed`);
const isAppQuery = (sql: unknown) => /FROM `?apps`?\b/.test(String(sql));
/** Unhandled rejections raised while `fn` runs, and shortly after. */
const unhandledDuring = async (fn: () => Promise<void>) => {
const seen: unknown[] = [];
const onUnhandled = (reason: unknown) => seen.push(reason);
process.on('unhandledRejection', onUnhandled);
try {
await fn();
await new Promise((resolve) => setTimeout(resolve, 20));
} finally {
process.off('unhandledRejection', onUnhandled);
}
return seen;
};
const makeSite = async (opts: { suspended?: boolean } = {}) => {
const owner = await makeUserWithHome();
if (opts.suspended) {
await server.stores.user.update(owner.id, { suspended: 1 });
}
const homePath = `/${owner.username}`;
const homeEntry = await server.stores.fsEntry.getEntryByPath(homePath);
const sub = `look-${Math.random().toString(36).slice(2, 8)}`;
await server.stores.subdomain.create({
userId: owner.id,
subdomain: sub,
rootDirId: homeEntry!.id,
});
await writeFile(
owner.id,
`${homePath}/index.html`,
Buffer.from('<html>hi</html>'),
'text/html',
);
return { owner, sub, rootDirId: homeEntry!.id };
};
const failRootLookup = () =>
vi
.spyOn(server.stores.fsEntry, 'getEntryById')
.mockRejectedValueOnce(failure('root'));
const failAppLookup = () => {
const db = server.clients.db;
const realRead = db.read.bind(db);
return vi
.spyOn(db, 'read')
.mockImplementation((sql, params) =>
isAppQuery(sql)
? Promise.reject(failure('app'))
: realRead(sql, params),
);
};
/** Spies on app queries and root lookups, failing both. */
const failAndCountLookups = () => {
const readSpy = failAppLookup();
const rootSpy = vi
.spyOn(server.stores.fsEntry, 'getEntryById')
.mockRejectedValue(failure('root'));
return {
appQueries: () =>
readSpy.mock.calls.filter(([sql]) => isAppQuery(sql)).length,
rootLookups: () => rootSpy.mock.calls.length,
restore: () => {
readSpy.mockRestore();
rootSpy.mockRestore();
},
};
};
it('starts the app and root lookups together once the owner is known', async () => {
const { owner, sub, rootDirId } = await makeSite();
const ownerRow = await server.stores.user.getById(owner.id);
let releaseOwner!: () => void;
const userSpy = vi
.spyOn(server.stores.user, 'getById')
.mockImplementationOnce(
() =>
new Promise((resolve) => {
releaseOwner = () => resolve(ownerRow);
}),
);
const rootSpy = vi.spyOn(server.stores.fsEntry, 'getEntryById');
const db = server.clients.db;
const realRead = db.read.bind(db);
const heldAppQueries: Array<() => void> = [];
const readSpy = vi
.spyOn(db, 'read')
.mockImplementation(async (sql, params) => {
if (isAppQuery(sql)) {
await new Promise<void>((resolve) => {
heldAppQueries.push(resolve);
});
}
return realRead(sql, params);
});
const appQueried = () =>
readSpy.mock.calls.some(([sql]) => isAppQuery(sql));
try {
const mw = buildMiddleware();
const { res, out } = makeRes();
const done = mw(
makeReq({
hostname: `${sub}.site.puter.localhost`,
path: '/index.html',
}),
res,
vi.fn(),
);
await new Promise<void>((resolve) => setImmediate(resolve));
expect(appQueried()).toBe(false);
expect(rootSpy).not.toHaveBeenCalled();
releaseOwner();
await vi.waitFor(() => expect(appQueried()).toBe(true));
expect(rootSpy).toHaveBeenCalledWith(rootDirId);
for (const release of heldAppQueries.splice(0)) release();
await done;
await new Promise<void>((resolve) => setImmediate(resolve));
expect(out.statusCode).toBe(200);
} finally {
for (const release of heldAppQueries.splice(0)) release();
userSpy.mockRestore();
rootSpy.mockRestore();
readSpy.mockRestore();
}
});
it.each([
['suspended', { suspended: true }],
['missing', {}],
] as const)(
'404s a %s owner without querying the app or the root',
async (kind, opts) => {
const { sub } = await makeSite(opts);
const userSpy =
kind === 'missing'
? vi
.spyOn(server.stores.user, 'getById')
.mockResolvedValueOnce(null)
: null;
const lookups = failAndCountLookups();
try {
let out: CapturedRes | undefined;
const unhandled = await unhandledDuring(async () => {
({ out } = await runMiddleware(
buildMiddleware(),
makeReq({ hostname: `${sub}.site.puter.localhost` }),
));
});
expect(out?.statusCode).toBe(404);
expect(out?.body).toBe('Subdomain not found');
expect(lookups.appQueries()).toBe(0);
expect(lookups.rootLookups()).toBe(0);
expect(unhandled).toEqual([]);
} finally {
userSpy?.mockRestore();
lookups.restore();
}
},
);
it("surfaces the owner lookup's failure before the others start", async () => {
const { sub } = await makeSite();
const userSpy = vi
.spyOn(server.stores.user, 'getById')
.mockRejectedValueOnce(failure('owner'));
const lookups = failAndCountLookups();
try {
const unhandled = await unhandledDuring(async () => {
await expect(
runMiddleware(
buildMiddleware(),
makeReq({ hostname: `${sub}.site.puter.localhost` }),
),
).rejects.toThrow('owner lookup failed');
});
expect(lookups.appQueries()).toBe(0);
expect(lookups.rootLookups()).toBe(0);
expect(unhandled).toEqual([]);
} finally {
userSpy.mockRestore();
lookups.restore();
}
});
it("surfaces the app lookup's failure ahead of the root's", async () => {
const { sub } = await makeSite();
const readSpy = failAppLookup();
const rootSpy = failRootLookup();
try {
const unhandled = await unhandledDuring(async () => {
await expect(
runMiddleware(
buildMiddleware(),
makeReq({ hostname: `${sub}.site.puter.localhost` }),
),
).rejects.toThrow('app lookup failed');
});
expect(unhandled).toEqual([]);
} finally {
readSpy.mockRestore();
rootSpy.mockRestore();
}
});
it('surfaces a failed root lookup once the gate lets the request through', async () => {
const { sub } = await makeSite();
const rootSpy = failRootLookup();
try {
const unhandled = await unhandledDuring(async () => {
await expect(
runMiddleware(
buildMiddleware(),
makeReq({ hostname: `${sub}.site.puter.localhost` }),
),
).rejects.toThrow('root lookup failed');
});
expect(unhandled).toEqual([]);
} finally {
rootSpy.mockRestore();
}
});
it('redirects off the private host and drops the failed root lookup', async () => {
const { sub } = await makeSite();
const rootSpy = failRootLookup();
try {
let out: CapturedRes | undefined;
const unhandled = await unhandledDuring(async () => {
({ out } = await runMiddleware(
buildMiddleware(),
makeReq({ hostname: `${sub}.app.puter.localhost` }),
));
});
expect(out?.redirected).toEqual({
status: 302,
url: `http://${sub}.site.puter.localhost/`,
});
expect(unhandled).toEqual([]);
} finally {
rootSpy.mockRestore();
}
});
it('asks a private-app visitor to sign in and drops the failed root lookup', async () => {
const { owner, sub } = await makeSite();
await server.clients.db.write(
'INSERT INTO `apps` (`uid`, `name`, `title`, `index_url`, `owner_user_id`, `is_private`) VALUES (?, ?, ?, ?, ?, ?)',
[
`app-${uuidv4()}`,
sub,
`Title ${sub}`,
`http://${sub}.app.puter.localhost/`,
owner.id,
1,
],
);
const rootSpy = failRootLookup();
try {
let out: CapturedRes | undefined;
const unhandled = await unhandledDuring(async () => {
({ out } = await runMiddleware(
buildMiddleware(),
makeReq({ hostname: `${sub}.app.puter.localhost` }),
));
});
expect(out?.statusCode).toBe(200);
expect(String(out?.body)).toContain(`Title ${sub}`);
expect(String(out?.body)).toContain('Sign in required');
expect(unhandled).toEqual([]);
} finally {
rootSpy.mockRestore();
}
});
});
// ── Private-app gate decisions ──────────────────────────────────────
describe('createPuterSiteMiddleware — private-app gate decisions', () => {
let entitled = false;
const listener = (_key: string, data: { result: { allowed: boolean } }) => {
if (entitled) data.result.allowed = true;
};
let visitor: Awaited<ReturnType<typeof makeUser>>;
let token: string;
beforeAll(async () => {
visitor = await makeUser();
({ token } = await server.services.auth.createSessionToken(
visitor as never,
));
server.clients.event.on('app.privateAccess.check', listener as never);
});
afterAll(() => {
server.clients.event.off('app.privateAccess.check', listener as never);
});
const makePrivateSite = async (opts: {
suspended?: boolean;
root?: 'home' | 'none' | 'foreign';
}) => {
const owner = await makeUserWithHome();
const sub = `gate-${Math.random().toString(36).slice(2, 8)}`;
let rootDirId: number | undefined;
if (opts.root === 'home') {
rootDirId = (await server.stores.fsEntry.getEntryByPath(
`/${owner.username}`,
))!.id;
await writeFile(
owner.id,
`/${owner.username}/index.html`,
Buffer.from('<html>gated</html>'),
'text/html',
);
} else if (opts.root === 'foreign') {
const other = await makeUserWithHome();
rootDirId = (await server.stores.fsEntry.getEntryByPath(
`/${other.username}`,
))!.id;
}
await server.stores.subdomain.create({
userId: owner.id,
subdomain: sub,
...(rootDirId !== undefined ? { rootDirId } : {}),
});
await server.clients.db.write(
'INSERT INTO `apps` (`uid`, `name`, `title`, `index_url`, `owner_user_id`, `is_private`) VALUES (?, ?, ?, ?, ?, ?)',
[
`app-${uuidv4()}`,
sub,
sub,
`http://${sub}.app.puter.localhost/`,
owner.id,
1,
],
);
if (opts.suspended) {
await server.stores.user.update(owner.id, { suspended: 1 });
}
return sub;
};
const visit = async (
sub: string,
host: 'site' | 'app',
identity: 'none' | 'cookie',
) => {
const mw = buildMiddleware({ cookie_name: 'puter_session' });
const { res, out } = makeRes();
await mw(
makeReq({
hostname: `${sub}.${host}.puter.localhost`,
path: '/index.html',
cookies: identity === 'cookie' ? { puter_session: token } : {},
}),
res,
vi.fn(),
);
await new Promise<void>((resolve) => setImmediate(resolve));
return out;
};
const decide = (out: CapturedRes, sub: string) => {
if (out.redirected) {
return `302 ${out.redirected.url.replaceAll(sub, 'SUB')}`;
}
const body = Buffer.isBuffer(out.body)
? out.body.toString()
: String(out.body);
if (body.includes('Sign in required'))
return `${out.statusCode} sign-in`;
if (body.includes('Subdomain or site is not pointing')) {
return `${out.statusCode} site-404`;
}
return `${out.statusCode} ${body}`;
};
it.each([
// A suspended owner hides the app entirely, whoever asks.
[
{ suspended: true, root: 'home' },
'app',
'cookie',
true,
'404 Subdomain not found',
],
[
{ suspended: true, root: 'home' },
'app',
'none',
false,
'404 Subdomain not found',
],
[
{ suspended: true, root: 'home' },
'site',
'cookie',
true,
'404 Subdomain not found',
],
// The public host always bounces to the private one first.
[
{ root: 'home' },
'site',
'cookie',
true,
'302 http://SUB.app.puter.localhost/index.html',
],
[
{ root: 'home' },
'site',
'none',
false,
'302 http://SUB.app.puter.localhost/index.html',
],
[{ root: 'home' }, 'app', 'none', true, '200 sign-in'],
[
{ root: 'home' },
'app',
'cookie',
false,
'302 https://puter.localhost/app/app-center/?item=SUB',
],
[{ root: 'home' }, 'app', 'cookie', true, '200 <html>gated</html>'],
// Root checks run only after entitlement passes.
[
{ root: 'none' },
'app',
'cookie',
false,
'302 https://puter.localhost/app/app-center/?item=SUB',
],
[{ root: 'none' }, 'app', 'cookie', true, '404 site-404'],
[{ root: 'foreign' }, 'app', 'none', false, '200 sign-in'],
[{ root: 'foreign' }, 'app', 'cookie', true, '404 site-404'],
] as const)(
'%o on %s host, identity %s, entitled %s → %s',
async (site, host, identity, allow, expected) => {
const sub = await makePrivateSite(site);
entitled = allow;
try {
expect(decide(await visit(sub, host, identity), sub)).toBe(
expected,
);
} finally {
entitled = false;
}
},
);
});
+59 -28
View File
@@ -26,6 +26,7 @@ import { HOSTED_ASSET_COOKIES } from '../../../services/auth/AuthService';
import { MANAGE_PERM_PREFIX } from '../../../services/permission/consts';
import type { puterStores } from '../../../stores';
import type { IConfig, LayerInstances } from '../../../types';
import { awaitLater } from '../../../util/concurrency';
import { makeActor } from '../../actor';
import {
buildAppCenterFallback,
@@ -94,6 +95,20 @@ const isWorkersSourcePath = (urlPath: string): boolean =>
.split('/')
.some((segment) => segment.toLowerCase() === WORKERS_FOLDER);
/**
* Checks the stored path of a looked-up entry. The URL-level guards are not
* enough on MySQL: `fsentries.path` is utf8mb4_unicode_ci, so `__wörkers`,
* fullwidth or zero-width spellings resolve to the real `__workers` row.
*/
const isServableEntryPath = (
entryPath: string | null | undefined,
rootPath: string,
): boolean => {
if (!entryPath?.startsWith(`${rootPath}/`)) return false;
const relPath = entryPath.slice(rootPath.length);
return !isWorkersSourcePath(relPath) && !isSiteConfigPath(relPath);
};
/** Inside some owner's top-level Trash, which is where Delete puts things. */
const isTrashedPath = (path: string): boolean =>
/^\/[^/]+\/Trash(\/|$)/u.test(path);
@@ -255,18 +270,32 @@ export const createPuterSiteMiddleware = (
return;
}
// The hosted app and the site root depend only on the site row, so
// both lookups run together. Each is awaited where it is read, so an
// early return drops the other and errors surface in the same order.
//
// The subdomain row's `associated_app_id` column is intentionally
// not consulted — it was previously user-writable without an
// ownership check (so any value there is either legacy or planted).
// Resolve "what app does this subdomain host?" directly from
// `apps.owner_user_id = site.user_id` + `index_url` match against
// the request host.
const associatedApp = (await resolveOwnedAppForHostedSite({
req,
site: { user_id: site.user_id },
db: layers.clients.db,
config: hostingCfg,
})) as AppRow | null;
const appLookup = awaitLater(
resolveOwnedAppForHostedSite({
req,
site: { user_id: site.user_id },
appStore: layers.stores.app,
config: hostingCfg,
}),
);
const rootLookup =
site.root_dir_id === null || site.root_dir_id === undefined
? null
: awaitLater(
layers.stores.fsEntry.getEntryById(site.root_dir_id),
);
const associatedApp = (await appLookup) as AppRow | null;
const isPrivateApp = Boolean(associatedApp?.is_private);
const privateApp = isPrivateApp ? associatedApp : null;
@@ -495,9 +524,7 @@ export const createPuterSiteMiddleware = (
return;
}
const rootEntry = await layers.stores.fsEntry.getEntryById(
site.root_dir_id,
);
const rootEntry = await rootLookup;
if (!rootEntry) {
res.status(404)
.type('text/html; charset=UTF-8')
@@ -579,24 +606,6 @@ export const createPuterSiteMiddleware = (
}
const filePath = rootPath + resolvedUrlPath;
// Best-effort site config load. A missing / malformed config
// never blocks the request — `loadSiteConfig` swallows all
// errors and returns null on any failure. The Redis cache
// keyed on `rootDirId` keeps the hot path off S3 for the
// common case where the same subdomain gets repeat visits.
let siteConfig: SiteConfig | null = null;
try {
siteConfig = await loadSiteConfig({
rootPath,
rootDirId: rootEntry.id,
fsEntryStore: layers.stores.fsEntry,
fsService: layers.services.fs,
cache: layers.clients.redis,
});
} catch (e) {
console.warn('[puter-site] loadSiteConfig threw', e);
}
// Hide the config file from public serving — visitors should
// see the same 404 as for any other missing path so the
// deployment shape isn't leaked.
@@ -620,6 +629,9 @@ export const createPuterSiteMiddleware = (
pathPosix.join(filePath, 'index.html'),
);
}
if (entry && !isServableEntryPath(entry.path, rootPath)) {
entry = null;
}
// Custom error page fallback. On a 404 we consult the site config
// for a rule and, if one resolves to an existing file, serve that
@@ -629,6 +641,21 @@ export const createPuterSiteMiddleware = (
// default 404 page rather than looping.
let statusOverride: number | undefined;
if (!entry || entry.isDir) {
// Best-effort site config load, only needed on a 404. A missing /
// malformed config never blocks the request — `loadSiteConfig`
// returns null on any failure, and caches by `rootDirId`.
let siteConfig: SiteConfig | null = null;
try {
siteConfig = await loadSiteConfig({
rootPath,
rootDirId: rootEntry.id,
fsEntryStore: layers.stores.fsEntry,
fsService: layers.services.fs,
cache: layers.clients.redis,
});
} catch (e) {
console.warn('[puter-site] loadSiteConfig threw', e);
}
const errorTarget = resolveErrorTarget(siteConfig, 404, rootPath);
// A config pointing `errors.404.file` into `__workers/` would
// publish worker source through the error page — refuse it the
@@ -640,7 +667,11 @@ export const createPuterSiteMiddleware = (
const candidate = await layers.stores.fsEntry.getEntryByPath(
errorTarget.absPath,
);
if (candidate && !candidate.isDir) {
if (
candidate &&
!candidate.isDir &&
isServableEntryPath(candidate.path, rootPath)
) {
entry = candidate;
statusOverride = errorTarget.status;
}
@@ -115,6 +115,31 @@ async function checkMemory(key, limit, windowMs) {
return true;
}
/**
* Spend up to `units` from one bucket at once and report how many fit: the
* many-unit form of `checkMemory`, for a caller that would otherwise loop.
*/
async function claimMemory(key, limit, windowMs, units) {
const now = Date.now();
let entry = memoryWindows.get(key);
if (!entry) {
if (memoryWindows.size >= MEMORY_MAX_KEYS) {
const oldest = memoryWindows.keys().next().value;
memoryWindows.delete(oldest);
}
entry = { ts: [], windowMs };
memoryWindows.set(key, entry);
} else {
entry.windowMs = windowMs;
}
const timestamps = entry.ts;
const cutoff = now - windowMs;
while (timestamps.length > 0 && timestamps[0] < cutoff) timestamps.shift();
const admitted = Math.max(0, Math.min(units, limit - timestamps.length));
for (let i = 0; i < admitted; i++) timestamps.push(now);
return admitted;
}
/**
* Read a bucket's state without spending from it. For gates whose budget is
* consumed by something other than the request being admitted — a failed
@@ -182,6 +207,32 @@ async function checkRedis(
return true;
}
/**
* `checkRedis` for many units in one round trip: add them all, count, and take
* back however many landed over the limit. Same add-then-count shape, so
* concurrent claims can under-admit but never over-admit.
*/
async function claimRedis(redis, key, limit, windowMs, units) {
const redisKey = `rate:${key}`;
const now = Date.now();
const members = Array.from(
{ length: units },
() => `${now}:${crypto.randomUUID()}`,
);
const results = await redis
.multi()
.zremrangebyscore(redisKey, 0, now - windowMs)
.zadd(redisKey, ...members.flatMap((member) => [now, member]))
.zcard(redisKey)
.pexpire(redisKey, windowMs)
.exec();
const count = Number(multiResult(results, 2));
const excess = Math.min(units, Math.max(0, count - limit));
if (excess > 0) await redis.zrem(redisKey, ...members.slice(-excess));
return units - excess;
}
async function peekRedis(redis, key, limit, windowMs) {
const redisKey = `rate:${key}`;
const results = await redis
@@ -218,6 +269,31 @@ async function checkKv(kv, key, limit, windowMs) {
return true;
}
// `checkKv` for many units: one read, then a row per admitted unit. Newest
// first for the same reason `checkKv` reads that way.
async function claimKv(kv, key, limit, windowMs, units) {
const prefix = `rate:${key}:`;
const { res } = await kv.list({
as: 'keys',
pattern: prefix,
limit: limit + 1,
reverse: true,
});
const keys = Array.isArray(res) ? res : (res?.items ?? []);
const admitted = Math.max(0, Math.min(units, limit - keys.length));
const now = Date.now();
await Promise.all(
Array.from({ length: admitted }, () =>
kv.set({
key: `${prefix}${now}:${crypto.randomUUID()}`,
value: 1,
expireAt: Math.ceil((now + windowMs) / 1000),
}),
),
);
return admitted;
}
// `list` filters by TTL, so a non-expired row is in-window — same read
// `checkKv` does, minus the write.
async function peekKv(kv, key, limit) {
@@ -395,6 +471,10 @@ function instrumentBackendPair(name, pair) {
withSpan('rate_limit.peek', attrs, () =>
pair.peek(key, limit, windowMs),
),
claim: (key, limit, windowMs, units) =>
withSpan('rate_limit.claim', attrs, () =>
pair.claim(key, limit, windowMs, units),
),
acquire: (key, limit) =>
withSpan('rate_limit.acquire', attrs, () =>
pair.acquire(key, limit),
@@ -405,6 +485,7 @@ function instrumentBackendPair(name, pair) {
const memoryBackendPair = instrumentBackendPair('memory', {
rate: checkMemory,
peek: peekMemory,
claim: claimMemory,
acquire: acquireMemoryConcurrent,
});
@@ -453,6 +534,8 @@ export function configureRateLimit({
checkRedis(redis, key, limit, windowMs),
peek: (key, limit, windowMs) =>
peekRedis(redis, key, limit, windowMs),
claim: (key, limit, windowMs, units) =>
claimRedis(redis, key, limit, windowMs, units),
acquire: (key, limit) => acquireRedisConcurrent(redis, key, limit),
});
}
@@ -460,6 +543,8 @@ export function configureRateLimit({
backends.kv = instrumentBackendPair('kv', {
rate: (key, limit, windowMs) => checkKv(kv, key, limit, windowMs),
peek: (key, limit) => peekKv(kv, key, limit),
claim: (key, limit, windowMs, units) =>
claimKv(kv, key, limit, windowMs, units),
acquire: (key, limit) => acquireKvConcurrent(kv, key, limit),
});
}
@@ -770,6 +855,25 @@ export async function checkRateLimit(
);
}
/**
* `checkRateLimit` for `units` hits at once, in one backend round trip.
* Resolves to how many were admitted (0..units) and spent; the rest were not
* charged. Fails open, admitting every unit, like the rest of this module.
*/
export async function claimRateLimit(key, limit, windowMs, units, backend) {
if (!(units > 0)) return 0;
const bk = resolveBackend(backend);
try {
return await bk.claim(key, limit, windowMs, units);
} catch (err) {
console.error(
'[rate-limit] imperative claim failed, failing open:',
err,
);
return units;
}
}
/**
* Charge a route-shaped rate-limit spec from inside a handler.
*
@@ -37,6 +37,7 @@ import {
acquireDriverConcurrent,
checkDriverRateLimit,
checkRateLimit,
claimRateLimit,
concurrencyGate,
consumeRouteRateLimit,
configureRateLimit,
@@ -1479,6 +1480,20 @@ describe('rate + concurrent limiting — kv backend', () => {
await Promise.all(held.map((slot) => slot.release()));
});
it('claims many units in one call, admitting only what fits', async () => {
const key = uniqueScope();
expect(await claimRateLimit(key, 5, 60_000, 3)).toBe(3);
expect(await claimRateLimit(key, 5, 60_000, 3)).toBe(2);
expect(await claimRateLimit(key, 5, 60_000, 1)).toBe(0);
});
it('claims against a full bucket as full while expired rows are unswept', async () => {
const key = uniqueScope();
await seedUnswept(`rate:${key}:`, 3);
expect(await claimRateLimit(key, 3, 60_000, 3)).toBe(3);
expect(await claimRateLimit(key, 3, 60_000, 1)).toBe(0);
});
it('holds a concurrent slot until it is released', async () => {
const req = { ip: '9.9.9.9', headers: {}, actor: undefined };
const opts = { limit: 1, backend: 'kv', scope: uniqueScope() };
@@ -1994,3 +2009,59 @@ describe('acquireConcurrent (imperative)', () => {
configureRateLimit();
});
});
describe('claimRateLimit', () => {
const uniqueKey = () => `claim-${Math.random().toString(36).slice(2, 10)}`;
describe.each([
['memory', () => configureRateLimit()],
[
'redis',
() => configureRateLimit({ default: 'redis', redis: new RedisMock() }),
],
])('%s backend', (_name, configure) => {
beforeEach(configure);
it('admits every unit while under the limit', async () => {
expect(await claimRateLimit(uniqueKey(), 5, 60_000, 3)).toBe(3);
});
it('admits exactly up to the limit and spends only that', async () => {
const key = uniqueKey();
expect(await claimRateLimit(key, 5, 60_000, 3)).toBe(3);
expect(await claimRateLimit(key, 5, 60_000, 3)).toBe(2);
// Nothing spent past the limit, so the bucket is exactly full.
expect(await checkRateLimit(key, 5, 60_000)).toBe(false);
expect(await checkRateLimit(key, 6, 60_000)).toBe(true);
});
it('admits nothing once the bucket is full', async () => {
const key = uniqueKey();
expect(await claimRateLimit(key, 2, 60_000, 2)).toBe(2);
expect(await claimRateLimit(key, 2, 60_000, 4)).toBe(0);
});
it('shares its bucket with checkRateLimit', async () => {
const key = uniqueKey();
expect(await checkRateLimit(key, 3, 60_000)).toBe(true);
expect(await claimRateLimit(key, 3, 60_000, 5)).toBe(2);
});
});
it('claims nothing for zero units', async () => {
const key = uniqueKey();
expect(await claimRateLimit(key, 1, 60_000, 0)).toBe(0);
expect(await checkRateLimit(key, 1, 60_000)).toBe(true);
});
it('fails open, admitting every unit, when the backend throws', async () => {
const redis = new RedisMock();
redis.multi = () => {
throw new Error('redis down');
};
configureRateLimit({ default: 'redis', redis });
const spy = vi.spyOn(console, 'error').mockImplementation(() => {});
expect(await claimRateLimit(uniqueKey(), 1, 60_000, 4)).toBe(4);
spy.mockRestore();
});
});
@@ -100,12 +100,14 @@ const makeUserWithPassword = async (
}> = {},
) => {
const hash = await bcrypt.hash(plainPassword, 4);
const username = extra.username ?? `up-${Math.random().toString(36).slice(2, 10)}`;
const username =
extra.username ?? `up-${Math.random().toString(36).slice(2, 10)}`;
const created = await userStore.create({
username,
uuid: uuidv4(),
password: hash,
email: extra.email !== undefined ? extra.email : `${username}@test.local`,
email:
extra.email !== undefined ? extra.email : `${username}@test.local`,
free_storage: 100 * 1024 * 1024,
requires_email_confirmation: false,
} as Parameters<UserStore['create']>[0]);
@@ -144,7 +146,7 @@ describe('userProtected — requireSessionCookie (step 1)', () => {
expect(arg).toBeUndefined();
});
it("passes through when the cookie is present and req.token is undefined (no probe-attached token)", async () => {
it('passes through when the cookie is present and req.token is undefined (no probe-attached token)', async () => {
// This covers test-only / bypass paths where the cookie is the
// only credential. The guard only fires when `req.token` is set
// AND differs from the cookie.
@@ -177,7 +179,7 @@ describe('userProtected — requireSessionCookie (step 1)', () => {
expect((arg as HttpError).statusCode).toBe(401);
});
it("honors a custom config.cookie_name", async () => {
it('honors a custom config.cookie_name', async () => {
const gates = createUserProtectedGate({
config: {
cookie_name: 'custom_session',
@@ -210,7 +212,9 @@ describe('userProtected — refreshUser (step 2)', () => {
);
const [, refreshUser] = buildGate();
const req: Partial<Request> = { actor: { user: { id: user.id, uuid: user.uuid } } };
const req: Partial<Request> = {
actor: { user: { id: user.id, uuid: user.uuid } },
};
const arg = await run(refreshUser, req);
expect(isHttpError(arg)).toBe(true);
expect((arg as HttpError).statusCode).toBe(403);
@@ -220,13 +224,15 @@ describe('userProtected — refreshUser (step 2)', () => {
it('passes through and stashes the fresh row on req.userProtected', async () => {
const user = await makeUserWithPassword('hunter2');
const [, refreshUser] = buildGate();
const req: Partial<Request> = { actor: { user: { id: user.id, uuid: user.uuid } } };
const req: Partial<Request> = {
actor: { user: { id: user.id, uuid: user.uuid } },
};
const arg = await run(refreshUser, req);
expect(arg).toBeUndefined();
expect((req as Request).userProtected?.user.uuid).toBe(user.uuid);
});
it("throws 401 when the actor lacks a user id (defensive — earlier gates should catch this)", async () => {
it('throws 401 when the actor lacks a user id (defensive — earlier gates should catch this)', async () => {
const [, refreshUser] = buildGate();
const arg = await run(refreshUser, { actor: undefined });
expect(isHttpError(arg)).toBe(true);
@@ -266,7 +272,7 @@ describe('userProtected — verifyIdentity (step 3)', () => {
expect(arg).toBeUndefined();
});
it("returns 400 password_mismatch when bcrypt says no", async () => {
it('returns 400 password_mismatch when bcrypt says no', async () => {
const user = await makeUserWithPassword('correct-horse');
const [, , verifyIdentity] = buildGate();
const arg = await run(
@@ -278,7 +284,63 @@ describe('userProtected — verifyIdentity (step 3)', () => {
expect((arg as HttpError).legacyCode).toBe('password_mismatch');
});
it("returns 403 password_required when password account submits no credentials", async () => {
it('stops guessing after ten wrong passwords on the same account', async () => {
const user = await makeUserWithPassword('correct-horse');
const [, , verifyIdentity] = buildGate();
const guess = () =>
run(
verifyIdentity,
withUser(user, { body: { password: 'wrong-guess' } }),
);
for (let i = 0; i < 10; i++) {
expect((((await guess()) as HttpError) ?? {}).legacyCode).toBe(
'password_mismatch',
);
}
const refused = (await guess()) as HttpError;
expect(refused.statusCode).toBe(429);
expect(refused.legacyCode).toBe('too_many_requests');
});
it('bounds attempts rather than rounds, so a burst cannot outrun it', async () => {
const user = await makeUserWithPassword('correct-horse');
const [, , verifyIdentity] = buildGate();
const guess = () =>
run(
verifyIdentity,
withUser(user, { body: { password: 'wrong-guess' } }),
);
// All at once: the budget is spent before any comparison runs, so
// the same ten get through whether they arrive in step or together.
const answers = (await Promise.all(
Array.from({ length: 30 }, guess),
)) as HttpError[];
const compared = answers.filter(
(a) => a.legacyCode === 'password_mismatch',
);
expect(compared).toHaveLength(10);
});
it('keeps an exhausted budget on one route away from the others', async () => {
const user = await makeUserWithPassword('correct-horse');
const [, , verifyIdentity] = buildGate();
const on = (path: string, password: string) =>
run(verifyIdentity, {
...withUser(user, { body: { password } }),
route: { path },
} as Partial<Request>);
for (let i = 0; i < 10; i++) await on('/a', 'wrong-guess');
expect(((await on('/a', 'wrong-guess')) as HttpError).statusCode).toBe(
429,
);
// A different route behind the same gate still answers.
expect(await on('/b', 'correct-horse')).toBeUndefined();
});
it('returns 403 password_required when password account submits no credentials', async () => {
// No password in body, no revalidation cookie → reject.
const user = await makeUserWithPassword('hunter2');
const [, , verifyIdentity] = buildGate();
@@ -288,7 +350,7 @@ describe('userProtected — verifyIdentity (step 3)', () => {
expect((arg as HttpError).legacyCode).toBe('password_required');
});
it("accepts a valid puter_revalidation cookie in lieu of a password", async () => {
it('accepts a valid puter_revalidation cookie in lieu of a password', async () => {
const user = await makeUserWithPassword('hunter2');
// Sign a real revalidation token via the real TokenService.
const cookieValue = tokenService.sign('oidc-state', {
@@ -347,7 +409,7 @@ describe('userProtected — verifyIdentity (step 3)', () => {
expect((arg as HttpError).legacyCode).toBe('password_required');
});
it("falls through silently when the revalidation cookie is unparseable (bad signature)", async () => {
it('falls through silently when the revalidation cookie is unparseable (bad signature)', async () => {
const user = await makeUserWithPassword('pwd');
const [, , verifyIdentity] = buildGate();
const arg = await run(
@@ -415,10 +477,12 @@ describe('userProtected — OIDC-only accounts', () => {
});
expect(isHttpError(arg)).toBe(true);
expect((arg as HttpError).statusCode).toBe(403);
expect((arg as HttpError).legacyCode).toBe('oidc_revalidation_required');
expect((arg as HttpError).legacyCode).toBe(
'oidc_revalidation_required',
);
});
it("returns oidc_revalidation_required when a password-less user submits no credentials", async () => {
it('returns oidc_revalidation_required when a password-less user submits no credentials', async () => {
const user = await makeUserWithPassword('seed-then-null');
await server.clients.db.write(
'UPDATE user SET password = NULL WHERE id = ?',
@@ -434,6 +498,8 @@ describe('userProtected — OIDC-only accounts', () => {
cookies: {},
});
expect(isHttpError(arg)).toBe(true);
expect((arg as HttpError).legacyCode).toBe('oidc_revalidation_required');
expect((arg as HttpError).legacyCode).toBe(
'oidc_revalidation_required',
);
});
});
@@ -21,6 +21,7 @@ import type { Request, RequestHandler, Response, NextFunction } from 'express';
import bcrypt from 'bcrypt';
import { isPlainUserActor } from '../../actor';
import { HttpError } from '../HttpError';
import { checkRateLimit } from './rateLimit.js';
import type { IConfig } from '../../../types';
import type { UserStore, UserRow } from '../../../stores/user/UserStore';
import type { OIDCService } from '../../../services/auth/OIDCService';
@@ -48,7 +49,9 @@ import type { TokenService } from '../../../services/auth/TokenService';
* cookie (signed via `services.token.sign('oidc-state')`) is required.
* OIDC-only accounts (no password) MUST use the revalidation cookie —
* password path returns `oidc_revalidation_required` with a `revalidate_url`
* so the GUI can open the OIDC popup.
* so the GUI can open the OIDC popup. Each password check is charged to a
* per-account, per-route hourly budget, so a route behind this gate is
* bounded whether or not it declares a limit of its own.
*/
const REVALIDATION_COOKIE_NAME = 'puter_revalidation';
@@ -138,6 +141,11 @@ async function buildRevalidateFields(
};
}
/** Password checks per account per route per hour, as the siblings bound. */
const PASSWORD_ATTEMPT_LIMIT = 10;
const PASSWORD_ATTEMPT_WINDOW_MS = 60 * 60_000;
const PASSWORD_ATTEMPT_SCOPE = 'user-protected-password';
export const createUserProtectedGate = (
deps: UserProtectedGateDeps,
options: UserProtectedGateOptions = {},
@@ -219,6 +227,23 @@ export const createUserProtectedGate = (
fields,
});
}
// Charged before the comparison, or everything already in
// flight is answered before any of it is spent. Per route, so
// one exhausted budget cannot close the other five.
const budgetKey =
`${PASSWORD_ATTEMPT_SCOPE}:${req.route?.path ?? req.path}` +
`:${user.id}`;
if (
!(await checkRateLimit(
budgetKey,
PASSWORD_ATTEMPT_LIMIT,
PASSWORD_ATTEMPT_WINDOW_MS,
))
) {
throw new HttpError(429, 'Too many requests.', {
legacyCode: 'too_many_requests',
});
}
let match = false;
try {
match = await bcrypt.compare(
@@ -0,0 +1,135 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Response } from 'express';
import http from 'node:http';
import { Socket, type AddressInfo } from 'node:net';
import { PassThrough, Readable } from 'node:stream';
import { afterEach, describe, expect, it } from 'vitest';
import { pipeToResponse } from './pipeToResponse.js';
interface Served {
res: http.ServerResponse;
piped: Promise<void>;
closeListeners: { before: number; after: number };
}
/** One request against a server that pipes `makeBody()` into its response. */
const serve = (makeBody: () => Readable) => {
let onServed!: (served: Served) => void;
const served = new Promise<Served>((resolve) => {
onServed = resolve;
});
const server = http.createServer((_req, res) => {
const before = res.listenerCount('close');
const piped = pipeToResponse(makeBody(), res as unknown as Response);
onServed({
res,
piped,
closeListeners: { before, after: res.listenerCount('close') },
});
});
servers.push(server);
const listening = new Promise<number>((resolve) => {
server.listen(0, '127.0.0.1', () =>
resolve((server.address() as AddressInfo).port),
);
});
return { served, listening };
};
const servers: http.Server[] = [];
afterEach(async () => {
for (const server of servers.splice(0)) {
server.closeAllConnections();
await new Promise((resolve) => server.close(resolve));
}
});
describe('pipeToResponse', () => {
it('streams the body byte for byte and resolves once the response is done', async () => {
const { served, listening } = serve(() =>
Readable.from([Buffer.from('a'), Buffer.from('bc')]),
);
const response = await fetch(`http://127.0.0.1:${await listening}/`);
expect(await response.text()).toBe('abc');
const { res, piped } = await served;
await piped;
expect(res.writableFinished).toBe(true);
});
it("adds no 'close' listener beyond the one pipe adds", async () => {
const { served, listening } = serve(() => Readable.from(['x']));
const response = await fetch(`http://127.0.0.1:${await listening}/`);
await response.text();
const { closeListeners } = await served;
expect(closeListeners.after - closeListeners.before).toBe(1);
});
it('releases the body when the client hangs up mid-stream', async () => {
const body = new PassThrough();
const { served, listening } = serve(() => body);
const abort = new AbortController();
const pending = fetch(`http://127.0.0.1:${await listening}/`, {
signal: abort.signal,
});
body.write('first');
const response = await pending;
await response.body!.getReader().read();
abort.abort();
const { piped } = await served;
await piped;
expect(body.destroyed).toBe(true);
});
it('cuts the response off when the body fails mid-stream', async () => {
const body = new PassThrough();
const { served, listening } = serve(() => body);
const pending = fetch(`http://127.0.0.1:${await listening}/`);
body.write('partial');
const response = await pending;
const reader = response.body!.getReader();
await reader.read();
body.destroy(new Error('upstream broke'));
const outcome = await (async () => {
try {
while (!(await reader.read()).done);
return 'ended';
} catch {
return 'cut off';
}
})();
expect(outcome).toBe('cut off');
const { res, piped } = await served;
await piped;
expect(res.destroyed).toBe(true);
});
it('releases the body at once when the response already closed', async () => {
const res = new http.ServerResponse(
new http.IncomingMessage(new Socket()),
);
res.destroy();
const body = new PassThrough();
await pipeToResponse(body, res as unknown as Response);
expect(body.destroyed).toBe(true);
});
});
+47
View File
@@ -0,0 +1,47 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Response } from 'express';
import type { Readable } from 'node:stream';
/**
* Streams `body` into the response. A body that fails mid-stream cuts the
* response off, and a response that closes first releases the body. Resolves
* once the response is done with the body; never rejects.
*/
export const pipeToResponse = (
body: Readable,
res: Response,
): Promise<void> => {
const { promise, resolve } = Promise.withResolvers<void>();
body.on('error', (err) => res.destroy(err));
if (res.destroyed) {
body.destroy();
resolve();
return promise;
}
// `pipe` unpipes once the response finishes, closes or fails, so this
// needs no 'close' listener of its own on top of the one `pipe` adds.
res.once('unpipe', () => {
if (!body.readableEnded) body.destroy();
resolve();
});
body.pipe(res);
return promise;
};
+11 -2
View File
@@ -28,7 +28,12 @@ import type { Actor } from '../actor';
* it instead of accepting any token that authenticates.
*/
export type TokenSource =
'body' | 'header' | 'x-api-key' | 'cookie' | 'query' | 'handshake';
| 'body'
| 'header'
| 'x-api-key'
| 'cookie'
| 'query'
| 'handshake';
/** Express router methods plus the WebDAV verbs some endpoints use. */
export type RouteMethod =
@@ -334,6 +339,9 @@ export const ROUTES_METADATA_KEY = '__puterRoutes' as const;
/** Internal: the property name used to stash a controller's path prefix. */
export const PREFIX_METADATA_KEY = '__puterControllerPrefix' as const;
/** How long a browser may cache a CORS preflight reply, in seconds. */
export const PREFLIGHT_MAX_AGE = '86400';
// -- Type narrowing helpers ------------------------------------------
//
// When a route declares a gate option (requireAuth, requireUserActor,
@@ -359,7 +367,8 @@ type AuthImplyingOptions =
| { requireCardVerified: true }
| {
requireAnyVerified:
readonly VerificationFactor[] | VerificationFactor[];
| readonly VerificationFactor[]
| VerificationFactor[];
}
| { requireSubscription: true | readonly string[] | string[] }
| { requireReputation: string };
+81
View File
@@ -0,0 +1,81 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* What core asks of Teams. Nothing here is an implementation: the pair is
* registered by whatever provides the feature, and a deployment that does not
* run it registers neither, so every caller treats both as absent.
*/
/** A team, as much of one as core reads back. */
export interface TeamRow {
id: number;
uid: string;
owner_user_id: number;
kind: string;
name: string | null;
handle: string | null;
deleted_at: string | null;
created_at: string;
/** Whether an app acting for a member may read the member list. */
directory_enabled: number;
/** Whether the accounts this team provisioned must hold 2FA. */
require_2fa: number;
}
/** The seat a user holds, for the gates that ask whether they hold one. */
export interface OrgSeatRow {
id: number;
user_id: number;
uuid: string;
username: string;
team_uid: string;
team_name: string | null;
owner_user_id: number;
/** Carried here so the per-request 2FA gate costs no extra read. */
require_2fa: number;
}
/** Most members one notification fans out to. */
export const NOTIFY_FANOUT_CAP = 500;
/** The reads core makes of teams it did not create. */
export interface TeamStoreContract {
getOrgSeat(userId: number): Promise<OrgSeatRow | null>;
isMember(teamUid: string, userId: number): Promise<boolean>;
getByUid(uid: string): Promise<TeamRow | null>;
getByHandle(handle: string): Promise<TeamRow | null>;
getByIdIncludingDeleted(id: number): Promise<TeamRow | null>;
getByIdsIncludingDeleted(ids: number[]): Promise<Map<number, TeamRow>>;
listGroupIdsForUser(userId: number): Promise<number[]>;
listMemberIdsByGroupId(groupId: number): Promise<number[]>;
memberIdsAmong(teamUid: string, userIds: number[]): Promise<number[]>;
countOwned(ownerUserId: number): Promise<number>;
countOwnedTeamsWithAccounts(ownerUserId: number): Promise<number>;
}
/** The calls core makes when an account it owns changes. */
export interface TeamServiceContract {
teamsAvailableTo(userId: number, email: string | null): Promise<boolean>;
captureSeatForBilling(userId: number): Promise<unknown | null>;
emitSeatDeleted(seat: unknown): void;
forgetSeat(seat: unknown): Promise<void>;
releaseUnconfirmedSeatEmail(userId: number): Promise<boolean>;
recordPasswordSelfChange(userId: number): Promise<void>;
}
@@ -0,0 +1,199 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { createServer, type Server } from 'node:http';
import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
import { clearUnhealthyRoutes } from './utils/providerHealth.js';
import { makeActor } from '../../core/actor.js';
const policyMessage =
'Policy Violation: this user has been blocked for a previous policy violation. Learn more: https://vendor.test/private';
const routes = [
'/drivers/call',
'/puterai/openai/v1/chat/completions',
'/puterai/anthropic/v1/messages',
];
const cases = [
{
name: 'policy block',
status: 403,
body: { error: { code: 403, message: policyMessage } },
expected: [400, 404, 404],
},
{
name: 'in-band policy block',
status: 200,
body: { error: { code: 403, message: policyMessage } },
expected: [400, 404, 404],
},
{
name: 'bad credentials',
status: 401,
body: { error: { code: 401, message: 'Invalid API key' } },
expected: [500, 500, 500],
},
{
name: 'generic forbidden',
status: 403,
body: { error: { code: 403, message: 'Forbidden' } },
expected: [500, 500, 500],
},
{
name: 'credit exhaustion',
status: 403,
body: {
error: {
code: 'insufficient_user_quota',
message: 'user quota is not enough',
},
},
expected: [503, 503, 503],
},
{
name: 'outage',
status: 503,
body: { error: { code: 503, message: 'service unavailable' } },
expected: [400, 400, 502],
},
{
name: 'empty choices',
status: 200,
body: { choices: [] },
expected: [400, 400, 400],
},
];
describe('model availability through real HTTP', () => {
let upstream: Server;
let env: PuterTestEnv;
let response = cases[0];
beforeAll(async () => {
upstream = createServer((req, res) => {
res.setHeader('content-type', 'application/json');
if (req.url === '/v1/models') {
const model = {
id: 'vendor/availability-test',
name: 'Availability test',
context_length: 4096,
top_provider: { max_completion_tokens: 256 },
pricing: { prompt: 0.000001, completion: 0.000001 },
};
res.end(
JSON.stringify({
data: [
model,
{
...model,
id: `${model.id}:batch`,
name: 'Batch only',
},
],
}),
);
return;
}
req.resume();
req.on('end', () => {
res.statusCode = response.status;
res.end(JSON.stringify(response.body));
});
});
await new Promise<void>((resolve) =>
upstream.listen(0, '127.0.0.1', resolve),
);
const port = (upstream.address() as { port: number }).port;
env = await setupPuterTestEnv({
providers: {
openrouter: {
apiKey: 'synthetic-key',
apiBaseUrl: `http://127.0.0.1:${port}/v1`,
},
},
meteringEnforcement: { subscriptions: false },
} as never);
}, 120000);
afterAll(async () => {
await env?.shutdown();
upstream?.closeAllConnections();
if (upstream)
await new Promise<void>((resolve) =>
upstream.close(() => resolve()),
);
});
beforeEach(() => clearUnhealthyRoutes());
it('excludes batch IDs and aliases', async () => {
const models = await env.server.drivers.aiChat.models();
expect(
models.some((m) => m.id === 'openrouter:vendor/availability-test'),
).toBe(true);
expect(JSON.stringify(models)).not.toContain(':batch');
});
for (const [index, route] of routes.entries()) {
it.each(cases)(
`$name on ${route}`,
async (current) => {
response = current;
const testUser = index === 2 ? env.users.other : env.users.user;
const args = {
model: 'openrouter:vendor/availability-test',
messages: [{ role: 'user', content: 'hi' }],
max_tokens: 16,
stream: false,
};
const res = await fetch(new URL(route, env.apiOrigin), {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${testUser.apiToken}`,
Origin: env.apiOrigin,
},
body: JSON.stringify(
route === '/drivers/call'
? {
interface: 'puter-chat-completion',
service: 'ai-chat',
method: 'complete',
args,
}
: args,
),
});
const body = await res.text();
expect(res.status, body).toBe(current.expected[index]);
if (current.name.includes('policy block')) {
expect(body).toMatch(/model.*(not available|unavailable)/i);
expect(body).not.toMatch(
/vendor\.test|policy violation|private/i,
);
}
const user = await env.server.stores.user.getByUsername(
testUser.username,
);
expect(
await env.server.services.metering.getOutstandingHolds(
makeActor({ user: user! }),
),
).toBe(0);
},
30000,
);
}
});
@@ -35,7 +35,7 @@ describe('chat cancellation over HTTP', () => {
let env: PuterTestEnv;
let restoreFetch: () => void;
let actor: Actor;
let onRequest: (res: ServerResponse) => void;
let onRequest: (res: ServerResponse, body: string) => void;
beforeAll(async () => {
upstream = createServer((req, res) => {
@@ -44,8 +44,12 @@ describe('chat cancellation over HTTP', () => {
res.end(JSON.stringify({ models: [{ name: 'cancel-test' }] }));
return;
}
req.resume();
req.once('end', () => onRequest(res));
let body = '';
req.setEncoding('utf8');
req.on('data', (chunk: string) => {
body += chunk;
});
req.once('end', () => onRequest(res, body));
});
await new Promise<void>((resolve) =>
upstream.listen(0, '127.0.0.1', resolve),
@@ -143,24 +147,137 @@ describe('chat cancellation over HTTP', () => {
);
};
const usage = { prompt_tokens: 3, completion_tokens: 5, total_tokens: 8 };
const responsesUsage = {
input_tokens: 3,
output_tokens: 5,
total_tokens: 8,
input_tokens_details: { cached_tokens: 0 },
output_tokens_details: { reasoning_tokens: 0 },
};
const response = {
id: 'resp_1',
object: 'response',
status: 'completed',
output: [
{
type: 'message',
id: 'msg_1',
role: 'assistant',
status: 'completed',
content: [
{ type: 'output_text', text: 'done', annotations: [] },
],
},
],
usage: responsesUsage,
};
const sse = (events: unknown[]) =>
events.map((event) => `data: ${JSON.stringify(event)}\n\n`).join('');
/**
* A complete upstream answer, with usage, in the shape the request asked
* for.
*/
const finishAnswer = (res: ServerResponse, body: string) => {
const stream = (JSON.parse(body) as { stream?: boolean }).stream;
const responses = res.req.url === '/v1/responses';
if (!stream) {
res.setHeader('Content-Type', 'application/json');
res.end(
JSON.stringify(
responses
? response
: {
id: 'chatcmpl_1',
object: 'chat.completion',
created: 0,
model: 'cancel-test',
choices: [
{
index: 0,
message: {
role: 'assistant',
content: 'done',
},
finish_reason: 'stop',
},
],
usage,
},
),
);
return;
}
if (!res.headersSent)
res.setHeader('Content-Type', 'text/event-stream');
res.end(
responses
? sse([
{ type: 'response.output_text.delta', delta: 'done' },
{ type: 'response.completed', response },
])
: `${sse([
{
id: 'chatcmpl_1',
object: 'chat.completion.chunk',
choices: [
{
index: 0,
delta: { content: 'done' },
finish_reason: 'stop',
},
],
},
{
id: 'chatcmpl_1',
object: 'chat.completion.chunk',
choices: [],
usage,
},
])}data: [DONE]\n\n`,
);
};
/**
* Whether a usage entry the upstream reported, not an estimate, was
* recorded.
*/
const meteredReportedUsage = (spy: ReturnType<typeof vi.spyOn>) =>
spy.mock.calls
.flatMap((call) => (call[1] as { usageType: string }[]) ?? [])
.some(
(u) =>
!u.usageType.includes('estimated_') &&
/completion|output_tokens/.test(u.usageType),
);
it.each(
routes.flatMap((route) =>
[false, true].map((stream) => ({ route, stream })),
),
)(
'cancels upstream while waiting for headers: $route (stream=$stream)',
'bills an upstream the caller left while waiting for headers: $route (stream=$stream)',
async ({ route, stream }) => {
let upstreamClosed = false;
let upstreamRes!: ServerResponse;
let upstreamBody = '';
let upstreamClosedEarly = false;
let received!: () => void;
const ready = new Promise<void>((resolve) => {
received = resolve;
});
onRequest = (res) => {
onRequest = (res, body) => {
upstreamRes = res;
upstreamBody = body;
res.once('close', () => {
upstreamClosed = true;
if (!res.writableFinished) upstreamClosedEarly = true;
});
received();
};
const metered = vi.spyOn(
env.server.services.metering,
'batchIncrementUsages',
);
const abort = new AbortController();
const request = call(route, abort.signal, stream).catch(
(error: unknown) => error,
@@ -176,36 +293,50 @@ describe('chat cancellation over HTTP', () => {
);
}),
]);
abort.abort();
expect(await request).toMatchObject({ name: 'AbortError' });
await new Promise((resolve) => setTimeout(resolve, 100));
expect(upstreamClosedEarly).toBe(false);
expect(
await env.server.services.metering.getOutstandingHolds(
actor,
),
).toBeGreaterThan(0);
abort.abort();
expect(await request).toMatchObject({ name: 'AbortError' });
await expect.poll(() => upstreamClosed).toBe(true);
finishAnswer(upstreamRes, upstreamBody);
await expect
.poll(() =>
env.server.services.metering.getOutstandingHolds(actor),
)
.toBe(0);
expect(upstreamClosedEarly).toBe(false);
expect(meteredReportedUsage(metered)).toBe(true);
} finally {
abort.abort();
metered.mockRestore();
upstream.closeAllConnections();
}
},
);
it.each(routes)(
'cancels an idle upstream after partial output: %s',
'bills an upstream the caller left after partial output: %s',
async (route) => {
let upstreamClosed = false;
onRequest = (res) => {
let upstreamRes!: ServerResponse;
let upstreamBody = '';
let upstreamClosedEarly = false;
onRequest = (res, body) => {
upstreamRes = res;
upstreamBody = body;
res.once('close', () => {
upstreamClosed = true;
if (!res.writableFinished) upstreamClosedEarly = true;
});
writeText(res);
};
const metered = vi.spyOn(
env.server.services.metering,
'batchIncrementUsages',
);
const abort = new AbortController();
try {
const response = await call(route, abort.signal);
@@ -219,14 +350,20 @@ describe('chat cancellation over HTTP', () => {
text += new TextDecoder().decode(chunk.value);
}
abort.abort();
await expect.poll(() => upstreamClosed).toBe(true);
await new Promise((resolve) => setTimeout(resolve, 100));
expect(upstreamClosedEarly).toBe(false);
finishAnswer(upstreamRes, upstreamBody);
await expect
.poll(() =>
env.server.services.metering.getOutstandingHolds(actor),
)
.toBe(0);
expect(upstreamClosedEarly).toBe(false);
expect(meteredReportedUsage(metered)).toBe(true);
} finally {
abort.abort();
metered.mockRestore();
upstream.closeAllConnections();
}
},
@@ -537,6 +537,82 @@ describe('ChatCompletionDriver exhausted-chain classification', () => {
legacyCode: 'upstream_bad_request',
fields: expect.objectContaining({ upstreamStatus: 404 }),
});
expect(err.alarmSeverity).toBeUndefined();
});
it('maps an upstream 403 model_not_found to the model-unavailable 400, not an auth failure, and redacts its URL', async () => {
const err = await errorFor({
status: 403,
message:
'The model `fake` is not available to your account. See https://vendor.test/models (request id: req-secret)',
error: {
code: 'model_not_found',
message: 'The model `fake` is not available to your account.',
},
});
expect(err).toMatchObject({
statusCode: 400,
legacyCode: 'upstream_bad_request',
fields: expect.objectContaining({ upstreamStatus: 404 }),
});
expect(err.noAlarm).toBeFalsy();
expect(err.alarmSeverity).toBe('info');
expect(err.message).toBe('Model not available: fake');
expect(JSON.stringify(attemptsFrom(err))).not.toMatch(
/http|request id/i,
);
});
it.each([
['no code', {}],
['another code', { error: { code: 'access_denied' } }],
])(
'keeps a 403 with %s as 500 upstream_auth_failed',
async (_label, extra) => {
const err = await errorFor({
status: 403,
message: 'Forbidden',
...extra,
});
expect(err).toMatchObject({
statusCode: 500,
legacyCode: 'upstream_auth_failed',
});
},
);
it('keeps a 404 whose message reads like a credential failure an auth failure', async () => {
const err = await errorFor(
Object.assign(new Error('Unauthorized'), { status: 404 }),
);
expect(err).toMatchObject({
statusCode: 500,
legacyCode: 'upstream_auth_failed',
});
});
it('keeps a 401 model_not_found an auth failure', async () => {
const err = await errorFor({
status: 401,
message: 'invalid api key',
error: { code: 'model_not_found' },
});
expect(err).toMatchObject({
statusCode: 500,
legacyCode: 'upstream_auth_failed',
});
});
it('classifies a 403 insufficient_user_quota as credit exhaustion', async () => {
const err = await errorFor({
status: 403,
message: 'user quota is not enough',
error: { code: 'insufficient_user_quota' },
});
expect(err).toMatchObject({
statusCode: 503,
legacyCode: 'upstream_credits_exhausted',
});
});
it('sniffs a status out of the message when the provider throws a bare Error', async () => {
@@ -557,6 +633,40 @@ describe('ChatCompletionDriver exhausted-chain classification', () => {
expect(err.message).toBe('unsupported parameter: top_k');
});
it('maps a gateway 4xx with no upstream error body to 400 upstream_provider_unavailable', async () => {
const err = await errorFor({
status: 400,
message: '400 bad response status code 400',
error: {
type: 'upstream_error',
message: 'bad response status code 400',
code: 400,
},
});
expect(err).toMatchObject({
statusCode: 400,
legacyCode: 'upstream_provider_unavailable',
});
expect(err.message).toBe('AI provider unavailable');
});
it('keeps a gateway 4xx that carries the vendor message as the caller’s upstream_bad_request', async () => {
const err = await errorFor({
status: 400,
message:
'400 bad response status code 400, message: max_tokens too large, body: {}',
error: {
type: 'upstream_error',
message:
'bad response status code 400, message: max_tokens too large, body: {}',
},
});
expect(err).toMatchObject({
statusCode: 400,
legacyCode: 'upstream_bad_request',
});
});
it('maps a completion with no choices to a 400, not an internal 500', async () => {
vi.spyOn(FakeChatProvider.prototype, 'complete').mockImplementation(
() =>
@@ -733,6 +843,26 @@ describe('ChatCompletionDriver cross-provider fallback', () => {
expect(line).toContain('first 422');
});
it('falls back past a gateway 4xx with no upstream error body and marks the route unhealthy', async () => {
vi.spyOn(console, 'warn').mockImplementation(() => undefined);
vi.spyOn(AzureChatProvider.prototype, 'complete').mockRejectedValue(
Object.assign(new Error('400 bad response status code 400'), {
status: 400,
error: { type: 'upstream_error' },
}),
);
vi.spyOn(OpenAiChatProvider.prototype, 'complete').mockResolvedValue({
message: { role: 'assistant', content: 'from the fallback' },
usage: { prompt_tokens: 1, completion_tokens: 1 },
} as never);
const result = (await completeShared()) as {
message: { content: string };
};
expect(result.message.content).toBe('from the fallback');
expect(isRouteUnhealthy('azure-openai', SHARED_MODEL)).toBe(true);
});
it('surfaces the primary route-level 4xx after a non-strict fallback also fails', async () => {
vi.spyOn(AzureChatProvider.prototype, 'complete').mockRejectedValue(
Object.assign(new Error('unsupported parameter: frobnicate'), {
@@ -47,7 +47,8 @@ import { ChatCompletionDriver } from './ChatCompletionDriver.js';
import { clearUnhealthyRoutes, isRouteUnhealthy } from './utils/providerHealth.js';
import { FakeChatProvider } from './providers/FakeChatProvider.js';
import type { IChatCompleteResult } from './types.js';
import { type AIChatStream, ChatStreamAbortedError } from './utils/Streaming.js';
import { SDK_TIMEOUT_MS } from './utils/sdkClient.js';
import type { AIChatStream } from './utils/Streaming.js';
let server: PuterServer;
@@ -905,7 +906,7 @@ describe('ChatCompletionDriver when the caller hangs up', () => {
).toBe(true);
});
it('stops the generation and gives the hold back', async () => {
it('keeps generating after the hang-up and bills the reported usage', async () => {
const actor = freeUser() as never;
const metering = server.services.metering;
const abort = new AbortController();
@@ -914,6 +915,7 @@ describe('ChatCompletionDriver when the caller hangs up', () => {
resume = r;
});
let providerError: unknown;
let finished = false;
vi.spyOn(FakeChatProvider.prototype, 'complete').mockResolvedValueOnce(
streamOf(async ({ chatStream }) => {
@@ -921,8 +923,13 @@ describe('ChatCompletionDriver when the caller hangs up', () => {
block.addText('first');
await paused;
try {
// An upstream still generating: keeps writing until told.
for (let i = 0; i < 1000; i++) block.addText('more');
// An upstream still generating after the caller left.
for (let i = 0; i < 1000; i++) {
await chatStream.waitForDrain();
block.addText('more');
}
chatStream.end({ input_tokens: 3, output_tokens: 1001 });
finished = true;
} catch (e) {
providerError = e;
throw e;
@@ -930,6 +937,7 @@ describe('ChatCompletionDriver when the caller hangs up', () => {
}) as never,
);
const metered = vi.spyOn(metering, 'batchIncrementUsages');
const events = vi.spyOn(server.clients.event, 'emit');
const { stream } = (await withTestActor(() => {
Context.set('abortSignal', abort.signal);
@@ -950,15 +958,120 @@ describe('ChatCompletionDriver when the caller hangs up', () => {
}
await new Promise((r) => setTimeout(r, 20));
expect(providerError).toBeInstanceOf(ChatStreamAbortedError);
// Nobody is listening, so no error line is written for the abort.
expect(body).not.toContain('"type":"error"');
// The hold goes back once the usage write lands.
expect(providerError).toBeUndefined();
expect(finished).toBe(true);
// Nothing reaches a caller who has gone, including the usage line.
expect(body).not.toContain('more');
expect(body).not.toContain('"type":"usage"');
await vi.waitFor(async () =>
expect(await metering.getOutstandingHolds(actor)).toBe(0),
);
expect(
meteredUsages(metered).some((u) =>
u.usageType.includes('estimated_'),
),
).toBe(false);
expect(
events.mock.calls.some(
([key, data]) =>
key === 'ai.prompt.cost-calculated' &&
(data as { usage?: { output_tokens?: number } }).usage
?.output_tokens === 1001,
),
).toBe(true);
});
it('bills a stream whose caller left before the first token', async () => {
const actor = freeUser() as never;
const abort = new AbortController();
vi.spyOn(FakeChatProvider.prototype, 'complete').mockImplementationOnce(
async () => {
// The caller hangs up while the upstream is still answering.
abort.abort();
return streamOf(async ({ chatStream }) => {
chatStream
.message()
.contentBlock({ type: 'text' })
.addText('answer');
chatStream.end({ input_tokens: 3, output_tokens: 2 });
});
},
);
const events = vi.spyOn(server.clients.event, 'emit');
const { stream } = (await withTestActor(() => {
Context.set('abortSignal', abort.signal);
return driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'go' }],
stream: true,
});
}, actor)) as unknown as { stream: Readable };
await drain(stream);
expect(
events.mock.calls.some(
([key, data]) =>
key === 'ai.prompt.cost-calculated' &&
(data as { usage?: { output_tokens?: number } }).usage
?.output_tokens === 2,
),
).toBe(true);
await vi.waitFor(async () =>
expect(await server.services.metering.getOutstandingHolds(actor)).toBe(0),
);
});
it('cuts off an upstream that outlives its caller by the SDK timeout', async () => {
const actor = freeUser() as never;
const metering = server.services.metering;
const abort = new AbortController();
let upstreamAborted = false;
vi.spyOn(FakeChatProvider.prototype, 'complete').mockResolvedValueOnce(
streamOf(async ({ chatStream }) => {
chatStream
.message()
.contentBlock({ type: 'text' })
.addText('x'.repeat(400));
const signal = Context.get('upstreamSignal')!;
// A stalled upstream: no more chunks until it's cut off.
await new Promise((_resolve, reject) => {
signal.addEventListener('abort', () => {
upstreamAborted = true;
reject(signal.reason);
});
});
}) as never,
);
const metered = vi.spyOn(metering, 'batchIncrementUsages');
const timers = vi.spyOn(globalThis, 'setTimeout');
const { stream } = (await withTestActor(() => {
Context.set('abortSignal', abort.signal);
return driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'go' }],
stream: true,
});
}, actor)) as unknown as { stream: Readable };
abort.abort();
const limit = timers.mock.calls.find(
([, ms]) => ms === SDK_TIMEOUT_MS,
);
expect(limit).toBeDefined();
await new Promise((r) => setTimeout(r, 20));
expect(upstreamAborted).toBe(false);
(limit![0] as () => void)();
await drain(stream);
expect(upstreamAborted).toBe(true);
expect(
meteredUsages(metered).some((u) =>
u.usageType.endsWith('estimated_output_tokens'),
),
).toBe(true);
await vi.waitFor(async () =>
expect(await metering.getOutstandingHolds(actor)).toBe(0),
);
// What was generated before the hang-up is still charged.
expect(metered).toHaveBeenCalled();
});
});
@@ -993,3 +1106,129 @@ describe('stream pump termination', () => {
expect(warning).toHaveBeenCalledWith('Chat stream cleanup failed:', 'cleanup failed');
});
});
// A list-price change can land while a completion is running. The attempt is
// held, metered and estimated at the rates in force when it started, so the
// hold covers what is charged and the charge doesn't jump mid-request.
describe('ChatCompletionDriver pricing across a scheduled rate change', () => {
const REPRICED_MODEL = {
...PRICED_MODEL,
id: 'repriced',
pricingOverrides: [
{
startsAt: '2026-12-01T00:00:00Z',
costs: { input_tokens: 2000, output_tokens: 4000 },
},
],
};
const BEFORE = new Date('2026-11-30T23:59:59.000Z');
const AFTER = new Date('2026-12-01T00:00:05.000Z');
beforeEach(async () => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValue([
REPRICED_MODEL,
PRICED_MODEL,
] as never);
driver = await makeDriver();
vi.useFakeTimers({ toFake: ['Date'], now: BEFORE });
});
const listedOutputRate = async (id: string) => ({
listed: (await driver.models()).find((m) => m.id === id)!.costs
.output_tokens,
reported: driver
.getReportedCosts()
.find((c) => String(c.usageType).endsWith(`:${id}:output_tokens`))!
.ucentsPerUnit,
});
it('lists and reports the rates in force, before and after the change', async () => {
expect(await listedOutputRate('repriced')).toEqual({
listed: 2000,
reported: 2000,
});
vi.setSystemTime(AFTER);
expect(await listedOutputRate('repriced')).toEqual({
listed: 4000,
reported: 4000,
});
// A model with no scheduled change lists its own rates either way.
expect(await listedOutputRate('priced')).toEqual({
listed: 2000,
reported: 2000,
});
});
afterEach(() => {
vi.useRealTimers();
});
it('holds, meters and estimates an in-flight stream at its start instant', async () => {
const actor = freeUser() as never;
const reserve = vi.spyOn(server.services.metering, 'reserveCredits');
const metered = vi.spyOn(
server.services.metering,
'batchIncrementUsages',
);
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockImplementationOnce(async () => {
vi.setSystemTime(AFTER);
return streamOf(async ({ chatStream }) => {
chatStream
.message()
.contentBlock({ type: 'text' })
.addText('x'.repeat(4000));
chatStream.end(undefined as never);
}) as never;
});
const result = (await withTestActor(
() =>
driver.complete({
model: 'repriced',
messages: [{ role: 'user', content: 'go' }],
stream: true,
max_tokens: 10,
}),
actor,
)) as unknown as { stream: Readable };
await drain(result.stream);
expect(completeSpy.mock.calls[0]![2]).toEqual(BEFORE);
const [, held] = reserve.mock.calls[0]!;
expect(held).toBeGreaterThanOrEqual(10 * 2000);
expect(held).toBeLessThan(10 * 4000);
const output = meteredUsages(metered).find((u) =>
u.usageType.endsWith('estimated_output_tokens'),
)!;
// 4000 characters at 4 chars/token, at the launch output rate.
expect(output.costOverride).toBe(1000 * 2000);
});
it('prices an attempt that starts after the change at the new rates', async () => {
vi.setSystemTime(AFTER);
const reserve = vi.spyOn(server.services.metering, 'reserveCredits');
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockResolvedValueOnce({
message: { role: 'assistant', content: 'ok' },
usage: { input_tokens: 1, output_tokens: 1 },
finish_reason: 'stop',
} as never);
await withTestActor(
() =>
driver.complete({
model: 'repriced',
messages: [{ role: 'user', content: 'go' }],
max_tokens: 10,
}),
freeUser() as never,
);
expect(completeSpy.mock.calls[0]![2]).toEqual(AFTER);
const [, held] = reserve.mock.calls[0]!;
expect(held).toBeGreaterThanOrEqual(10 * 4000);
});
});
@@ -40,6 +40,12 @@ import {
type MockInstance,
} from 'vitest';
import type { Request } from 'express';
import {
renderAnthropicError,
renderOpenAIError,
} from '../../controllers/puterai/wireErrors.js';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
@@ -52,6 +58,7 @@ import {
markRouteUnhealthy,
} from './utils/providerHealth.js';
import { InfronProvider } from './providers/infron/InfronProvider.js';
import { UpstreamDeadlineError } from '../util/upstreamErrors.js';
// -- OpenAI SDK mock ------------------------------------------------
// Gemini reaches Google through `new openai.OpenAI()` (default export) and
@@ -143,6 +150,7 @@ const OPENROUTER_CATALOG = [
// also the alias the Meta provider publishes — the two have to land in
// one bucket rather than becoming separate models.
'meta/muse-spark-1.2',
'openai/o1-pro',
].map((id) => ({
id,
name: `${id} (via OpenRouter)`,
@@ -475,6 +483,366 @@ describe('ChatCompletionDriver muse spark routing', () => {
});
});
describe('ChatCompletionDriver model-unavailable routes', () => {
// Shaped like the OpenAI SDK's APIError for an OpenAI-convention body.
const modelNotFound = () =>
Object.assign(
new Error('403 The model is not available to your account.'),
{
status: 403,
error: {
code: 'model_not_found',
message: 'The model is not available to your account.',
},
},
);
const statusError = (status: number, message: string) =>
Object.assign(new Error(message), { status });
const reply = {
choices: [
{
message: { role: 'assistant', content: 'hi' },
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 1, completion_tokens: 1 },
};
const complete = (model: string, provider?: string) =>
withTestActor(() =>
driver.complete({
model,
...(provider ? { provider } : {}),
messages: [{ role: 'user', content: 'hi' }],
}),
).catch((e: unknown) => e as HttpError);
const statusesOf = (err: HttpError) =>
(
err as unknown as { fields: { attempts: { status?: number }[] } }
).fields.attempts.map((a) => a.status);
it('falls back past a gated route and marks only that route', async () => {
createMock
.mockRejectedValueOnce(modelNotFound())
.mockResolvedValueOnce(reply);
await expect(complete('deepseek-v4-pro')).resolves.toMatchObject({
message: { content: 'hi' },
});
expect(isRouteUnhealthy('deepseek', 'deepseek-v4-pro')).toBe(true);
expect(
isRouteUnhealthy('infron', 'infron:deepseek/deepseek-v4-pro'),
).toBe(false);
});
it('keeps serving a model from its vendor after the gateway route is gated', async () => {
createMock
.mockRejectedValueOnce(modelNotFound())
.mockResolvedValueOnce(reply)
.mockResolvedValueOnce(reply);
await expect(
complete('muse-spark-1.2', 'openrouter'),
).resolves.toMatchObject({ message: { content: 'hi' } });
expect(createMock.mock.calls.at(-1)![0].model).toBe('muse-spark-1.2');
expect(
isRouteUnhealthy('openrouter', 'openrouter:meta/muse-spark-1.2'),
).toBe(true);
expect(isRouteUnhealthy('meta', 'muse-spark-1.2')).toBe(false);
await expect(complete('meta/muse-spark-1.2')).resolves.toMatchObject({
message: { content: 'hi' },
});
expect(createMock.mock.calls.at(-1)![0].model).toBe('muse-spark-1.2');
});
it('keeps a gated route marked for as long as bad credentials', async () => {
vi.useFakeTimers({ toFake: ['Date'] });
const start = Date.now();
try {
createMock
.mockRejectedValueOnce(modelNotFound())
.mockResolvedValueOnce(reply);
await complete('deepseek-v4-pro');
vi.setSystemTime(start + 61 * 1000);
expect(isRouteUnhealthy('deepseek', 'deepseek-v4-pro')).toBe(true);
vi.setSystemTime(start + (10 * 60 + 1) * 1000);
expect(isRouteUnhealthy('deepseek', 'deepseek-v4-pro')).toBe(false);
} finally {
vi.useRealTimers();
}
});
it('tells the caller the model is unavailable when every route is gated or missing', async () => {
createMock
.mockRejectedValueOnce(modelNotFound())
.mockRejectedValue(statusError(404, 'model not found'));
const err = await complete('deepseek-v4-pro');
expect(err).toMatchObject({
statusCode: 400,
legacyCode: 'upstream_bad_request',
fields: expect.objectContaining({ upstreamStatus: 404 }),
});
expect(statusesOf(err)).toEqual([403, 404, 404]);
});
it('still pages when a gated route sits beside a bad credential', async () => {
createMock
.mockRejectedValueOnce(modelNotFound())
.mockRejectedValue(statusError(401, 'invalid api key'));
const err = await complete('deepseek-v4-pro');
expect(err).toMatchObject({
statusCode: 500,
legacyCode: 'upstream_auth_failed',
});
expect(statusesOf(err)).toEqual([403, 401, 401]);
});
it('classifies a gated route beside an outage as a mixed upstream failure', async () => {
createMock
.mockRejectedValueOnce(modelNotFound())
.mockRejectedValue(statusError(503, 'service unavailable'));
const err = await complete('deepseek-v4-pro');
expect(err).toMatchObject({
statusCode: 400,
legacyCode: 'upstream_failed',
});
expect(statusesOf(err)).toEqual([403, 503, 503]);
});
});
describe('ChatCompletionDriver OpenRouter account policy block', () => {
const POLICY_MESSAGE =
'Policy Violation: this user has been blocked for a previous policy violation. Learn more: https://openrouter.ai/docs/policy (request id: req-secret)';
// Shaped like the OpenAI SDK's APIError for OpenRouter's error body.
const policyBlock = () =>
Object.assign(new Error(`403 ${POLICY_MESSAGE}`), {
status: 403,
code: 403,
error: { code: 403, message: POLICY_MESSAGE },
});
const reply = {
choices: [
{
message: { role: 'assistant', content: 'hi' },
finish_reason: 'stop',
},
],
usage: { prompt_tokens: 1, completion_tokens: 1 },
};
const complete = (model: string, provider?: string) =>
withTestActor(() =>
driver.complete({
model,
...(provider ? { provider } : {}),
messages: [{ role: 'user', content: 'hi' }],
}),
).catch((e: unknown) => e as HttpError);
const attemptsOf = (err: HttpError) =>
(
err as unknown as {
fields: {
attempts: {
provider: string;
status?: number;
code?: unknown;
}[];
};
}
).fields.attempts;
it.each(['openai/o1-pro', 'openrouter:openai/o1-pro', 'o1-pro'])(
'tells a caller asking for %s that the model is unavailable',
async (model) => {
createMock.mockRejectedValueOnce(policyBlock());
const err = await complete(model);
expect(err).toMatchObject({
statusCode: 400,
legacyCode: 'upstream_bad_request',
message: 'Model not available: openrouter:openai/o1-pro',
fields: expect.objectContaining({ upstreamStatus: 404 }),
});
expect(JSON.stringify(err.fields)).not.toMatch(
/policy|http|request id/i,
);
},
);
it('treats the in-band policy block of a 200 with no choices the same way', async () => {
createMock.mockResolvedValueOnce({
error: { code: 403, message: POLICY_MESSAGE },
});
const err = await complete('openai/o1-pro');
expect(err).toMatchObject({
statusCode: 400,
legacyCode: 'upstream_bad_request',
fields: expect.objectContaining({ upstreamStatus: 404 }),
});
expect(attemptsOf(err)).toEqual([
expect.objectContaining({
provider: 'openrouter',
status: 403,
code: 'model_not_found',
}),
]);
});
it('keeps a no-choice completion without the policy block a request-level 400', async () => {
createMock.mockResolvedValueOnce({});
const err = await complete('openai/o1-pro');
expect(err).toMatchObject({
statusCode: 400,
legacyCode: 'upstream_bad_request',
message: 'no completion was returned',
});
expect(err.fields?.upstreamStatus).toBe(400);
expect(isRouteUnhealthy('openrouter', 'openrouter:openai/o1-pro')).toBe(
false,
);
});
it('falls back from a policy-blocked gateway route to the vendor, under the Anthropic route too', async () => {
for (const strict of [false, true]) {
clearUnhealthyRoutes();
createMock
.mockRejectedValueOnce(policyBlock())
.mockResolvedValueOnce(reply);
const res = await withTestActor(() => {
if (strict) Context.set('strictUpstreamErrors', true);
return driver.complete({
model: 'muse-spark-1.2',
provider: 'openrouter',
messages: [{ role: 'user', content: 'hi' }],
});
});
expect(res).toMatchObject({ message: { content: 'hi' } });
expect(createMock.mock.calls.at(-1)![0].model).toBe(
'muse-spark-1.2',
);
expect(
isRouteUnhealthy(
'openrouter',
'openrouter:meta/muse-spark-1.2',
),
).toBe(true);
expect(isRouteUnhealthy('meta', 'muse-spark-1.2')).toBe(false);
}
});
it('renders a gated route beside a missing one as not found on both wire formats', async () => {
createMock
.mockRejectedValueOnce(policyBlock())
.mockRejectedValueOnce(
Object.assign(new Error('model not found'), { status: 404 }),
);
const err = await complete('muse-spark-1.2', 'openrouter');
expect(attemptsOf(err).map((a) => a.status)).toEqual([403, 404]);
const anthropic = renderAnthropicError(err, {} as Request);
const openai = renderOpenAIError(err, {} as Request);
expect(anthropic.status).toBe(404);
expect(anthropic.body).toMatchObject({
error: {
type: 'not_found_error',
message: 'Model not available: openrouter:meta/muse-spark-1.2',
},
});
expect(openai.status).toBe(404);
expect(openai.body).toMatchObject({
error: {
type: 'not_found_error',
message: 'Model not available: openrouter:meta/muse-spark-1.2',
},
});
expect(JSON.stringify([anthropic.body, openai.body])).not.toMatch(
/policy|http|request id/i,
);
});
it('leaves the same body from another provider an auth failure', async () => {
createMock.mockRejectedValue(policyBlock());
const err = await complete('muse-spark-1.2');
expect(err).toMatchObject({
statusCode: 500,
legacyCode: 'upstream_auth_failed',
});
expect(attemptsOf(err)).toEqual([
expect.objectContaining({
provider: 'meta',
status: 403,
code: 403,
}),
expect.objectContaining({
provider: 'openrouter',
status: 403,
code: 'model_not_found',
}),
]);
});
it.each([
[
'a generic 403',
{ status: 403, error: { code: 403, message: 'Forbidden' } },
500,
'upstream_auth_failed',
],
[
'a 401',
{ status: 401, error: { code: 401, message: 'User not found.' } },
500,
'upstream_auth_failed',
],
[
'a 402',
{
status: 402,
error: { code: 402, message: 'Insufficient credits' },
},
503,
'upstream_credits_exhausted',
],
[
'a 503',
{ status: 503, error: { code: 503, message: POLICY_MESSAGE } },
400,
'upstream_provider_unavailable',
],
])(
'keeps %s from OpenRouter classified as before',
async (_label, shape, statusCode, legacyCode) => {
createMock.mockRejectedValueOnce(
Object.assign(new Error(shape.error.message), shape),
);
const err = await complete('openai/o1-pro');
expect(err).toMatchObject({ statusCode, legacyCode });
},
);
});
describe('ChatCompletionDriver duplicate-model fallback', () => {
// deepseek-v4-pro is served directly by DeepSeek, by Infron, and twice by
// OpenRouter (two upstream orgs) — four routes in one bucket.
@@ -712,6 +1080,25 @@ describe('ChatCompletionDriver unhealthy-route skipping', () => {
expect(isRouteUnhealthy('deepseek', 'deepseek-v4-pro')).toBe(true);
});
it('returns 504 for a request that outlived its deadline and keeps the route', async () => {
createMock.mockRejectedValue(
new UpstreamDeadlineError('Gemini request timed out'),
);
const err = await withTestActor(() =>
driver.complete({
model: 'deepseek-v4-pro',
messages: [{ role: 'user', content: 'hi' }],
}),
).catch((e: unknown) => e as HttpError);
expect(err).toMatchObject({
statusCode: 504,
legacyCode: 'upstream_timeout',
});
expect(isRouteUnhealthy('deepseek', 'deepseek-v4-pro')).toBe(false);
});
describe('how long a failed route stays marked', () => {
let start: number;
beforeEach(() => {
@@ -20,19 +20,17 @@
/**
* Offline unit tests for ChatCompletionDriver.
*
* Boots a real PuterServer (in-memory sqlite + dynamo + s3 + mock
* redis) so the driver runs against the live `EventClient`,
* `MeteringService`, stores, and `FSService`. The driver is
* exercised through its public `complete()` interface against the
* always-available `FakeChatProvider` — no real upstream API keys
* are needed because no `config.providers.*` entries are set, so
* Boots a real PuterServer (in-memory sqlite + dynamo + s3 + mock redis) so the
* driver runs against the live `EventClient`, `MeteringService`, stores, and
* `FSService`. The driver is exercised through its public `complete()`
* interface against the always-available `FakeChatProvider` — no real upstream
* API keys are needed because no `config.providers.*` entries are set, so
* `fake-chat` is the only provider registered.
*
* Provider behaviour (request shape, streaming dialect) is covered
* separately by each provider's test file. This file pins down
* driver-level behaviour: auth gate, model resolution, validation
* event routing, credit/quota gates, max_tokens cap, fallback,
* event emission and cost calculation.
* Provider behaviour (request shape, streaming dialect) is covered separately
* by each provider's test file. This file pins down driver-level behaviour:
* auth gate, model resolution, validation event routing, credit/quota gates,
* max_tokens cap, fallback, event emission and cost calculation.
*/
import type { Readable } from 'node:stream';
import {
@@ -256,7 +254,10 @@ describe('ChatCompletionDriver model map', () => {
describe('ChatCompletionDriver.complete auth and model resolution', () => {
it('uses a free fake completion in test mode without taking a credit hold', async () => {
const creditGate = vi.spyOn(server.services.metering, 'getUsageHeadroom');
const creditGate = vi.spyOn(
server.services.metering,
'getUsageHeadroom',
);
const complete = vi.spyOn(FakeChatProvider.prototype, 'complete');
const result = await withTestActor(() =>
@@ -290,10 +291,15 @@ describe('ChatCompletionDriver.complete auth and model resolution', () => {
);
expect(complete.mock.calls[0]![0].model).toBe('fake');
const streamResult = result as unknown as { dataType: string; stream: Readable };
const streamResult = result as unknown as {
dataType: string;
stream: Readable;
};
expect(streamResult.dataType).toBe('stream');
const chunks = await collectStream(streamResult.stream);
expect(chunks).toContainEqual(expect.objectContaining({ type: 'text' }));
expect(chunks).toContainEqual(
expect.objectContaining({ type: 'text' }),
);
});
it('throws 401 when no actor is in context', async () => {
@@ -368,7 +374,10 @@ describe('ChatCompletionDriver.complete auth and model resolution', () => {
const completeSpy = vi.spyOn(FakeChatProvider.prototype, 'complete');
completeSpy.mockResolvedValueOnce({
message: { role: 'assistant', content: [{ type: 'text', text: 'ok' }] },
message: {
role: 'assistant',
content: [{ type: 'text', text: 'ok' }],
},
usage: {},
finish_reason: 'stop',
} as never);
@@ -617,7 +626,11 @@ describe('ChatCompletionDriver.complete events and cost emission', () => {
id: 'priced',
aliases: [],
costs_currency: 'usd-cents',
costs: { input_tokens: 1000, output_tokens: 2000, web_search_requests: 1_000_000 },
costs: {
input_tokens: 1000,
output_tokens: 2000,
web_search_requests: 1_000_000,
},
long_context_pricing: {
threshold: 5,
input_multiplier: 2,
@@ -633,7 +646,11 @@ describe('ChatCompletionDriver.complete events and cost emission', () => {
role: 'assistant',
content: [{ type: 'text', text: 'ok' }],
},
usage: { input_tokens: 10, output_tokens: 7, web_search_requests: 1 },
usage: {
input_tokens: 10,
output_tokens: 7,
web_search_requests: 1,
},
finish_reason: 'stop',
} as never);
@@ -751,7 +768,10 @@ describe('ChatCompletionDriver.complete validation event routing', () => {
describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
it('throws 402 `insufficient_funds` when the actor has no remaining credits', async () => {
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 0, held: 0 });
vi.spyOn(
server.services.metering,
'getUsageHeadroom',
).mockResolvedValue({ balance: 0, held: 0 });
await expect(
withTestActor(() =>
@@ -786,7 +806,10 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
// Approx input cost is tiny (very short prompt), so allowed
// output ≈ 100_000 / 2000 = 50 tokens, comfortably below the
// caller's 10_000 ceiling.
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 100_000, held: 0 });
vi.spyOn(
server.services.metering,
'getUsageHeadroom',
).mockResolvedValue({ balance: 100_000, held: 0 });
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
@@ -813,62 +836,102 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
expect(passed.max_tokens!).toBeGreaterThan(0);
});
it.each(['multipliers', 'catalog overrides'])('caps `max_tokens` at the conditional output rate using %s', async pricingKind => {
it.each(['multipliers', 'catalog overrides'])(
'caps `max_tokens` at the conditional output rate using %s',
async (pricingKind) => {
vi.spyOn(
FakeChatProvider.prototype,
'models',
).mockResolvedValueOnce([
{
id: 'capme',
aliases: [],
costs_currency: 'usd-cents',
costs: { input_tokens: 1000, output_tokens: 2000 },
...(pricingKind === 'multipliers'
? {
long_context_pricing: {
threshold: 10,
input_multiplier: 2,
output_multiplier: 1.5,
},
}
: {
pricingOverrides: [
{
minPromptTokens: 10,
costs: {
input_tokens: 2000,
output_tokens: 3000,
},
},
],
}),
max_tokens: 8192,
},
]);
const d = await makeDriver();
// A ~100-token prompt is past the threshold. 1_000_000 microcents at
// 2000 * 1.5 per output token leaves at most 333 output tokens before
// the prompt is paid for; the standard rate would allow ~450 after it.
vi.spyOn(
server.services.metering,
'getUsageHeadroom',
).mockResolvedValue({ balance: 1_000_000, held: 0 });
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockResolvedValueOnce({
message: {
role: 'assistant',
content: [{ type: 'text', text: 'ok' }],
},
usage: { input_tokens: 1, output_tokens: 1 },
finish_reason: 'stop',
} as never);
await withTestActor(() =>
d.complete({
model: 'capme',
messages: [{ role: 'user', content: 'x'.repeat(400) }],
max_tokens: 10_000,
}),
);
const passed = completeSpy.mock.calls[0]![0] as ICompleteArguments;
expect(passed.max_tokens!).toBeLessThanOrEqual(333);
expect(passed.max_tokens!).toBeGreaterThan(0);
},
);
it('reserves a catalog fixed request fee before calling the provider', async () => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValueOnce([
{
id: 'capme',
aliases: [],
id: 'request-fee',
costs_currency: 'usd-cents',
costs: { input_tokens: 1000, output_tokens: 2000 },
...(pricingKind === 'multipliers' ? { long_context_pricing: {
threshold: 10,
input_multiplier: 2,
output_multiplier: 1.5,
} } : { pricingOverrides: [{ minPromptTokens: 10, costs: { input_tokens: 2000, output_tokens: 3000 } }] }),
max_tokens: 8192,
max_tokens: 1000,
costs: {
input_tokens: 1,
output_tokens: 1,
request: 1_000_001,
},
},
]);
const d = await makeDriver();
// A ~100-token prompt is past the threshold. 1_000_000 microcents at
// 2000 * 1.5 per output token leaves at most 333 output tokens before
// the prompt is paid for; the standard rate would allow ~450 after it.
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 1_000_000, held: 0 });
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockResolvedValueOnce({
message: {
role: 'assistant',
content: [{ type: 'text', text: 'ok' }],
},
usage: { input_tokens: 1, output_tokens: 1 },
finish_reason: 'stop',
} as never);
await withTestActor(() =>
d.complete({
model: 'capme',
messages: [{ role: 'user', content: 'x'.repeat(400) }],
max_tokens: 10_000,
}),
);
const passed = completeSpy.mock.calls[0]![0] as ICompleteArguments;
expect(passed.max_tokens!).toBeLessThanOrEqual(333);
expect(passed.max_tokens!).toBeGreaterThan(0);
});
it('reserves a catalog fixed request fee before calling the provider', async () => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValueOnce([{
id: 'request-fee', costs_currency: 'usd-cents', max_tokens: 1000,
costs: { input_tokens: 1, output_tokens: 1, request: 1_000_001 },
}]);
const d = await makeDriver();
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 1_000_000, held: 0 });
vi.spyOn(
server.services.metering,
'getUsageHeadroom',
).mockResolvedValue({ balance: 1_000_000, held: 0 });
const completeSpy = vi.spyOn(FakeChatProvider.prototype, 'complete');
await expect(withTestActor(() => d.complete({ model: 'request-fee', messages: [{ role: 'user', content: 'hi' }] })))
.rejects.toMatchObject({ statusCode: 402 });
await expect(
withTestActor(() =>
d.complete({
model: 'request-fee',
messages: [{ role: 'user', content: 'hi' }],
}),
),
).rejects.toMatchObject({ statusCode: 402 });
expect(completeSpy).not.toHaveBeenCalled();
});
@@ -889,7 +952,10 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
// Pass the cheap pre-flight check but leave a balance too small to
// afford a single 2000-microcent output token.
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 100, held: 0 });
vi.spyOn(
server.services.metering,
'getUsageHeadroom',
).mockResolvedValue({ balance: 100, held: 0 });
const completeSpy = vi.spyOn(FakeChatProvider.prototype, 'complete');
@@ -927,23 +993,31 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
// extraCost alone (50_000) eats most of a 100_000-microcent balance;
// the fast cost keys (half the standard rate) are what is left to
// afford output with.
(FakeChatProvider.prototype as unknown as {
requestPricing: NonNullable<IChatProvider['requestPricing']>;
}).requestPricing = () => ({
(
FakeChatProvider.prototype as unknown as {
requestPricing: NonNullable<IChatProvider['requestPricing']>;
}
).requestPricing = () => ({
inputKey: 'fast_input_tokens',
outputKey: 'fast_output_tokens',
extraCost: 50_000,
});
try {
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({
vi.spyOn(
server.services.metering,
'getUsageHeadroom',
).mockResolvedValue({
balance: 100_000,
held: 0,
});
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockResolvedValueOnce({
message: { role: 'assistant', content: [{ type: 'text', text: 'ok' }] },
message: {
role: 'assistant',
content: [{ type: 'text', text: 'ok' }],
},
usage: { input_tokens: 1, output_tokens: 1 },
finish_reason: 'stop',
} as never);
@@ -963,12 +1037,13 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
expect(passed.max_tokens!).toBeLessThanOrEqual(50);
expect(passed.max_tokens!).toBeGreaterThan(30);
} finally {
delete (FakeChatProvider.prototype as unknown as Record<string, unknown>)
.requestPricing;
delete (
FakeChatProvider.prototype as unknown as Record<string, unknown>
).requestPricing;
}
});
it('rejects with 402 when a low balance cannot cover a tool\'s extraCost reservation', async () => {
it("caps `max_tokens` and the hold at the provider's requestPricing maxTokens", async () => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValueOnce([
{
id: 'capme',
@@ -980,16 +1055,87 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
]);
const d = await makeDriver();
(FakeChatProvider.prototype as unknown as {
requestPricing: NonNullable<IChatProvider['requestPricing']>;
}).requestPricing = () => ({ extraCost: 1_000_000 });
(
FakeChatProvider.prototype as unknown as {
requestPricing: NonNullable<IChatProvider['requestPricing']>;
}
).requestPricing = () => ({ maxTokens: 100 });
try {
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({
vi.spyOn(
server.services.metering,
'getUsageHeadroom',
).mockResolvedValue({
balance: 100_000_000,
held: 0,
});
const reserve = vi.spyOn(
server.services.metering,
'reserveCredits',
);
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockResolvedValueOnce({
message: {
role: 'assistant',
content: [{ type: 'text', text: 'ok' }],
},
usage: { input_tokens: 1, output_tokens: 1 },
finish_reason: 'stop',
} as never);
await withTestActor(() =>
d.complete({
model: 'capme',
messages: [{ role: 'user', content: 'hi' }],
max_tokens: 10_000,
}),
);
const passed = completeSpy.mock.calls[0]![0] as ICompleteArguments;
expect(passed.max_tokens).toBe(100);
// The balance and the request both allow more; the hold is the
// tiny prompt plus 100 output tokens at 2000.
const [, held] = reserve.mock.calls[0]!;
expect(held).toBeGreaterThanOrEqual(100 * 2000);
expect(held).toBeLessThan(101 * 2000);
} finally {
delete (
FakeChatProvider.prototype as unknown as Record<string, unknown>
).requestPricing;
}
});
it("rejects with 402 when a low balance cannot cover a tool's extraCost reservation", async () => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValueOnce([
{
id: 'capme',
aliases: [],
costs_currency: 'usd-cents',
costs: { input_tokens: 1000, output_tokens: 2000 },
max_tokens: 8192,
},
]);
const d = await makeDriver();
(
FakeChatProvider.prototype as unknown as {
requestPricing: NonNullable<IChatProvider['requestPricing']>;
}
).requestPricing = () => ({ extraCost: 1_000_000 });
try {
vi.spyOn(
server.services.metering,
'getUsageHeadroom',
).mockResolvedValue({
balance: 100,
held: 0,
});
const completeSpy = vi.spyOn(FakeChatProvider.prototype, 'complete');
const completeSpy = vi.spyOn(
FakeChatProvider.prototype,
'complete',
);
await expect(
withTestActor(() =>
@@ -999,11 +1145,15 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
tools: [{ type: 'web_search' }],
}),
),
).rejects.toMatchObject({ statusCode: 402, legacyCode: 'insufficient_funds' });
).rejects.toMatchObject({
statusCode: 402,
legacyCode: 'insufficient_funds',
});
expect(completeSpy).not.toHaveBeenCalled();
} finally {
delete (FakeChatProvider.prototype as unknown as Record<string, unknown>)
.requestPricing;
delete (
FakeChatProvider.prototype as unknown as Record<string, unknown>
).requestPricing;
}
});
@@ -1011,14 +1161,20 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
// unexempted `web_search_preview` became an unnamed function tool, which
// OpenAI rejects and the provider's search pricing no longer recognizes.
it('passes web_search_preview through to the provider unchanged', async () => {
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({
vi.spyOn(
server.services.metering,
'getUsageHeadroom',
).mockResolvedValue({
balance: 100_000_000,
held: 0,
});
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockResolvedValueOnce({
message: { role: 'assistant', content: [{ type: 'text', text: 'ok' }] },
message: {
role: 'assistant',
content: [{ type: 'text', text: 'ok' }],
},
usage: { input_tokens: 1, output_tokens: 1 },
finish_reason: 'stop',
} as never);
@@ -1103,7 +1259,10 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
},
]);
const d = await makeDriver();
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 1_000_000, held: 0 });
vi.spyOn(
server.services.metering,
'getUsageHeadroom',
).mockResolvedValue({ balance: 1_000_000, held: 0 });
vi.spyOn(
server.services.metering,
'getActorSubscription',
@@ -1134,7 +1293,10 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
},
]);
const d = await makeDriver();
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 1_000_000, held: 0 });
vi.spyOn(
server.services.metering,
'getUsageHeadroom',
).mockResolvedValue({ balance: 1_000_000, held: 0 });
vi.spyOn(
server.services.metering,
'getActorSubscription',
@@ -1173,7 +1335,10 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
]);
const d = await makeDriver();
// Plenty of credits so the credit gate doesn't intercept first.
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 1_000_000, held: 0 });
vi.spyOn(
server.services.metering,
'getUsageHeadroom',
).mockResolvedValue({ balance: 1_000_000, held: 0 });
await expect(
withTestActor(() =>
@@ -1303,9 +1468,10 @@ describe('ChatCompletionDriver Claude usage.thinking_tokens', () => {
const d = await makeClaudeDriver();
// Affords exactly 300 output tokens at 1000 µ¢ each: above the real
// 233 output tokens, below 233 + 110 thinking_tokens (343).
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue(
{ balance: 300_000, held: 0 },
);
vi.spyOn(
server.services.metering,
'getUsageHeadroom',
).mockResolvedValue({ balance: 300_000, held: 0 });
vi.spyOn(ClaudeProvider.prototype, 'complete').mockResolvedValueOnce({
message: { role: 'assistant', content: 'hi' },
finish_reason: 'stop',
@@ -1323,8 +1489,7 @@ describe('ChatCompletionDriver Claude usage.thinking_tokens', () => {
messages: [{ role: 'user', content: 'hi' }],
});
return Context.get('driverMetadata') as
| Record<string, unknown>
| undefined;
Record<string, unknown> | undefined;
});
expect(metadata?.usage_limited).toBeUndefined();
@@ -1370,7 +1535,10 @@ describe('ChatCompletionDriver.complete normalization', () => {
])(
'normalizes a raw wire-form tool_choice (%j) before it reaches the provider',
async (wireToolChoice, normalized) => {
const completeSpy = vi.spyOn(FakeChatProvider.prototype, 'complete');
const completeSpy = vi.spyOn(
FakeChatProvider.prototype,
'complete',
);
completeSpy.mockResolvedValueOnce({
message: { role: 'assistant', content: 'ok' },
usage: {},
@@ -1672,9 +1840,7 @@ describe('ChatCompletionDriver.complete OpenAI-shape normalization', () => {
content_type: 'application/x-ndjson',
});
// Drain so the fake provider's populator finishes cleanly.
await collectStream(
(res as unknown as { stream: Readable }).stream,
);
await collectStream((res as unknown as { stream: Readable }).stream);
});
it('a blocked prompt rerouted to fake-chat keeps its historical native shape', async () => {
@@ -1733,8 +1899,11 @@ describe('ChatCompletionDriver.complete fallback and error envelope', () => {
legacyCode: 'upstream_timeout',
message: 'AI provider timed out',
});
const attempts = (caught as unknown as { fields: { attempts: { timedOut?: boolean }[] } })
.fields.attempts;
const attempts = (
caught as unknown as {
fields: { attempts: { timedOut?: boolean }[] };
}
).fields.attempts;
expect(attempts).toHaveLength(1);
expect(attempts[0].timedOut).toBe(true);
});
@@ -1829,8 +1998,9 @@ describe('ChatCompletionDriver.complete fallback and error envelope', () => {
expect(caught).toBeInstanceOf(HttpError);
expect(caught!.statusCode).toBe(500);
expect(caught!.message).toBe('All providers failed');
const attempts = (caught as unknown as { fields: { attempts: unknown[] } })
.fields.attempts;
const attempts = (
caught as unknown as { fields: { attempts: unknown[] } }
).fields.attempts;
expect(Array.isArray(attempts)).toBe(true);
// No second provider serves `fake`, so we record exactly one attempt
// before falling out of the loop.
@@ -2020,8 +2190,9 @@ describe('ChatCompletionDriver.countTokens', () => {
it('prefers the provider’s own countTokens hook when it exists', async () => {
const hook = vi.fn().mockResolvedValue(42);
(FakeChatProvider.prototype as unknown as { countTokens?: unknown }).countTokens =
hook;
(
FakeChatProvider.prototype as unknown as { countTokens?: unknown }
).countTokens = hook;
try {
const result = await driver[COUNT_TOKENS]({
model: 'fake',
@@ -2034,8 +2205,11 @@ describe('ChatCompletionDriver.countTokens', () => {
provider: 'fake-chat',
});
} finally {
delete (FakeChatProvider.prototype as unknown as { countTokens?: unknown })
.countTokens;
delete (
FakeChatProvider.prototype as unknown as {
countTokens?: unknown;
}
).countTokens;
}
});
@@ -42,6 +42,7 @@ import {
isUpstreamConnectionError,
isUpstreamTimeoutError,
sanitizeUpstreamMessage,
UpstreamDeadlineError,
} from '../util/upstreamErrors.js';
import { AlibabaProvider } from './providers/alibaba/AlibabaProvider.js';
import { AzureChatProvider } from './providers/azure/AzureChatProvider.js';
@@ -102,6 +103,7 @@ import {
costKeys,
costsForRequest,
isFreeModel,
listedCosts,
longContextMultipliers,
trackedOutputTokens,
usageCost,
@@ -114,6 +116,7 @@ import {
UNHEALTHY_TTL_SEC,
} from './utils/providerHealth.js';
import { CATALOG_TTL_MS } from './utils/cachedRemoteCatalog.js';
import { SDK_TIMEOUT_MS } from './utils/sdkClient.js';
import { fromFinishReason } from './utils/stopReason.js';
import { AIChatStream } from './utils/Streaming.js';
import {
@@ -309,13 +312,29 @@ const isRateLimit = (a: ProviderAttempt) =>
a.status === 429 ||
/rate[\s_-]?limit|too many requests|quota/i.test(a.error);
const isModelAccessDenied = (a: ProviderAttempt) =>
a.status === 403 && a.code === 'model_not_found';
const isModelUnavailable = (a: ProviderAttempt) =>
a.status === 404 || isModelAccessDenied(a);
const isAuthFailure = (a: ProviderAttempt) =>
a.status === 401 ||
a.status === 403 ||
/unauthorized|forbidden|invalid api key/i.test(a.error);
!isModelAccessDenied(a) &&
(a.status === 401 ||
a.status === 403 ||
/unauthorized|forbidden|invalid api key/i.test(a.error));
/**
* A new-api gateway's (Infron's) wrapper for an upstream that answered without
* a JSON error body. It mirrors the upstream's status, so the 4xx says nothing
* about the caller's request; a real vendor rejection carries its own message.
*/
const isOpaqueGatewayFailure = (a: ProviderAttempt) =>
/bad response status code \d+(?:,\s*body:|$)/i.test(a.error);
const isUpstream5xx = (a: ProviderAttempt) =>
(a.status !== undefined && a.status >= 500) ||
isOpaqueGatewayFailure(a) ||
/provider returned error|internal server error|service unavailable|bad gateway/i.test(
a.error,
);
@@ -333,7 +352,7 @@ const isUpstream5xx = (a: ProviderAttempt) =>
*/
const isRouteLevelFailure = (a: ProviderAttempt) =>
a.status === undefined ||
a.status === 404 ||
isModelUnavailable(a) ||
a.status === 408 ||
isCreditExhaustion(a) ||
isRateLimit(a) ||
@@ -373,8 +392,6 @@ const requestLevelError = (a: ProviderAttempt): HttpError =>
const isOverloaded = (a: ProviderAttempt) =>
a.status === 529 || a.upstreamType === 'overloaded_error';
const isModelUnavailable = (a: ProviderAttempt) => a.status === 404;
/**
* The HTTP status an in-band mid-stream error chunk reports — a stream
* populator failure never goes through `classifyAttempts` (there is no fallback
@@ -404,11 +421,12 @@ const streamErrorStatus = (a: ProviderAttempt): number => {
* - All credit-exhausted → 503 `upstream_credits_exhausted` (alerted)
* - All rate-limited → 429 `upstream_rate_limited` (alerted, unless every attempt
* was on a free model — see `allModelsFree`)
* - All auth failures → 500 `upstream_auth_failed` (paged: our config)
* - All auth failures, or any alongside 403 `model_not_found` → 500
* `upstream_auth_failed` (paged: our config)
* - All overloaded (529 / `overloaded_error`) → 400
* `upstream_provider_unavailable`, `upstreamStatus: 529` (no page)
* - All model-not-found (404) → 400 `upstream_bad_request`, `upstreamStatus: 404`
* (no page)
* - All model-not-found (404, or 403 `model_not_found`) → 400
* `upstream_bad_request`, `upstreamStatus: 404` (no page)
* - All upstream 5xx or dropped connections → 400 `upstream_provider_unavailable`
* (no page)
* - All upstream 4xx (other) → 400 `upstream_bad_request` (no page)
@@ -449,7 +467,10 @@ const classifyAttempts = (
noAlarm: allModelsFree,
});
}
if (attempts.every(isAuthFailure)) {
if (
attempts.some(isAuthFailure) &&
attempts.every((a) => isAuthFailure(a) || isModelAccessDenied(a))
) {
return new HttpError(500, 'AI provider authentication failed', {
legacyCode: 'upstream_auth_failed',
fields,
@@ -462,9 +483,16 @@ const classifyAttempts = (
});
}
if (attempts.every(isModelUnavailable)) {
return new HttpError(400, attempts[0].error, {
const accessDenied = attempts.some(isModelAccessDenied);
const message = accessDenied
? `Model not available: ${attempts[0].model}`
: attempts[0].error;
return new HttpError(400, message, {
legacyCode: 'upstream_bad_request',
fields: { ...fields, upstreamStatus: 404 },
// Our provider account lost access to the model; low alarm so
// someone notices.
...(accessDenied ? { alarmSeverity: 'info' as const } : {}),
});
}
if (
@@ -600,6 +628,10 @@ export class ChatCompletionDriver extends PuterDriver {
seen.add(model.id);
return true;
})
.map((model) => {
const costs = listedCosts(model);
return costs === model.costs ? model : { ...model, costs };
})
.sort((a, b) => {
if (a.provider === b.provider) return a.id.localeCompare(b.id);
return a.provider!.localeCompare(b.provider!);
@@ -619,7 +651,7 @@ export class ChatCompletionDriver extends PuterDriver {
if (seen.has(key)) continue;
seen.add(key);
for (const [costKey, raw] of Object.entries(
model.costs ?? {},
model.costs ? listedCosts(model) : {},
)) {
// `tokens` is a scale descriptor ("costs expressed per N
// tokens"), not a real per-operation cost — skip it.
@@ -649,6 +681,10 @@ export class ChatCompletionDriver extends PuterDriver {
throw new HttpError(401, 'Authentication required', {
legacyCode: 'unauthorized',
});
// The caller hanging up stops nothing that has reached an upstream: it
// would bill the request anyway, so it runs to its usage report.
const upstream = new AbortController();
Context.set('upstreamSignal', upstream.signal);
// AI charges recorded in this request before the completion started.
const priorCharges = requestAiCosts().length;
@@ -784,12 +820,15 @@ export class ChatCompletionDriver extends PuterDriver {
let fundsCap: number | undefined;
// The cost factor the current attempt's charges are recorded at.
let costFactor = 1;
// The instant the current attempt is priced at, from its hold to its
// charge, so a rate change mid-request can't split the two.
let pricingTime = new Date();
if (!useFakeProvider) {
({ hold, fundsCap, costFactor } = await this.#applyCreditGate(
actor,
model,
args,
{ promptTokenEstimate, requestedMaxTokens },
{ promptTokenEstimate, requestedMaxTokens, pricingTime },
));
}
@@ -822,7 +861,12 @@ export class ChatCompletionDriver extends PuterDriver {
const attempt = toAttempt(failed.id, failed.provider!, err);
attempts.push(attempt);
if (!isFreeModel(failed)) allModelsFree = false;
if (isRouteLevelFailure(attempt)) {
// A request that outlived its deadline says nothing about the
// route's health.
if (
isRouteLevelFailure(attempt) &&
!(err instanceof UpstreamDeadlineError)
) {
markRouteUnhealthy(
failed.provider!,
failed.id,
@@ -841,6 +885,8 @@ export class ChatCompletionDriver extends PuterDriver {
if (!useFakeProvider) {
await this.#resolvePuterPaths(attemptProvider, args, actor);
}
// Last point where a caller who left costs nothing.
abortSignal?.throwIfAborted();
return attemptProvider.complete(
{
...args,
@@ -848,6 +894,7 @@ export class ChatCompletionDriver extends PuterDriver {
provider: attemptModel.provider,
},
this.#catalogEntries.get(attemptModel),
pricingTime,
);
};
@@ -908,11 +955,13 @@ export class ChatCompletionDriver extends PuterDriver {
// be capped against what is actually left.
// The previous attempt released its hold when it failed, so
// this one starts from nothing held.
pricingTime = new Date();
if (!useFakeProvider) {
({ hold, fundsCap, costFactor } =
await this.#applyCreditGate(actor, fallback, args, {
promptTokenEstimate,
requestedMaxTokens,
pricingTime,
}));
}
@@ -1000,10 +1049,20 @@ export class ChatCompletionDriver extends PuterDriver {
const init = res.init_chat_stream;
const cleanup = res.finally_fn;
// Providers also receive this signal to cancel while waiting for output.
const abortStream = () => chatStream.abort();
if (abortSignal?.aborted) abortStream();
else abortSignal?.addEventListener('abort', abortStream);
// A caller who leaves stops getting output, but the upstream keeps
// going until it reports usage, with as long to finish as a
// non-streaming completion gets.
let detachedLimit: ReturnType<typeof setTimeout> | undefined;
const detachStream = () => {
chatStream.detach();
detachedLimit = setTimeout(
() => upstream.abort(),
SDK_TIMEOUT_MS,
);
detachedLimit.unref?.();
};
if (abortSignal?.aborted) detachStream();
else abortSignal?.addEventListener('abort', detachStream);
// Intercept `chatStream.end(usage)` to fire complete + cost events
// (mirrors the non-streaming branch). Clone usage so providers that
@@ -1015,16 +1074,21 @@ export class ChatCompletionDriver extends PuterDriver {
const usageCosts = chatStream.usageCosts ?? undefined;
const enrichedUsage = usage ? { ...usage } : usage;
if (enrichedUsage) {
this.#injectUsdCents(enrichedUsage, model, usageCosts);
this.#injectUsdCents(
enrichedUsage,
model,
usageCosts,
pricingTime,
);
// An empty report recorded nothing; the stream is charged
// an estimate once it ends instead.
if (Object.values(usage!).some(Number.isFinite)) {
enrichedUsage.cost = charged();
}
}
// A cancelled stream isn't a completion, but whatever it
// metered still gets its cost row.
if (!chatStream.aborted) {
// A completion nobody received isn't announced, but it still
// gets its cost row.
if (!chatStream.detached) {
this.clients.event.emit(
'ai.prompt.complete',
{
@@ -1047,6 +1111,7 @@ export class ChatCompletionDriver extends PuterDriver {
model,
intendedProvider,
usageCosts,
pricingTime,
});
}
// Fill in sensible defaults when the provider reported no
@@ -1090,13 +1155,13 @@ export class ChatCompletionDriver extends PuterDriver {
(async () => {
try {
await init({ chatStream });
if (!chatStream.aborted && !passthrough.writableEnded) {
if (!passthrough.writableEnded) {
throw new Error('Stream ended before completion');
}
} catch (e) {
if (!chatStream.aborted) {
if (!chatStream.detached) {
const attempt = toAttempt(model.id, model.provider!, e);
passthrough.write(
chatStream.write(
`${JSON.stringify({
type: 'error',
message: sanitizeUpstreamMessage(
@@ -1112,7 +1177,8 @@ export class ChatCompletionDriver extends PuterDriver {
passthrough.end();
} finally {
clearInterval(renewHold);
abortSignal?.removeEventListener('abort', abortStream);
clearTimeout(detachedLimit);
abortSignal?.removeEventListener('abort', detachStream);
// Providers report usage the moment they meter it (see
// `AIChatStream.reportUsage`); a stream that never got
// there was never charged for.
@@ -1126,6 +1192,7 @@ export class ChatCompletionDriver extends PuterDriver {
completionId,
username,
intendedProvider,
pricingTime,
});
}
} finally {
@@ -1178,7 +1245,7 @@ export class ChatCompletionDriver extends PuterDriver {
const usageCosts = (res as IChatMessageResult).usageCosts;
if ('usage' in res && res.usage) {
this.#injectUsdCents(res.usage, model, usageCosts);
this.#injectUsdCents(res.usage, model, usageCosts, pricingTime);
this.#emitCostCalculated({
completionId,
username,
@@ -1186,6 +1253,7 @@ export class ChatCompletionDriver extends PuterDriver {
model,
intendedProvider,
usageCosts,
pricingTime,
});
res.usage.cost = charged();
}
@@ -1361,15 +1429,19 @@ export class ChatCompletionDriver extends PuterDriver {
* inherit the ceiling computed at an expensive model's price.
*/
requestedMaxTokens: number | undefined;
/** The instant this attempt is priced at. */
pricingTime: Date;
},
): Promise<{ hold: CreditHold; fundsCap?: number; costFactor: number }> {
const metering = this.services.metering;
const { promptTokenEstimate, requestedMaxTokens } = estimates;
const { promptTokenEstimate, requestedMaxTokens, pricingTime } =
estimates;
// A provider can name its own cost keys for this request (Claude's
// fast-mode rate card) and size an extra reservation the per-token
// math can't see (a web-search/advisor tool that could run several
// times before the model answers) — the same clamps it applies when
// building the request, so the hold and the request agree.
// fast-mode rate card), cap its output, and size an extra reservation
// the per-token math can't see (a web-search/advisor tool that could
// run several times before the model answers) — the same clamps it
// applies when building the request, so the hold and the request
// agree.
const pricing =
this.#providers[model.provider!]?.requestPricing?.(args, model, {
promptTokenEstimate,
@@ -1387,7 +1459,11 @@ export class ChatCompletionDriver extends PuterDriver {
actor,
this.#meteringModelKey(model),
);
const requestCosts = costsForRequest(model, promptTokenEstimate);
const requestCosts = costsForRequest(
model,
promptTokenEstimate,
pricingTime,
);
// `|| 0` also catches NaN from a malformed cost table.
const inputTokenCost =
(Number(requestCosts[inputKey] ?? 0) || 0) *
@@ -1433,6 +1509,7 @@ export class ChatCompletionDriver extends PuterDriver {
const limit = Math.min(
requestedMaxTokens ?? Number.POSITIVE_INFINITY,
outputCeiling(model, promptTokenEstimate),
positiveOrInfinity(pricing.maxTokens),
);
const cap = Math.floor(
Math.min(limit, affordableOutputTokens(remainingCredits)),
@@ -1521,6 +1598,7 @@ export class ChatCompletionDriver extends PuterDriver {
completionId: string;
username?: string;
intendedProvider: string;
pricingTime: Date;
}): void {
const {
actor,
@@ -1530,6 +1608,7 @@ export class ChatCompletionDriver extends PuterDriver {
completionId,
username,
intendedProvider,
pricingTime,
} = params;
const outputTokens = estimateOutputTokens(chatStream.outputChars ?? 0);
@@ -1542,7 +1621,7 @@ export class ChatCompletionDriver extends PuterDriver {
[outputKey]: outputTokens,
};
const cost = usageCost(usage, model);
const cost = usageCost(usage, model, undefined, pricingTime);
this.#aiMetering.utilRecordUsageObject(
{
[`estimated_${inputKey}`]: inputTokens,
@@ -1568,6 +1647,7 @@ export class ChatCompletionDriver extends PuterDriver {
usage,
model,
intendedProvider,
pricingTime,
});
}
@@ -1577,7 +1657,8 @@ export class ChatCompletionDriver extends PuterDriver {
#injectUsdCents(
usage: Record<string, number>,
model: IChatModel,
usageCosts?: Record<string, number>,
usageCosts: Record<string, number> | undefined,
pricingTime: Date,
): void {
if (
typeof usage.usd_cents === 'number' &&
@@ -1585,7 +1666,7 @@ export class ChatCompletionDriver extends PuterDriver {
) {
return;
}
const cost = usageCost(usage, model, usageCosts);
const cost = usageCost(usage, model, usageCosts, pricingTime);
if (!cost) {
(usage as Record<string, number | null>).usd_cents = null;
return;
@@ -1604,6 +1685,7 @@ export class ChatCompletionDriver extends PuterDriver {
model: IChatModel;
intendedProvider: string;
usageCosts?: Record<string, number>;
pricingTime: Date;
}) {
const {
completionId,
@@ -1612,9 +1694,10 @@ export class ChatCompletionDriver extends PuterDriver {
model,
intendedProvider,
usageCosts,
pricingTime,
} = params;
const cost = usageCost(usage, model, usageCosts);
const cost = usageCost(usage, model, usageCosts, pricingTime);
const { inputKey, outputKey } = costKeys(model);
const inputTokens = cost
? Number(
@@ -25,10 +25,10 @@ import { AIChatStream } from '../utils/Streaming.js';
import { FakeChatProvider } from './FakeChatProvider.js';
describe('fake chat streaming lifecycle', () => {
it('cancels the simulated delay before writing output', async () => {
it('cancels the simulated delay when the upstream signal aborts', async () => {
await withTestActor(async () => {
const abort = new AbortController();
Context.set('abortSignal', abort.signal);
Context.set('upstreamSignal', abort.signal);
const result = await new FakeChatProvider().complete({
model: 'fake',
stream: true,
@@ -59,7 +59,7 @@ describe('fake chat streaming lifecycle', () => {
.map((line) => JSON.parse(line));
expect(events.map((event) => event.type)).toEqual(['text', 'usage']);
});
it('does not write after the chat stream was aborted independently', async () => {
it('finishes without writing after the caller detaches', async () => {
const result = await new FakeChatProvider().complete({
model: 'fake',
stream: true,
@@ -67,8 +67,9 @@ describe('fake chat streaming lifecycle', () => {
});
const stream = new PassThrough();
const chatStream = new AIChatStream({ stream });
chatStream.abort();
chatStream.detach();
await result.init_chat_stream!({ chatStream });
expect(stream.readableLength).toBe(0);
expect(stream.writableEnded).toBe(true);
});
});
@@ -101,12 +101,11 @@ export class FakeChatProvider implements IChatProvider {
chatStream: AIChatStream;
}) => {
await delay(500, undefined, {
signal: Context.get('abortSignal'),
signal: Context.get('upstreamSignal'),
});
const response = await resp;
if (chatStream.aborted) return;
await chatStream.waitForDrain();
chatStream.stream.write(
chatStream.write(
`${JSON.stringify({
type: 'text',
text: response.message.content[0].text,
@@ -161,6 +161,33 @@ describe('OpenAICompatProvider', () => {
expect('tools' in sent[1]!).toBe(false);
});
it("drops a replayed tool call's Gemini extra_content", async () => {
const { provider, sent } = makeProvider({});
await complete(provider, {
messages: [
{ role: 'user', content: 'Weather?' },
{
role: 'assistant',
content: [
{
type: 'tool_use',
id: 'fc_1',
name: 'get_weather',
input: {},
extra_content: {
google: { thought_signature: 'c2lnLTE=' },
},
},
],
},
] as ICompleteArguments['messages'],
});
const messages = sent[0]!.messages as Array<{
tool_calls?: Array<Record<string, unknown>>;
}>;
expect(messages[1]!.tool_calls![0]).not.toHaveProperty('extra_content');
});
it('asks for stream usage only when the vendor reports it in-band', async () => {
const stream = { async *[Symbol.asyncIterator]() {} };
const reporting = makeProvider({}, stream);
@@ -28,6 +28,7 @@ import type {
IChatProvider,
ICompleteArguments,
PuterMessage,
UsageDetails,
} from '../types.js';
import {
contextLengthRetryParams,
@@ -73,6 +74,7 @@ export interface ChatCompletionsClient {
export type UsageSource = {
usage: CompletionUsage;
setUsageCosts: (costs: Record<string, number>) => void;
setUsageDetails?: (details: UsageDetails) => void;
} & Record<string, unknown>;
/**
@@ -157,6 +159,7 @@ export class OpenAICompatProvider implements IChatProvider {
async complete(
args: ICompleteArguments,
resolved?: IChatModel,
attemptPricingTime?: Date,
): Promise<IChatCompleteResult> {
const options = this.#options;
const model = resolved ?? (await this.resolveModel(args.model));
@@ -172,7 +175,7 @@ export class OpenAICompatProvider implements IChatProvider {
}
}
const actor = Context.get('actor');
const signal = Context.get('abortSignal');
const signal = Context.get('upstreamSignal');
let messages = args.messages;
if (
@@ -188,7 +191,10 @@ export class OpenAICompatProvider implements IChatProvider {
});
}
messages = this.prepareMessages(
await OpenAIUtil.process_input_messages(messages),
await OpenAIUtil.process_input_messages(messages, {
preserveToolCallOrder: this.preservesToolCallOrder(model),
keepExtraContent: this.keepsToolCallExtraContent(model),
}),
);
const tools = args.tools
@@ -226,7 +232,7 @@ export class OpenAICompatProvider implements IChatProvider {
};
const params = this.vendorParams(shared, args, model, actor);
let pricingTime = new Date();
let pricingTime = attemptPricingTime ?? new Date();
let completion;
try {
completion = await options.client.chat.completions.create(
@@ -242,7 +248,7 @@ export class OpenAICompatProvider implements IChatProvider {
)
: undefined;
if (!retryParams) throw e;
pricingTime = new Date();
pricingTime = attemptPricingTime ?? new Date();
completion = await options.client.chat.completions.create(
retryParams as never,
{ signal },
@@ -259,10 +265,11 @@ export class OpenAICompatProvider implements IChatProvider {
}
: {}),
usage_calculator: (source: UsageSource) => {
const { usage, meterOptions } = this.meteredUsage(
const { usage, meterOptions, usageDetails } = this.meteredUsage(
source,
model,
);
if (usageDetails) source.setUsageDetails?.(usageDetails);
const metered = meterChatUsage(
this.#metering,
actor,
@@ -307,6 +314,16 @@ export class OpenAICompatProvider implements IChatProvider {
// -- Vendor hooks ---------------------------------------------------
/** Whether a replayed turn's tool calls must keep the model's order. */
protected preservesToolCallOrder(_model: IChatModel): boolean {
return false;
}
/** Whether replayed tool calls keep their vendor `extra_content`. */
protected keepsToolCallExtraContent(_model: IChatModel): boolean {
return false;
}
/** Last pass over the messages, after `process_input_messages`. */
protected prepareMessages(messages: PuterMessage[]): PuterMessage[] {
return messages;
@@ -322,11 +339,18 @@ export class OpenAICompatProvider implements IChatProvider {
return params;
}
/** The usage a completion meters as, and how it's billed. */
/**
* The usage a completion meters as, and how it's billed. `usageDetails`
* replaces the breakdown otherwise derived from `usage`.
*/
protected meteredUsage(
source: UsageSource,
_model: IChatModel,
): { usage: Record<string, number>; meterOptions?: MeterChatUsageOptions } {
): {
usage: Record<string, number>;
meterOptions?: MeterChatUsageOptions;
usageDetails?: UsageDetails;
} {
return { usage: OpenAIUtil.splitCachedPrompt(source.usage) };
}
@@ -311,7 +311,7 @@ export class AzureChatProvider implements IChatProvider {
const completion = await this.#openAi.chat.completions.create(
completionParams,
{ signal: Context.get('abortSignal') },
{ signal: Context.get('upstreamSignal') },
);
return OpenAiUtil.handle_completion_output({
@@ -326,7 +326,7 @@ export class AzureResponsesProvider implements IChatProvider {
const completion = await this.#openAi.responses.create(
completionParams,
{ signal: Context.get('abortSignal') },
{ signal: Context.get('upstreamSignal') },
);
return OpenAiUtil.handle_completion_output_responses_api({
usage_calculator: ({
@@ -54,6 +54,7 @@ import { PuterServer } from '../../../../server.js';
import { setupTestServer } from '../../../../testUtil.js';
import { generateDefaultFsentries } from '../../../../util/userProvisioning.js';
import { withTestActor } from '../../../integrationTestUtil.js';
import { normalize_messages } from '../../utils/Messages.js';
import { AIChatStream } from '../../utils/Streaming.js';
import { FILES_API_BETA } from './fileUpload.js';
import { CLAUDE_MODELS } from './models.js';
@@ -319,7 +320,7 @@ describe('ClaudeProvider.complete request shape', () => {
};
it.each([false, true])(
'passes cancellation to the SDK with stream=%s',
'passes the upstream signal to the SDK with stream=%s',
async (stream) => {
const { provider } = makeProvider();
const abort = new AbortController();
@@ -327,7 +328,7 @@ describe('ClaudeProvider.complete request shape', () => {
messagesStreamMock.mockReturnValueOnce(makeStreamLike([]));
else messagesCreateMock.mockResolvedValueOnce(baseResponse);
await withTestActor(() => {
Context.set('abortSignal', abort.signal);
Context.set('upstreamSignal', abort.signal);
return provider.complete({
model: 'claude-sonnet-4-6',
messages: [{ role: 'user', content: 'hi' }],
@@ -963,6 +964,63 @@ describe('ClaudeProvider.complete request shape', () => {
).toMatchObject({ type: 'thinking', signature: 'sig_1' });
});
it.each([
['string', '{"city":"Paris"}'],
['object', { city: 'Paris' }],
])(
'drops a Gemini thought signature from a replayed tool call with %s arguments',
async (_form, args) => {
// A Gemini turn replayed against Claude on fallback keeps only
// Anthropic-compatible metadata, including cache_control.
const { provider } = makeProvider();
messagesCreateMock.mockResolvedValueOnce(baseResponse);
const messages = normalize_messages([
{ role: 'user', content: 'Weather in Paris?' },
{
role: 'assistant',
content: null,
cache_control: { type: 'ephemeral', ttl: '1h' },
tool_calls: [
{
id: 'fc_1',
type: 'function',
function: { name: 'get_weather', arguments: args },
extra_content: {
google: { thought_signature: 'c2lnLTE=' },
},
},
],
},
{ role: 'tool', tool_call_id: 'fc_1', content: '18C' },
]);
const before = JSON.parse(JSON.stringify(messages));
await withTestActor(() =>
provider.complete({
model: 'claude-haiku-4-5-20251001',
messages,
}),
);
const [sent] = messagesCreateMock.mock.calls[0]!;
expect(sent.messages[1].content).toEqual([
{
type: 'tool_use',
id: 'fc_1',
name: 'get_weather',
input: { city: 'Paris' },
cache_control: { type: 'ephemeral', ttl: '1h' },
},
]);
// The driver hands these same objects to the next fallback route.
expect(messages).toEqual(before);
expect(messages[1].content[0].extra_content).toEqual({
google: { thought_signature: 'c2lnLTE=' },
});
},
);
it('survives the same messages array being sent twice', async () => {
// This is the fallback hazard the copy-on-write exists for: the driver
// reuses one messages array across attempts, so if attempt 1 strips
@@ -1667,6 +1725,98 @@ describe('ClaudeProvider.complete streaming', () => {
expect(part).toEqual({ puter_path: path });
});
it('answers without waiting for the uploaded files to be deleted', async () => {
const { provider } = makeProvider();
const { actor, path } = await makeUserWithFile();
filesUploadMock.mockResolvedValue({ id: 'file_reply_1' });
// A delete that never settles must not hold the reply.
filesDeleteMock.mockReturnValue(new Promise(() => {}));
messagesCreateMock.mockResolvedValueOnce({
content: [{ type: 'text', text: 'hi' }],
usage: { input_tokens: 1, output_tokens: 1 },
});
const result = await withTestActor(
() =>
provider.complete({
model: 'claude-haiku-4-5-20251001',
messages: [
{ role: 'user', content: [{ puter_path: path }] },
],
}),
actor,
);
expect(result).toMatchObject({ finish_reason: 'stop' });
expect(filesDeleteMock).toHaveBeenCalledWith('file_reply_1', {
betas: [FILES_API_BETA],
});
});
it('hands a refused stream to the fallback without waiting for the deletes', async () => {
const { provider } = makeProvider();
const { actor, path } = await makeUserWithFile();
filesUploadMock.mockResolvedValue({ id: 'file_stream_2' });
filesDeleteMock.mockReturnValue(new Promise(() => {}));
const refused = Object.assign(new Error('Overloaded'), {
status: 529,
});
messagesStreamMock.mockReturnValueOnce({
...makeStreamLike([]),
withResponse: () => Promise.reject(refused),
});
const part: Record<string, unknown> = { puter_path: path };
await expect(
withTestActor(
() =>
provider.complete({
model: 'claude-haiku-4-5-20251001',
messages: [{ role: 'user', content: [part] }],
stream: true,
}),
actor,
),
).rejects.toBe(refused);
expect(filesDeleteMock).toHaveBeenCalledWith('file_stream_2', {
betas: [FILES_API_BETA],
});
expect(part).toEqual({ puter_path: path });
});
it('leaves nothing unhandled when deleting an uploaded file fails', async () => {
const { provider } = makeProvider();
const { actor, path } = await makeUserWithFile();
filesUploadMock.mockResolvedValue({ id: 'file_gone_1' });
filesDeleteMock.mockRejectedValue(new Error('already gone'));
messagesCreateMock.mockResolvedValueOnce({
content: [{ type: 'text', text: 'hi' }],
usage: { input_tokens: 1, output_tokens: 1 },
});
const unhandled: unknown[] = [];
const onUnhandled = (reason: unknown) => unhandled.push(reason);
process.on('unhandledRejection', onUnhandled);
try {
await withTestActor(
() =>
provider.complete({
model: 'claude-haiku-4-5-20251001',
messages: [
{ role: 'user', content: [{ puter_path: path }] },
],
}),
actor,
);
await new Promise((resolve) => setTimeout(resolve, 20));
} finally {
process.off('unhandledRejection', onUnhandled);
}
expect(filesDeleteMock).toHaveBeenCalledWith('file_gone_1', {
betas: [FILES_API_BETA],
});
expect(unhandled).toEqual([]);
});
it('surfaces a failure the event iterator swallowed instead of ending the stream clean', async () => {
const { provider } = makeProvider();
const dropped = Object.assign(new Error('Overloaded'), {
@@ -389,10 +389,10 @@ export class ClaudeProvider implements IChatProvider {
if (args.stream) {
const completion = usesBeta
? this.anthropic.beta.messages.stream(sdkParams as never, {
signal: Context.get('abortSignal'),
signal: Context.get('upstreamSignal'),
})
: this.anthropic.messages.stream(sdkParams as never, {
signal: Context.get('abortSignal'),
signal: Context.get('upstreamSignal'),
});
// Subscribed before the request is awaited: the SDK only queues
// events for iterators that already exist.
@@ -403,7 +403,8 @@ export class ClaudeProvider implements IChatProvider {
try {
await completion.withResponse();
} catch (e) {
await cleanupUploads();
// Best-effort and never rejects, so the fallback need not wait.
void cleanupUploads();
restoreUploads();
throw e;
}
@@ -431,10 +432,10 @@ export class ClaudeProvider implements IChatProvider {
try {
const msg = await (usesBeta
? this.anthropic.beta.messages.create(sdkParams as never, {
signal: Context.get('abortSignal'),
signal: Context.get('upstreamSignal'),
})
: this.anthropic.messages.create(sdkParams as never, {
signal: Context.get('abortSignal'),
signal: Context.get('upstreamSignal'),
}));
const { usage, costs, details, advisorModel } = this.#meter(
(msg as Message).usage,
@@ -512,7 +513,8 @@ export class ClaudeProvider implements IChatProvider {
restoreUploads();
throw e;
} finally {
await cleanupUploads();
// Best-effort and never rejects, so the response need not wait.
void cleanupUploads();
}
}
@@ -931,7 +933,8 @@ export class ClaudeProvider implements IChatProvider {
};
});
// Claude requires tool_use.input to be a dictionary
// Claude requires tool_use.input to be a dictionary. Gemini metadata
// is not part of Anthropic's tool_use schema.
messages = messages.map((message) => {
if (!Array.isArray(message.content)) return message;
let changed = false;
@@ -945,14 +948,14 @@ export class ClaudeProvider implements IChatProvider {
} catch {
input = {};
}
changed = true;
} else if (input === undefined || input === null) {
input = {};
changed = true;
} else {
} else if (!('extra_content' in part)) {
return part;
}
return { ...part, input };
changed = true;
const { extra_content, ...rest } = part;
return { ...rest, input };
},
);
return changed ? { ...message, content } : message;
@@ -47,6 +47,7 @@ import type { MeteringService } from '../../../../services/metering/MeteringServ
import { PuterServer } from '../../../../server.js';
import { setupTestServer } from '../../../../testUtil.js';
import { withTestActor } from '../../../integrationTestUtil.js';
import { costsForRequest, usageCost } from '../../utils/pricing.js';
import { AIChatStream } from '../../utils/Streaming.js';
import { DEEPSEEK_MODELS } from './models.js';
import { DeepSeekProvider } from './DeepSeekProvider.js';
@@ -621,6 +622,141 @@ describe('DeepSeekProvider.complete non-stream output', () => {
});
});
// ── Peak-only pricing ───────────────────────────────────────────────
describe('DeepSeekProvider peak-only pricing', () => {
// 1M prompt tokens of which 250k are cache hits, plus 100k output.
const upstreamUsage = {
prompt_tokens: 1_000_000,
completion_tokens: 100_000,
prompt_tokens_details: { cached_tokens: 250_000 },
};
const meteredUsage = {
prompt_tokens: 750_000,
completion_tokens: 100_000,
cached_tokens: 250_000,
};
// µ¢ per token equals cents per 1M tokens.
const peakCosts = {
'deepseek-flash': {
prompt_tokens: 750_000 * 30,
completion_tokens: 100_000 * 120,
cached_tokens: 250_000 * 0.6,
},
'deepseek-v4-pro': {
prompt_tokens: 750_000 * 132,
completion_tokens: 100_000 * 396,
cached_tokens: 250_000 * 4.4,
},
};
// DeepSeek's peak windows are 01:00-04:00 and 06:00-10:00 UTC on
// weekdays, excluding Chinese public holidays; every instant still
// bills at peak rates.
const requestTimes = {
'last instant before the 01:00 peak': '2026-10-14T00:59:59.999Z',
'01:00 peak start': '2026-10-14T01:00:00.000Z',
'last instant of the 01:00-04:00 peak': '2026-10-14T03:59:59.999Z',
'04:00 peak end': '2026-10-14T04:00:00.000Z',
'last instant before the 06:00 peak': '2026-10-14T05:59:59.999Z',
'06:00 peak start': '2026-10-14T06:00:00.000Z',
'last instant of the 06:00-10:00 peak': '2026-10-14T09:59:59.999Z',
'10:00 peak end': '2026-10-14T10:00:00.000Z',
'weekday off-peak between windows': '2026-10-14T05:00:00.000Z',
'weekday off-peak evening': '2026-10-14T20:00:00.000Z',
'Saturday 02:00': '2026-10-17T02:00:00.000Z',
'Sunday 08:00': '2026-10-18T08:00:00.000Z',
'Chinese National Day (Thursday) 02:00': '2026-10-01T02:00:00.000Z',
'Spring Festival holiday (Monday) 08:00': '2027-02-08T08:00:00.000Z',
};
const cases = Object.keys(peakCosts).flatMap((modelId) =>
Object.entries(requestTimes).map(([label, iso]) => ({
modelId: modelId as keyof typeof peakCosts,
label,
iso,
})),
);
afterEach(() => {
vi.useRealTimers();
});
it('carries the published peak rates per 1M tokens', () => {
const rates = Object.fromEntries(
DEEPSEEK_MODELS.map((m) => [m.id, m.costs]),
);
expect(rates).toEqual({
'deepseek-flash': {
tokens: 1_000_000,
prompt_tokens: 30,
cached_tokens: 0.6,
completion_tokens: 120,
},
'deepseek-v4-pro': {
tokens: 1_000_000,
prompt_tokens: 132,
cached_tokens: 4.4,
completion_tokens: 396,
},
});
for (const m of DEEPSEEK_MODELS) {
expect(m.pricingOverrides).toBeUndefined();
}
});
it.each(cases)(
'bills $modelId at peak rates for a $label request',
async ({ modelId, iso }) => {
vi.useFakeTimers({ toFake: ['Date'], now: new Date(iso) });
const { provider } = makeProvider();
createMock.mockResolvedValueOnce({
choices: [
{
message: { content: 'ok', role: 'assistant' },
finish_reason: 'stop',
},
],
usage: upstreamUsage,
});
await withTestActor(() =>
provider.complete({
model: modelId,
messages: [{ role: 'user', content: 'hi' }],
}),
);
expect(recordSpy).toHaveBeenCalledTimes(1);
const [usage, , prefix, overrides] = recordSpy.mock.calls[0]!;
expect(prefix).toBe(`deepseek:${modelId}`);
expect(usage).toEqual(meteredUsage);
expect(overrides).toEqual(peakCosts[modelId]);
},
);
it.each(cases)(
'quotes $modelId at peak rates for a $label request',
({ modelId, iso }) => {
vi.useFakeTimers({ toFake: ['Date'], now: new Date(iso) });
const model = DEEPSEEK_MODELS.find((m) => m.id === modelId)!;
const expected = peakCosts[modelId];
// The credit gate prices the request at the current instant.
expect(costsForRequest(model, 1_000_000)).toEqual(model.costs);
// The reported usd_cents, from recorded costs or the model's rates.
for (const recorded of [expected, undefined]) {
expect(usageCost(meteredUsage, model, recorded)).toEqual({
inputMicroCents: Math.round(
expected.prompt_tokens + expected.cached_tokens,
),
outputMicroCents: expected.completion_tokens,
});
}
},
);
});
// ── Streaming deltas ────────────────────────────────────────────────
describe('DeepSeekProvider.complete streaming', () => {
@@ -19,9 +19,9 @@
import type { IChatModel } from '../../types.js';
// Hardcoded from https://api-docs.deepseek.com/quick_start/pricing. DeepSeek
// bills half price off-peak; we charge the peak rate since the window isn't
// known when a request is priced.
// Peak rates from https://api-docs.deepseek.com/quick_start/pricing. Direct
// DeepSeek always bills at these rates, including off-peak hours, weekends and
// holidays, so keep conditional `pricingOverrides` off these models.
export const DEEPSEEK_MODELS: IChatModel[] = [
{
puterId: 'deepseek:deepseek/deepseek-flash',
Loaded 100 of 467 files, more files were not shown because too many files have changed in this diff. Show more