Compare commits

..
Author SHA1 Message Date
Daniel Salazar 1f1e5674d8 chore: bump puter js ver (#4073) 2026-10-04 17:17:38 -07:00
Daniel Salazar 8db4fb4f5b fix(ai): chat credit gate ceiling, in-flight holds and disconnects (#4071)
* docs(agents): no credits or bug-report mentions in commits and PRs

* fix(ai): chat credit gate ceiling, in-flight holds and disconnects

- Cap output at min(max_tokens, context - prompt). The gate used
  max_tokens - prompt, so any prompt longer than a model's output limit
  was refused with 402 regardless of balance. Gemini 2.5 Pro's output
  limit corrected to 65,536.
- Abort the upstream stream when the caller disconnects (driver calls and
  the OpenAI/Anthropic-compatible routes), which ends the credit hold.
- A request refused only because the account's own in-flight requests
  hold its balance gets 429 `credits_reserved` instead of 402; puter.js
  retries it with exponential backoff (1s to 30s, about 2 minutes).
- When the balance caps a completion and it uses the whole cap, the
  response carries `usage_limited` (non-stream `metadata`, stream `usage`
  line), which puter.js already turns into the upgrade prompt.

* perf(ai): read the balance and its holds once per credit gate

The gate re-read holds, and the whole balance when any were held, to tell
spent from committed. getUsageHeadroom returns both halves from the same
reads getRemainingUsage already did.
2026-10-04 17:10:51 -07:00
Daniel Salazar e7c89b0639 fix(apps): keep ids longer than an app uid out of the app-stats query (#4070)
ClickHouse's app_opens.app_uid is FixedString(40) and fails the whole
query on a longer value. Marketplace callers pass item ids into
getAppsStats, so one long listing-request id turned the seller's
listing-request list into a 500. Such ids can't have opens; they now
resolve to zero without being queried.
2026-10-04 17:03:19 -07:00
Daniel Salazar 83d64ac685 fix: follow-ups from the reviews of #4047, #4049, #4051, #4053, #4055 and #4057 (#4068)
* fix(auth): issue reauth tokens only for browser sessions

A rejected token's 401 carried a signed reauth_token, which /signup turns
into a session on a temp account. Any revoked app or access token, a stale
token from a deleted app whose uid was reused, or a worker credential could
get one. Only a GUI/browser session's rejection now carries a reauth_token;
everything else still gets reauth_required without it. The stale-app check
also runs before the session is touched.

* fix(events): refuse worker creation from an events handler

workers.create minted a worker token with no handler depth, so a handler
could deploy a worker and chain writes through it at depth 0. A handler
actor can no longer create a worker; delivery, deploys and hot reload mint
from plain owner actors and are unaffected.

* fix(ai-ocr): bound the page count's parsing and close its under-counts

- Object-stream headers are capped (/N, /First) and read incrementally, and
  the inflate budget scales with the file, so a crafted PDF can't spend
  seconds and gigabytes before the credit check.
- An escaped /ObjStm type, an indirect /Count, untyped page leaves and an
  object redefined to a smaller tree no longer lower the count.
- An account with no balance is refused before the file is parsed.

* fix(fs): add a switch to stop accepting pre-binding signed URLs

Signed URLs issued before owner binding are still accepted, and owner-signed
ones never expire, so the move-to-another-tree attack stays open for them
indefinitely. legacy_file_signatures (default on) can now turn them off, a
rate-limited warning counts their use so it's visible when they've died
out, and a non-UUID uid is refused before the entry lookup.

* fix(fs): hide the issuer's home uid from a scoped token's entries

A token scoped below the home could still learn the home's uid as the
parent_uid of a direct child (e.g. stat on ~/Documents). For access-token
actors that can't list the home, a direct child's parent uid is now null in
the v2 and legacy entry shapes, using the same rule as the root listing.

* fix(gui): ask for the 2FA code on the last step, right before enabling

The setup flow verified a code before the recovery-codes screen and sent
that same code to enable afterwards; by then it had usually expired, and
the user was sent back to the start. The code is now entered last and goes
straight to enable; a wrong code keeps the user on that step, and other
failures show the server's message. Enter submits the password step.

* test(fs): expect the legacy-signature flag in the signing config

* fix: apply the review's follow-ups to the merged-PR fixes

- Reauth tokens also require the session row to be a browser ('web')
  session, not just the absence of the worker claim.
- The OCR page count only treats an untyped object as a page when it sits in
  a page tree's /Kids, so outline items, widgets and name-tree leaves can't
  inflate an honest PDF's count.
- The signed-URL uid guard checks the uuid's shape only, so a legacy row
  whose uuid has nonstandard version/variant bits still resolves.
2026-10-04 14:46:21 -07:00
Daniel Salazar 5e34258da5 fix: drop database_id from the subdomain reclaim; MySQL has no such column (#4067)
reclaimByUuid (#4064) cleared `database_id`, which exists only in the SQLite
and Postgres schemas. On MySQL the UPDATE fails with "Unknown column", so
ensureSystemSite throws: the puter-profiles row stays registered to the
admin, the site keeps 404ing, and every profile save returns 500 as it
retries the bootstrap. puter-app-icons takes the same path if its row isn't
already protected. System rows never carry a database binding, so there is
nothing to clear.
2026-10-04 02:45:03 -07:00
Daniel Salazar 51eba174b3 fix: alert errors from timeouts (#4066) 2026-10-04 02:44:40 -07:00
Daniel Salazar f533eb84ae fix(ai-ocr): check credits for the whole document before calling the provider (#4055)
* fix(ai-ocr): check credits for the whole document before calling the provider

The pre-flight checked one page's cost because the page count was only
known from the provider's answer, so a 130-page PDF passed on a balance
worth a fraction of it and drove the account negative.

The driver now estimates pages before the call: a PDF's own count (read
from its page tree, including compressed object streams), capped by a
`pages` selection; one page for images and Textract, whose synchronous
API reads single-page documents. Other documents and PDFs it can't read
count 20 pages per MB, Mistral's 1,000-page limit at its 50 MB size cap.
The estimated cost is checked and held while the provider runs, so
concurrent calls see it. Usage is still metered from the pages the
provider reports.

* fix(ai-ocr): price the credit hold at the AI cost factor

Billing scales OCR usage by the ai.cost.factor hook, but the hold was sized
at the raw cost, so a balance covering N pages passed the check and was
then charged more. Hold through reserveAiCredits like the other AI drivers.
2026-10-04 00:53:19 -07:00
Nikhil Jangid 3dfc70cab7 fix: clarify team directory app access (#4031) 2026-10-04 00:51:02 -07:00
Daniel Salazar ebf5086e1c fix(fs): keep restore metadata out of reach of anyone but the tree's owner (PUT-2050) (#4063)
Follow-up to #4048. original_path/original_name decide where a GUI restore
lands, and the owner's restore is unchecked, so whoever sets them can aim a
file at another app's AppData root and block that app's launch. #4048 only
pinned them on move for non-account actors; a write could still set them,
and a share recipient's session counts as account context.

Writes now drop the trash keys for every caller. On move, only the account
that owns both source and destination may set them; anyone else moving into
the owner's Trash gets them recomputed from the entry, and other moves drop
them, including keys already on the row. Copies drop them on every branch.
The GUI only offers Restore for direct children of Trash, since nested rows
never carry legitimate trash metadata. The AppData/Trash checks for
cross-app deletes and create grants now ignore case, like the root guard.
2026-10-04 00:48:48 -07:00
Daniel Salazar 59e28bed23 fix(metering): keep a failed lookup's fallback out of the credit cache (PUT-2002) (#4062)
Follow-up to #4052. A failed subscription lookup was cached for 5 s, but the
credit cache built from its fallback plan lived 15 s, so a paid user who
had spent past the free allowance got 402s for ~15 s after one failed read.
Credit entries built from a provisional answer (a failed lookup, or an
actor missing its email or numeric id) are now capped at that answer's
expiry and don't drive credit-state alerts.

A failed lookup that finishes after a concurrent successful one no longer
replaces the good cached answer, actors without a numeric id aren't cached
(resolvers key on it), and a failed credit refresh fails open briefly
instead of replaying a stale answer or blocking on the failing store.
2026-10-04 00:48:35 -07:00
Daniel Salazar 73ebe3f47a fix(metering): decide app-usage access in MeteringService only (PUT-2015) (#4061)
The /metering/usage/:app handler repeated the "an app reads only its own
usage" check while the service still let any app read the global bucket.
The service now refuses an app actor anything but its own app, and the
handler only resolves names. Status codes for app actors are unchanged; a
user reading os-global gets their global usage instead of a name-lookup 404.

whoami no longer sets taskbar_items to undefined for app actors, and only
keys with a value count as built, so a listener can't fill one in for an
app. getMonthlyUsage's JSDoc notes allowanceInfo is account-wide.
2026-10-04 00:47:37 -07:00
Daniel Salazar c981a51f21 fix(fs): bind signed file URLs to the owner they were issued under (#4053)
New /sign signatures are an HMAC over uid, action, expiry and the entry's
owner at signing time. /file and /writeFile resolve the entry first and
verify against its current owner, so a move that hands the entry to another
account ends its URLs and stops /file minting child URLs from them. The URL
shape is unchanged; signatures in the previous format still verify, without
the owner check.
2026-10-04 00:47:16 -07:00
Daniel Salazar 4ef5e6c110 fix(socket): hold unlisted free plans to the free connection cap (PUT-1830) (#4060)
The per-account socket cap looked a plan up directly in its map, so
org_seat_free (team seats on a free team) got the paid cap. Route and
events limits already gave an unlisted free plan the default free entry;
that rule now lives in one helper, subscriptionOverride, used by all three.

Also publishes the socket connection caps, corrects the seat-plan line in
rate-limits-and-quotas.md, and makes the isFreeSubscription test assert
literal ids.
2026-10-04 00:46:59 -07:00
Daniel Salazar 794469845d fix(thumbnails): bind thumbnail keys to their entry and size-bound uploads (PUT-1999) (#4054) 2026-10-04 00:46:35 -07:00
Daniel Salazar 9ca77be5a9 fix(auth): require a step-up to set up 2FA and a valid code to enable it (#4051)
Setup returned a fresh secret and recovery codes to any session, and enable
flipped otp_enabled without proof of the secret, so a stolen session could
enroll its own authenticator and lock the owner out. Setup now lives at
/user-protected/setup-2fa behind the same password / OIDC revalidation gate as
disable-2fa, and enable requires a live code for the stored secret. The GUI
setup window asks for the password (or opens the revalidation popup) first,
and sends the code it already collected to enable.
2026-10-04 00:26:28 -07:00
Daniel Salazar 007d5391a1 fix(events): carry handler depth onto tokens a handler mints (#4049)
An events handler could mint a fresh token for its own app through /sign or
/auth/create-access-token, and writes made with it started a new chain. An app
token minted from a handler token now carries its handler_depth and expires
with it; an access token carries the depth at the lifetime asked for.
2026-10-04 00:25:54 -07:00
Daniel Salazar c0f4fe93b1 fix(auth): stop a deleted app's sessions authenticating as the next app with its uid (PUT-2049) (#4057) 2026-10-04 00:24:55 -07:00
Daniel Salazar d57f638eda fix: keep system sites writable and their names out of reach (#4064)
Follow-up to #4056. Icons and profiles are now written as the system user,
so on a storage-limited install they counted against its allowance (100 MB
by default) and every write past that returned 413, including a user's
profile save. Both writes now pass the unmetered allowance, as writes the
system makes on users' behalf already may.

ensureSystemSite reclaims a mismatched subdomain row in place instead of
delete+create, so the name is never free for a user to register, the row
keeps its uuid, and `protected` is never cleared; the previous holder's
bindings are dropped. If the row vanished since the read it falls through
to create. puter-app-icons is now protected like puter-profiles, both names
are reserved from user registration, and they live in util/systemSite so
the subdomain driver doesn't import the services. The icon service's
shutdown wait is capped at 5 s so a stalled run can't hold up the metering
flush.
2026-10-04 00:23:01 -07:00
Daniel Salazar 26abd4e4a5 fix(fs): hide the issuer's home from scoped-token root listings (PUT-2053) (#4047) 2026-10-04 00:22:43 -07:00
Daniel Salazar 3defae3cfc fix: own system sites by the system user (#4056)
* fix: register system sites to the owner of their directory

The hosting middleware refuses a site whose user doesn't own its root
directory (absent a manage grant). The app-icons and profiles bootstraps
always registered their subdomain to the admin, but the directory can belong
to another user: a carried-over `/system` owned by the `system` user, which a
new child directory inherits. Those sites then 404 on every request.

Both bootstraps now go through ensureSystemSite, which registers the
subdomain to the directory's owner and replaces a row registered to anyone
else or pointing at another directory.

Fixes #4045

* fix: own system sites by the system user

`/system`, `/system/app_icons` and `/system/profiles` and their subdomains
belong to the system user, and icon and profile files are written as it.
ensureSystemSite re-owns directories another account created and replaces a
subdomain row registered to anyone else, so installs bootstrapped under the
admin move over on boot.

The bootstrap no longer waits for an admin account, so DefaultUserService
stops re-running it, and deployments without an admin get icons and profiles.
AppIconService lets in-flight icon runs finish on shutdown.
2026-10-03 17:06:13 -07:00
Daniel Salazar 205f19a167 fix(metering): stop caching subscription answers from email-less actors and failed lookups (PUT-2002) (#4052) 2026-10-03 16:39:08 -07:00
Daniel Salazar 2cfc1461e4 fix: scope whoami billing details and monthly usage to the calling app (PUT-2015) (#4050)
* fix: scope whoami billing details and monthly usage to the calling app (PUT-2015)

* fix(metering): refuse an app another app's detailed usage (PUT-2015)
2026-10-03 16:38:31 -07:00
Daniel Salazar 3fa23bb809 fix(fs): refuse renaming or moving AppData roots (PUT-2050) (#4048)
* fix(fs): refuse renaming or moving AppData roots (PUT-2050)

* fix(fs): refuse a forged trash original_path from apps (PUT-2050)
2026-10-03 16:37:36 -07:00
Daniel Salazar e59f6285ba refactor(metering): route free-plan checks through isFreeSubscription (PUT-1830) (#4046) 2026-10-03 16:36:05 -07:00
Filip Kujundžić c3abec8663 Merge pull request #4035 from HeyPuter/filipkujundzic/put-1895-chat-credit-gate-cost-factor
fix(ai): price the chat credit gate at the factored cost (PUT-1895)
2026-10-03 19:44:51 +02:00
Neal Shah 04406373f4 event based FBL implementation (#3969) 2026-10-03 05:54:04 -04:00
Daniel Salazar 91aff909e1 fix: keep launch tokens out of URLs and storage (PUT-2040) (#4038)
* fix: keep launch tokens out of URLs and storage (PUT-2040)

- Strip the consumed launch token param from the app URL after load
- Never persist user session (godmode) tokens to localStorage; purge any stored one at boot
- Worker log socket sends the token as its first message instead of in the URL path

* fix: strip the private-app sign-in token from top-level URLs, keep godmode tokens across reloads (PUT-2040)

- Private-app gate: a top-level GET carrying ?puter.auth.token= that passes the gate gets the sticky cookie and a 302 to the same URL without the token. Framed launches are untouched.
- Godmode session tokens go to sessionStorage (never localStorage), so a frame reload stays signed in; cleared on sign-out or when an app token replaces it.
2026-10-03 00:26:31 -07:00
Daniel Salazar a242105540 fix(events): keep app-less scoped tokens off the account's durable rows and mailbox (#4042)
A personal access token scoped below the account (e.g. the one in an
account-minted getReadURL() URL) has effectiveApp === null, which the durable
list/unsubscribe/ack scope check and notif fetch read as account context. Gate
those on isAccountContext, as workers/handlers/kv-handles already do, and refuse
such tokens on durable subscribe.
2026-10-02 22:37:07 -07:00
Reynaldi Chernando 47510897ed add ai builder link to docs (#4044) 2026-10-03 10:27:23 +07:00
dependabot[bot] d0da8c5b1e chore(deps): bump engine.io from 6.6.9 to 6.6.11 (#4017)
Bumps [engine.io](https://github.com/socketio/socket.io) from 6.6.9 to 6.6.11.
- [Release notes](https://github.com/socketio/socket.io/releases)
- [Changelog](https://github.com/socketio/socket.io/blob/main/CHANGELOG.md)
- [Commits](https://github.com/socketio/socket.io/compare/[email protected]@6.6.11)

---
updated-dependencies:
- dependency-name: engine.io
  dependency-version: 6.6.11
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-02 15:03:34 -07:00
dependabot[bot] 303ee6c2c7 chore(deps-dev): bump brace-expansion from 1.1.18 to 1.1.21 (#4015)
Bumps [brace-expansion](https://github.com/juliangruber/brace-expansion) from 1.1.18 to 1.1.21.
- [Release notes](https://github.com/juliangruber/brace-expansion/releases)
- [Commits](https://github.com/juliangruber/brace-expansion/compare/v1.1.18...v1.1.21)

---
updated-dependencies:
- dependency-name: brace-expansion
  dependency-version: 1.1.21
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-02 15:02:56 -07:00
Daniel Salazar 37c1678897 fix(fs): keep a vacated username off-limits to other accounts for an hour (#4039)
* fix(fs): keep a vacated username off-limits to other accounts for an hour

Renaming a home rewrites descendant paths in the database, but their
cached entries keep the old path until they expire. ACL grants a home by
path prefix, so whoever takes the old name next must not be able to
before those entries are gone. renameUserHome now records the vacated
name, and the claim check every username-claim site already runs treats
it as taken for anyone but the account that left it.

* test: seed recommended apps that are still in the default list

The default list no longer includes editor or camera, so the ordering
test found neither.
2026-10-02 14:53:34 -07:00
Filip KujundžićandClaude Opus 5.5 7b8b3752e4 fix(ai): price the chat credit gate at the factored cost
The chat gate read `model.costs` directly to decide affordability, cap
`max_tokens` and size the credit hold, while usage is recorded at cost
times the AI cost factor, so all three ran low by the factor on every
request. The gate now resolves the factor and applies it to both rates.

The factor is looked up by the key the provider records usage under.
Providers expose that key through `meteringModelKey` and use it when
recording, so the gate and the charge can't drift apart; the
unreported-stream backstop records under the same key.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 21:21:18 +02:00
404oopsandClaude Opus 5.5 521812be02 refactor(ai): move the AI cost factor out of MeteringService
MeteringService knew about AI: it built the cost-factor facade, resolved
the `ai.cost.factor.*` hook and, since the previous commit, offered
`reserveAiCredits`. Metering stays agnostic to what it meters, so all of
that now lives with the AI drivers in `drivers/util/aiCostFactor.ts`.

`withAiCostFactor(metering, events, driver)` returns an
`AiMeteringService`: the service with recording scaled by the factor,
plus `costFactor` and `reserveAiCredits` for gates. The speech providers
take that type. MeteringService itself has no AI methods left.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-10-02 21:20:24 +02:00
Nariman Jelveh bb38e8c86f Update RecommendedAppsService.ts 2026-10-02 11:29:42 -07:00
Nariman Jelveh 74d89b64fd Update RecommendedAppsService.ts 2026-10-02 11:19:37 -07:00
Daniel Salazar 31e5b64a0f fix: make app-data delete grants imply write and read (PUT-1998) (#4022) 2026-10-02 10:47:58 -07:00
Daniel Salazar e96cf77982 feat(metering): per-month allowance override for partly billed plan changes (PUT-1996) (#4027) 2026-10-02 10:40:52 -07:00
Filip KujundžićandClaude Opus 5.5 22d52cf544 fix(ai): name data-URL audio with its real extension
A data URL's filename came from its MIME subtype, so `audio/mpeg`
became `input.mpeg` and `audio/x-wav` became `input.x-wav`. OpenAI's
transcription API picks the decoder from the extension and rejected
both. Map the MIME types whose subtype isn't the extension.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 19:28:13 +02:00
Daniel Salazar 86bb94d3d9 fix: cap FS batch array lengths (PUT-1894) (#4024) 2026-10-02 10:06:21 -07:00
Daniel Salazar a4f1e665df fix: return 400 when a streamed upload is shorter than its declared size (#4023)
A streamed write sends the caller's declared size to the object store as
the body length. A body that ends short of it was rejected as
IncompleteBody and surfaced as a 500.
2026-10-02 10:05:13 -07:00
Filip KujundžićandClaude Opus 5.5 abd0d809d5 fix(ai): price speech credit gates at the charged cost and hold them in flight
Text-to-speech, speech-to-text and voice-changer providers checked
affordability against the unscaled provider cost and never reserved it,
so the gate ran low by the AI cost factor and concurrent requests could
each spend the whole balance. They now go through
`reserveAiCredits`, which scales the amount by the model's cost factor,
checks it, and holds it until the request finishes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 18:56:53 +02:00
Filip KujundžićandClaude Opus 5.5 82bd7a7360 chore(ai): sync model catalogs with provider listings and deprecations
Remove models providers have deprecated or retired, add newly released
ones, and move provider defaults off retired models. Gemini video (Veo
via the Gemini API) is removed; Infron and OpenRouter skip models their
listings mark deprecated. xAI bills cached prompt tokens only at the
cached rate. Claude Sonnet 4.6 max output raised to 128K.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 18:52:35 +02:00
Reynaldi Chernando b32ae7dcfc Add rate limit detail for contact form recipe (#4030)
* Add rate limit detail for contact form recipe

* handle ipv6
2026-10-02 23:19:41 +07:00
Juan Fernando Castro d5e6d5ea07 Merge pull request #4007 from HeyPuter/juancastro/put-1896-file-and-sandboxed-iframe-origins-get-a-hard-400-and-cant
fix: refuse opaque origins cleanly instead of a logged 400 (PUT-1896)
2026-10-02 09:58:13 -04:00
Reynaldi ChernandoandCopilot Autofix powered by AI f46d187d51 contact form recipe (#4029)
* contact form recipe

* Potential fix for pull request finding 'Network failures cause unhandled fetch rejections'

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

* minor

---------

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
2026-10-02 20:24:05 +07:00
Reynaldi ChernandoandCopilot Autofix powered by AI 0fcee62884 Update recipes for storing a small list and store per id (#4028)
* Update recipes for storing a small list and store per id

* Potential fix for pull request finding 'Clarify that IDs must be path-safe or properly escaped'

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

---------

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
2026-10-02 19:23:39 +07:00
Reynaldi Chernando 384524a027 improve teams docs (#4025) 2026-10-02 14:45:54 +07:00
Daniel Salazar 42b204f46e fix: don't count failed username change towards rate limit (#4021) 2026-10-02 00:26:37 -07:00
Daniel Salazar 91c8b2385b fix: run a broadcast persistent handler in the worker or the clients, not both (PUT-1889) (#4016) 2026-10-02 00:08:02 -07:00
jelveh 05e2e7428a feat(gui): let users remove their profile picture
The account tab only offered changing the avatar. Add a "Remove photo"
action, shown while a picture is set, that clears it via
update_profile({ picture: null }) and resets every avatar to the default.
On failure the hint line says so and the picture stays.

The tab can render before the profile loads, so the profile loader also
reveals the button once a picture arrives. Also move the avatar hint
into i18n.
2026-10-01 23:21:57 -07:00
dependabot[bot] f387a7aa68 chore(deps): bump fast-uri from 3.1.7 to 3.1.8 (#4014)
Bumps [fast-uri](https://github.com/fastify/fast-uri) from 3.1.7 to 3.1.8.
- [Release notes](https://github.com/fastify/fast-uri/releases)
- [Commits](https://github.com/fastify/fast-uri/compare/v3.1.7...v3.1.8)

---
updated-dependencies:
- dependency-name: fast-uri
  dependency-version: 3.1.8
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-01 18:39:45 -07:00
Daniel Salazar e6bda950f0 perf: batch per-app origin, canonical and hosted-backing lookups in app listings (PUT-1892) (#4013) 2026-10-01 18:30:06 -07:00
Daniel Salazar 4f31388dad fix: cap app lookups and fields, dedupe icon processing, rate-limit app landing pages (PUT-1891) (#4012) 2026-10-01 17:54:25 -07:00
dependabot[bot] 376a047ecc chore(deps): bump @grpc/grpc-js from 1.14.4 to 1.14.5 (#4010)
Bumps [@grpc/grpc-js](https://github.com/grpc/grpc-node) from 1.14.4 to 1.14.5.
- [Release notes](https://github.com/grpc/grpc-node/releases)
- [Commits](https://github.com/grpc/grpc-node/compare/@grpc/[email protected]...@grpc/[email protected])

---
updated-dependencies:
- dependency-name: "@grpc/grpc-js"
  dependency-version: 1.14.5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-01 17:50:14 -07:00
dependabot[bot] fe7b0ce535 chore(deps): bump axios from 1.18.1 to 1.20.0 (#4008)
Bumps [axios](https://github.com/axios/axios) from 1.18.1 to 1.20.0.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v1.18.1...v1.20.0)

---
updated-dependencies:
- dependency-name: axios
  dependency-version: 1.20.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-01 17:49:54 -07:00
dependabot[bot] e85c6917f7 chore(deps): bump fastify from 5.12.3 to 5.12.5 (#4009)
Bumps [fastify](https://github.com/fastify/fastify) from 5.12.3 to 5.12.5.
- [Release notes](https://github.com/fastify/fastify/releases)
- [Commits](https://github.com/fastify/fastify/commits/v5.12.5)

---
updated-dependencies:
- dependency-name: fastify
  dependency-version: 5.12.5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-01 17:49:42 -07:00
Juan Castro 989399a857 Merge branch 'main' into juancastro/put-1896-file-and-sandboxed-iframe-origins-get-a-hard-400-and-cant 2026-10-01 20:30:41 -04:00
Juan Castro 143948ed2c Merge branch 'main' into juancastro/put-1896-file-and-sandboxed-iframe-origins-get-a-hard-400-and-cant 2026-10-01 20:30:00 -04:00
Juan Fernando Castro e48dc0cbf3 Merge pull request #4011 from HeyPuter/juancastro/put-1997-bound-grant-and-token-route-inputs
fix: bound the inputs on the grant and token routes (PUT-1997)
2026-10-01 20:29:42 -04:00
Juan Castro a67cdac380 fix: bound the inputs on the grant and token routes
Three permission routes sized their work by the request body rather than by
the route, so one call could cost far more than its rate-limit slot implies.

- `extra` and `meta` were only type-checked. They ride every row a grant
  writes (up to 16) and are re-serialised into the per-(user, app) cache on
  each miss, so they are now capped at 4 KiB.
- `/auth/create-access-token` took a `permissions` array of any length: one
  permission check and one sequential INSERT each. It now uses the same
  16-per-request cap the grant routes already had, and runs every entry
  through the validator those routes use.
- Withdrawing an app's cross-app data grants looks them up by permission
  text, and no index on `user_to_app_permissions` led with `permission`.
  Indexed per engine, mirroring what mysql_mig_22 / postgres_mig_11 /
  sqlite 0067 did for `user_to_user_permissions`.

`grant-dev-app` validated none of its input — not even the type of `extra` —
so it now runs the same validator as its user-app sibling.

Two paths could also carry a permission wider than the `varchar(255)` column
it lands in. `grantUserAppPermission` already rejected that after rewriting;
`grantDevAppPermission` now does the same, and `createAccessToken` checks it
before the session row a failing INSERT would otherwise orphan.

Caps are published in rate-limits-and-quotas.md. Every in-tree caller sends
one permission and a small `extra`, so none of them change behaviour.
2026-10-01 16:51:39 -04:00
dependabot[bot] c6da95beb0 chore(deps-dev): bump undici from 7.29.0 to 7.30.0 (#3991)
Bumps [undici](https://github.com/nodejs/undici) from 7.29.0 to 7.30.0.
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v7.29.0...v7.30.0)

---
updated-dependencies:
- dependency-name: undici
  dependency-version: 7.30.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-01 09:52:01 -07:00
Daniel Salazar ed7b8f828a fix: kv path errors and ttl hardening, events subscription end and race fixes (#4004) 2026-10-01 09:50:34 -07:00
Juan Castro 29c6f78b7c fix: refuse opaque origins cleanly instead of a logged 400
A page the browser gives no origin to — one opened straight from disk, or an
iframe sandboxed without `allow-same-origin` — serialises its origin as
`"null"`. `appUidFromOrigin` logged that at error level on every attempt and
the GUI sent it on three endpoints, so each refusal cost a 400 and an
error-level line while the user got nothing useful.

An opaque origin cannot name an app to mint a token for, and is not a valid
`postMessage` target to deliver one to, so these clients are refused rather
than supported. Make the refusal quiet, early and legible instead:

- AuthService: drop the `console.error` and say what the caller should do.
  The accept/reject set is unchanged; a 400 is already counted per route.
- puter.js: `signIn()` rejects with `unsupported_origin` before opening
  anything, which covers implicit auth too since it routes through `signIn`.
  The load-time warning now covers sandboxed iframes, console-only — a modal
  belongs in nobody else's embed.
- GUI: a popup whose opener has no attested origin says so and closes,
  instead of rendering a blank window and wedging on a failed exchange.

Also fixes two faults this made reachable: `PuterDialog.open()` prefers the
popup branch, which reaches the `puter` global before the constructor that
assigns it has returned (`puter is not defined`, no SDK at all); and
`showModal()` throws where modals are blocked. `openNotice()` does neither.

Ordinary http(s) sign-in is unaffected.
2026-10-01 11:09:04 -04:00
Juan Fernando Castro c542e4675b Merge pull request #4000 from HeyPuter/juancastro/put-2036-shared-file-issue
fix: let a share recipient create entries in a folder shared with them
2026-10-01 09:56:09 -04:00
Daniel SalazarandReynaldi Chernando d3b46a6b86 docs: easy kv recipes (PUT-1962) (#3999)
* docs: easy kv recipes (PUT-1962)

* docs: publish the append-to-list kv recipe (PUT-1962)

* docs: bare-object add rejects with invalid_path (PUT-1962)

* refine

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-10-01 18:03:07 +07:00
Juan Fernando Castro fd8b8288d3 Merge pull request #3988 from HeyPuter/juancastro/put-1953-get-user-app-token-and-login-500-on-fk-failures-when-a 2026-09-30 20:28:02 -04:00
Daniel Salazar fc62ea2ee1 Revert "variable color for text in input fields (#3996)" (#4001)
This reverts commit 8862420594.
2026-09-30 16:12:19 -07:00
Juan Castro d1b11c793c Merge branch 'main' into juancastro/put-1953-get-user-app-token-and-login-500-on-fk-failures-when-a
One conflict, in workerInvoker.integration.test.ts: #3997 fixed the same
gap-marker flake this branch did, with a different barrier. Took theirs
— waiting on the `ev:qf` counter, which `#handlerFailed` writes strictly
after `discard`, so the marker is in and the lease cleared by the time it
reads 1. Their ordering also asserts the counter before flipping `answer`
back to 200, which closes the redelivery-clears-failures hole this
branch's version had removed the line to avoid.

Converted the two `vi.waitFor` calls #3997 added to `waitUntil`, the
helper this branch introduced for the rest of the file. Neither is wrong
today, since the new block fakes no timers, but a `jump()` added near
them later would hit exactly the drained-backoff trap the helper exists
for.
2026-09-30 17:49:36 -04:00
Juan Castro d6b7c8ff52 docs: drop the replica-lag testing runbook from the open-source tree
It describes our local two-server MySQL replication setup, which is
internal tooling rather than anything a self-hoster or contributor
needs. It now lives in the heyputer repo alongside the other internal
docs.

The suite it documented stays: the header comment now states the
env-var gate directly instead of pointing at a file this repo no
longer carries.
2026-09-30 17:42:04 -04:00
Daniel Salazar 4448103ef0 fix: misc events and kv issues found during recipes (#3997)
* fix: misc events and kv issues found during recipes

* test: cover handler depth on narrowed actors and created notifications

* test: fix stale legacy batch directory upload test and seat user-agent flake

* test: wait for the refusal to be recorded before claiming its gap marker
2026-09-30 14:40:47 -07:00
Juan Castro 79e496e79d test: wait on the gap marker rather than a depth that never moves
The refused-delivery test waited for `depth(subId)` to reach 1 before
claiming the marker, but `discard` removes the entry and appends the
marker in its place, so the depth is 1 before, during and after. The
wait returned at once, and the claim came back `inflight` — the refused
delivery still held the lease it was invoked on — leaving the assertion
reading `undefined`. It only passed on the slack in `invoked`'s poll
interval, which a loaded runner takes away.

Waiting on the claim itself is the barrier the depth was standing in
for. Dropping `answer = 200` with it: `beforeEach` already resets it,
and a redelivery settling before the last assertion would clear the
failure count it reads.
2026-09-30 16:36:57 -04:00
Juan Castro 2e0bf26cac Merge branch 'main' into juancastro/put-1953-get-user-app-token-and-login-500-on-fk-failures-when-a 2026-09-30 16:31:42 -04:00
Juan Castro b126cf9c5a fix: carry the owner's path into WebDAV events and expand tilde before touch
WebDAV emits the same GUI events the FS controllers do, and had the same
confusion: the payload was masked for the actor and addressed to the owner. A
recipient mounting a shared folder in Finder reproduced the disappearing-item
symptom exactly.

/touch prefixed `/` before expanding, so `~/AppData/<app>/f.txt` became a literal
`/~/...` that no longer looked like a tilde path — the shape puter.js sends for
every app-relative write answered 404.
2026-09-30 16:21:39 -04:00
Juan Castro ec4d3b31cf fix: let a share recipient create entries in a folder shared with them
A recipient addresses the owner's entries by a masked `/<owner>/<uuid>/<name>`
path. Routes that resolve an existing row unmask it on the way in, but the ones
that create a new entry had nothing to resolve and ran the ACL check against the
mask itself, which names no row — so mkdir, touch and the batch write/mkdir/
shortcut ops answered 404 `subject_does_not_exist` inside any shared folder.
They now expand client paths the same way every other legacy route does.

The GUI events carried the same confusion the other way: the payload was masked
for whoever made the request but addressed to the entry's owner, who cannot
resolve another user's mask. The owner's client dropped the row by uid and then
failed to re-add it under a path it had never heard of, so anything a recipient
touched vanished from the owner's open window until a refresh. Those events now
publish the owner's real path; responses to the actor stay masked.

Also fixes the desktop's `item.moved` handler reading `metadata` as an object
when the wire carries a JSON string: the name of a trashed item came out empty
(trashing renames the entry to its uid), and re-encoding the string left the
restore path parsing a string instead of an object. The parse the dashboard and
the explorer already open-coded is now one helper.
2026-09-30 15:28:02 -04:00
hairyEagle 8862420594 variable color for text in input fields (#3996) 2026-09-30 11:41:32 -07:00
Neal ShahandReynaldi Chernando 3b4f3f6d06 PUT-1959 PUT-1960 subdomains & worker recipes (#3993)
* PUT-1960 worker recipes

* subdomain recipes

* fixes

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-10-01 01:33:29 +07:00
Juan Fernando CastroandReynaldi Chernando 3d1357455f docs: teams recipe and playground examples (PUT-1966) (#3967)
* docs: add the easy teams recipe and playground examples

Covers the read-only surface an app actually has: list(), listMembers()
and listDirectory(). Adds the teams recipe tag the build validates
against, and a Teams section to the playground.

Each example handles the two states that are easy to confuse: teams
turned off on the deployment (not_found) and a team that has not opened
its directory to apps (team_not_found).

* docs: open the teams recipe with what it lets an app build

* refine

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-10-01 00:11:42 +07:00
Juan Castro f8d8003fea test: stop the delivery-backoff test reading holds against a moving clock
`holds one it could not answer, for longer each time` freezes `Date` so a
hold written as `now + backoff` reads back as exactly `backoff`. But every
wait in the file went through `vi.waitFor`, which advances the faked clock
once per poll — measured at 50ms a poll. Each poll between the hold being
written and read therefore ate into the value: CI read a 2s hold as 525ms
and failed, on main one run and on this branch the next.

Polling now goes through `waitUntil`, which retries on `performance.now`
and real `setTimeout` — neither faked — so the clock only moves where
`jump` moves it. The holds then read exactly 2s/4s/8s/16s, so the
assertion is an equality rather than a one-second window that a slow
enough runner could always slide out of.
2026-09-30 12:25:40 -04:00
Juan Castro 23c8f73bcb test: verify the replica-lag fix against real MySQL replication
The unit tests inject the driver error because sqlite has neither a read
replica nor foreign-key enforcement, so neither half of this bug can
occur locally — they prove the handling, not that the situation arises.

This suite runs two MySQL servers in real replication and freezes the
follower with STOP REPLICA, so the row really is gone on the primary and
really is still served by the replica, and MySQL raises the foreign-key
errors itself. Against the pre-fix code all four cases fail, including
the unhandled "Cannot add or update a child row" the reported 500s were.

Opt-in behind PUTER_TEST_REPLICA_LAG, matching the env-gated provider
integration tests and their skipUnlessEnv helper, so CI and anyone
without the containers skip it. It builds and drops its own database
rather than touching local dev data.
2026-09-30 10:02:36 -04:00
Daniel Salazar b9efac2363 fix: misc fixes from alert logs (#3995) 2026-09-30 00:51:27 -07:00
Daniel Salazar ad12da2942 feat: add gpt 6.1 sol to our direct apis (#3994) 2026-09-29 17:38:40 -07:00
Neal Shah 9aa3989e9d fix PUT-1888 (#3992) 2026-09-29 17:06:28 -04:00
dependabot[bot] cd37cf147f chore(deps): bump socket.io-client from 4.8.3 to 4.8.4 (#3982)
Bumps [socket.io-client](https://github.com/socketio/socket.io) from 4.8.3 to 4.8.4.
- [Release notes](https://github.com/socketio/socket.io/releases)
- [Changelog](https://github.com/socketio/socket.io/blob/main/CHANGELOG.md)
- [Commits](https://github.com/socketio/socket.io/compare/[email protected]@4.8.4)

---
updated-dependencies:
- dependency-name: socket.io-client
  dependency-version: 4.8.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-29 12:56:15 -07:00
Filip Kujundžić f60d94ef3f Merge pull request #3987 from HeyPuter/fix/put-1874-chat-test-mode
Fix chat test mode routing to fake provider
2026-09-29 19:19:44 +02:00
Juan Castro 8bce62e880 Merge branch 'main' into juancastro/put-1953-get-user-app-token-and-login-500-on-fk-failures-when-a 2026-09-29 13:13:01 -04:00
Juan Castro abd27c0192 fix: replicate tombstones, and attribute foreign-key failures correctly
Review of the first pass turned up two holes that made it ineffective.

CacheReplicationService always deleted the keys a peer broadcast and
never adopted a payload, so tombstones stopped at the region boundary:
peers dropped the marker, kept caching deleted rows off their own
replicas, and the multi-region case the fix was for stayed broken.
Tombstones are now the one payload a peer adopts — they carry no row,
only the fact of a delete.

user_to_app_permissions keys both the app and the user, and the drivers
don't all name the constraint, so a deleted *account* was being read as a
deleted app: a live app got tombstoned, the request 404'd for the wrong
reason, and the dead account was never retired, so the client retried and
re-tombstoned indefinitely. The insert site now asks the primary which
parent actually went away and answers 404 or 401 to match.

Also: re-check the tombstone after a cache write lands, since a row
cached under a live tombstone is never looked at again; extend the guard
to #refreshCache, which update() could otherwise use to undo a delete;
stop markDeleted tombstoning an address the deleted row never owned,
which was blocking the account that did own it; let the primary rather
than the tombstone decide in batched lookups; tombstone the id key when
no cached row names the others; and correct the leaf-store comment in
stores/index.ts now that SessionStore depends on UserStore.
2026-09-29 13:12:31 -04:00
dependabot[bot] 6a31b2af9e chore(deps): bump libphonenumber-js from 1.13.13 to 1.13.14 (#3984)
Bumps [libphonenumber-js](https://gitlab.com/catamphetamine/libphonenumber-js) from 1.13.13 to 1.13.14.
- [Changelog](https://gitlab.com/catamphetamine/libphonenumber-js/blob/master/CHANGELOG.md)
- [Commits](https://gitlab.com/catamphetamine/libphonenumber-js/compare/v1.13.13...v1.13.14)

---
updated-dependencies:
- dependency-name: libphonenumber-js
  dependency-version: 1.13.14
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-29 10:11:03 -07:00
dependabot[bot] 3ebb753eb1 chore(deps): bump socket.io from 4.8.3 to 4.8.4 (#3985)
Bumps [socket.io](https://github.com/socketio/socket.io) from 4.8.3 to 4.8.4.
- [Release notes](https://github.com/socketio/socket.io/releases)
- [Changelog](https://github.com/socketio/socket.io/blob/main/CHANGELOG.md)
- [Commits](https://github.com/socketio/socket.io/compare/[email protected]@4.8.4)

---
updated-dependencies:
- dependency-name: socket.io
  dependency-version: 4.8.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-29 10:08:32 -07:00
Juan Fernando CastroandReynaldi Chernando ee4d55027d docs: easy perms recipe (PUT-1965) (#3972)
* docs: add the easy perms recipe

Covers the supported surface: request() and check(). Adds the perms
recipe tag the build validates against.

* docs: add a playground example per operation in the perms recipe

One runnable example for each section: request a folder, request the
email address, request apps or subdomains, and check before prompting.

* docs: open the perms recipe with what it lets an app build

* docs: ground the perms examples in the UI that would ask

A photo editor exporting to Pictures, a checkout emailing a receipt, a
launcher listing apps, and a settings toggle that reads its own state
without raising a prompt.

* refine

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-09-30 00:00:34 +07:00
dependabot[bot] 7eb9ace733 chore(deps-dev): bump prettier from 3.9.8 to 3.9.9 (#3986)
Bumps [prettier](https://github.com/prettier/prettier) from 3.9.8 to 3.9.9.
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/3.9.8...3.9.9)

---
updated-dependencies:
- dependency-name: prettier
  dependency-version: 3.9.9
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-29 09:51:06 -07:00
dependabot[bot] 14ae6a69ee chore(deps): bump postal-mime from 3.0.0 to 3.0.1 (#3983)
Bumps [postal-mime](https://github.com/postalsys/postal-mime) from 3.0.0 to 3.0.1.
- [Release notes](https://github.com/postalsys/postal-mime/releases)
- [Changelog](https://github.com/postalsys/postal-mime/blob/master/CHANGELOG.md)
- [Commits](https://github.com/postalsys/postal-mime/compare/v3.0.0...v3.0.1)

---
updated-dependencies:
- dependency-name: postal-mime
  dependency-version: 3.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-29 09:50:45 -07:00
Juan Fernando CastroandReynaldi Chernando 3b00cd2d14 docs: easy fs sharing recipe (PUT-1958) (#3973)
* docs: add the easy fs sharing recipe

share(), listShared(), getShares() and unshare() — the fs ground the
existing recipes do not already cover.

* docs: add a playground example per operation in the fs sharing recipe

One runnable example for each section of the recipe: share, mode,
listShared, getShares and unshare.

* docs: open the sharing recipe with what it lets an app build

* docs: ground the sharing examples in a document-editor flow

Each example now sits in the UI that would call it — a Share button, a
role picker, a shared-with-me inbox, an access list, ending a review.

* remove redundant examples and refine recipe

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-09-29 23:47:23 +07:00
Juan Castro 785304e807 fix: stop deleted apps and users from being resurrected by a lagging replica
A delete lands on the primary and invalidates the cache, but a concurrent
request that misses the cache reads a replica that hasn't applied it yet,
finds the row, and writes it back under a fresh TTL — 24h for apps, 15m
for users. Every write referencing that resurrected row then fails its
foreign key and escapes as a raw driver error, so the route 500s until
the TTL expires. Origin-bootstrapped app uids are deterministic, so the
same origin keeps hitting the stale row and never re-bootstraps.

Deletes now tombstone their cache keys before dropping them. While a
tombstone is live the read path goes to the primary and the write path
refuses to cache; re-creating a row clears it. As a backstop, a write
rejected for a missing parent is answered as one: 404 for a vanished app,
401 for a vanished account, with the stale entry retired either way.

Also fixes cascadeDelete's identity lookup, which selected a column some
schemas lack and so threw into its catch, skipping the invalidation
entirely; and drops SessionStore's private copy of isUniqueViolation in
favour of the shared helper.
2026-09-29 10:16:48 -04:00
Juan Fernando Castro 8e31d9149a Merge pull request #3966 from HeyPuter/juancastro/put-1994-group-scan-cycle
fix: stop group permission scans re-entering the sharer's own grant (PUT-1994)
2026-09-29 10:12:54 -04:00
Juan Castro 9539c7b553 test: cover team-share access through the real authorization path
The existing team-share suite resolves shares through listings and store
reads, neither of which enters the permission scan, so a scan that never
terminates leaves it green. These ask the question the product asks —
can this member open the file — via ACLService.check.
2026-09-29 09:41:05 -04:00
Juan Fernando Castro 27c4873042 Merge pull request #3971 from HeyPuter/juancastro/put-2033-team-create-read-after-write
fix: read a team back from the primary after writing it (PUT-2033)
2026-09-29 09:06:43 -04:00
404oops 0b1a4283f5 fix(ai): honor chat test mode 2026-09-29 12:17:33 +02:00
Daniel Salazar 1cc091206c fix: PUT-1885 PUT 1950 (#3981)
* fix: app uuid fixes

* fix: bound kv update/add createPaths writes (PUT-1885)

* fix: upload progress socket events on the v2 tracker (PUT-1950)
2026-09-28 23:38:34 -07:00
Daniel Salazar 09887747e0 fix: app uuid fixes (#3980) 2026-09-28 19:43:23 -07:00
Daniel Salazar f80d4ae136 fix: events docs (#3968)
* fix: events docs

* fix: docs
2026-09-28 17:24:37 -07:00
Filip KujundžićandClaude Opus 5.5 42e0c28d0a feat(ai): add Claude Sonnet 5.5 (#3970)
Add claude-sonnet-5-5 ($2/$10 per MTok, 1M context, 128K output) and move
the bare claude-sonnet / claude-sonnet-latest aliases to it. Sonnet 5.5
rejects non-default sampling params and disabled thinking, so it joins the
temperature-omit, effort, and adaptive-thinking gates.

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-09-28 15:29:21 -07:00
Juan Castro d888377138 Merge remote-tracking branch 'origin/main' into juancastro/put-2033-team-create-read-after-write 2026-09-28 17:20:43 -04:00
Juan Castro 8a53d14540 fix: read a team back from the primary after writing it (PUT-2033)
create() inserted on the primary then read the row back through the
replica, so a replica that had not caught up answered the same as a team
that was not there and the call threw. Nothing rolled the insert back,
and the owner membership was never written, leaving a team its owner
could not see but which still counted against their cap.

Read back through pread in create() and update(), seed the row cache
with the result, and delete the row if the read-back still finds
nothing. #cached no longer stores a null, which would otherwise hold a
lagging replica's answer for the full TTL.
2026-09-28 16:55:19 -04:00
Juan Fernando Castro 12e6bc3d71 Merge pull request #3950 from HeyPuter/juancastro/put-1988-team-force-2fa
feat: let a team require two-factor authentication of its accounts (PUT-1988)
2026-09-28 16:52:34 -04:00
Filip Kujundžić 359d68e51f Merge pull request #3949 from HeyPuter/fix/ai-ocr-model-sync-and-normalization 2026-09-28 21:23:28 +02:00
Filip Kujundžić 020e61065f Merge pull request #3947 from AnujShrivastava01/anuj/fix/remove-deprecated-ai-chat-models 2026-09-28 21:22:33 +02:00
Juan Castro 480d9d032b Merge remote-tracking branch 'origin/main' into juancastro/put-1988-team-force-2fa
# Conflicts:
#	src/backend/clients/database/SqliteDatabaseClient.test.ts
#	src/backend/clients/database/SqliteDatabaseClient.ts
#	src/backend/clients/database/migrations/mysql/mysql_mig_42.sql
#	src/backend/clients/database/migrations/postgres/postgres_mig_31.sql
2026-09-28 10:24:39 -04:00
Juan Castro 528eb192b5 fix: stop group permission scans re-entering the sharer's own grant
The group path recursed without the anti-cycle state, so a grant issued
into a group the issuer belongs to resolved back into itself and the
scan did not terminate. Thread the scan state through the group scanner
and skip an issuer already on the path, matching the user-to-user path.

Readings produced under a non-empty path are pruned for that path only,
so they are no longer written to or served from the shared scan cache.
2026-09-28 10:05:54 -04:00
Rupanjana ChoudhuryandDaniel Salazar ca52b5236c Fix/3638 shared link account (#3845)
* fix: open shared links with the intended account

* refactor: validate shared account hints in the GUI

* fix: scope the shared-link account hint to plain top-level visits

The hint no longer switches accounts in popups, embeds or action flows, and
only counts next to a share path the GUI would open. It is dropped from the
URL before any switch or prompt, so a reload never asks again. An unsaved
recipient gets a dismissible choice (sign in, or continue as the current
account) instead of a picker with no way out.

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-09-27 18:25:28 -07:00
Neal Shah 153f581011 fix PUT-1872 (#3952) 2026-09-25 22:34:53 -04:00
Neal Shah 1fc2eae1ae self hosted mail (#3932)
* self hosted mail

* update lock

* address daniel review
2026-09-25 21:48:15 -04:00
Daniel Salazar dcb61bc4fd fix: PUT-1824 + PUT-1884 + PUT-1952 + PUT-1827 (#3951) 2026-09-25 17:47:03 -07:00
Juan Fernando Castro b445680f10 Merge pull request #3937 from HeyPuter/juancastro/put-1879-oidc-popup-return-proof-binding
fix(auth): bind the OIDC popup-return proof to its purpose, browser, and popup
2026-09-25 18:04:59 -04:00
Juan Castro 95ee796157 Merge branch 'main' into juancastro/put-1879-oidc-popup-return-proof-binding 2026-09-25 17:22:12 -04:00
Nariman JelvehandDaniel Salazar 3d3678a555 feat: let the Contact Us form carry screenshots and recordings (#3563)
* feat: let the Contact Us form carry screenshots and recordings

The form doubles as our bug-report channel, and the bugs worth reporting
are often the ones that need a picture: a visual glitch, or something that
takes five steps to reach. Up to 5 images or videos may now ride along
with the message, delivered as attachments on the support email.

Nothing the client says about a file is believed. The endpoint takes bare
base64 and re-derives all three of the things that matter from the decoded
bytes:

- Type, sniffed from magic numbers and matched against an allow-list of
  PNG/JPEG/GIF/WebP and MP4/QuickTime/WebM. A declared MIME is never read,
  so it cannot smuggle anything past the list. SVG is excluded on purpose
  (it carries script, and support tooling renders what it is sent), as are
  the non-video brands that share MP4's `ftyp` box -- HEIC, M4A, JPEG 2000.
- The file name, reduced to a display label with the extension taken from
  the sniffed type. PNG bytes named `payload.html` arrive as
  `payload.png`; a name can never carry the CR/LF that would break out of
  a Content-Disposition header, nor the bidi overrides that make
  `report<RLO>gnp.exe` render as `report.exe.mp4`.
- Size: 10 MB per file, 15 MB per submission, counted on decoded bytes.
  Encoded length is capped before decoding, so an oversized payload costs
  a length check rather than a 10 MB allocation. The total stays well
  under the 25 MB most providers enforce, since the outgoing mail base64s
  these again.

Base64 is decoded strictly, reusing the round-tripping decoder that
already guards app icons -- `Buffer.from(s, 'base64')` silently drops
characters it does not recognise, and the round-trip is what rejects
bytes smuggled after the payload. That decoder and the image sniffer move
from appIcon.ts to a new mediaSniff.ts, which gains the video counterpart.

One request is now worth megabytes of parsing and outbound mail, so the
existing per-user rate limit gains a per-IP backstop (which the per-user
counter cannot see through freshly minted accounts), a concurrency cap,
and a Content-Length gate that refuses an impossible body before anything
decodes it.

Payloads are not stored. They ride the email; the new
`feedback.attachments` column records names, types and sizes only, so an
abusive submission stays attributable once the mail has been dealt with.

Also fixes the form posting an empty message when Send was pressed with
nothing typed, and surfaces submit failures instead of leaving the button
disabled with no explanation.

* fix: stream Contact Us attachments as multipart instead of base64 JSON

Base64 in a JSON body went through the global JSON parser before the route
ran: raw buffer, rawBody copy, decoded string, parsed strings, then decoded
Buffers plus a re-encode for the strict check. A max-size submission peaked
around 110-135 MB of heap for 15 MB of files, and the route's Content-Length
check ran after all of it.

Attachments now arrive as multipart/form-data, which the global parser skips.
Auth, rate limit, concurrency and the Content-Length 413 all run before the
body is read, and busboy enforces the count, per-file and total caps while
streaming, so only the decoded file bytes are held (~16 MB peak). On a broken
limit the reader stops and the response closes the connection.

JSON stays supported for message-only posts; a JSON `attachments` field is
rejected.

* fix: deliver Contact Us 413s instead of resetting the upload

Closing the socket on a mid-stream limit sent the 413 and then reset the
connection with the client still uploading, so the client could see a
reset instead of the error. After a limit trips, the rest of the body is
now read and discarded, bounded by the body budget; past the budget the
request is destroyed.

Adds an HTTP-level test through the full middleware stack (FormData
upload, unauthenticated refusal, 413 on declared length before the body
is sent, 413 on a chunked upload over the per-file cap), trims comments
to the AGENTS.md length rule, and drops a box-drawing test divider.

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-09-25 14:19:35 -07:00
hairyEagle a9ee3d1ea6 fix(selfhost): configure bucket CORS in s3-init so browser uploads work (#3934)
Browser uploads (Dev Center deploy, puter.fs.upload) PUT file bytes
directly to presigned URLs on the public S3 endpoint. That cross-origin
PUT is preflighted, and the bucket had no CORS rules, so RustFS answered
without Access-Control-Allow-Origin and the browser blocked the upload.

s3-init now runs put-bucket-cors on every boot after ensuring the bucket
exists (idempotent). Origins are open because the presigned URL is the
credential and the SDK sends no cookies. Document how to re-run and
verify in doc/self-hosting.md.
2026-09-25 12:12:02 -07:00
Juan Castro ee3f928b0e fix: annotate the seat-lookup catch so the baselined typecheck passes 2026-09-25 14:39:23 -04:00
Juan Castro 176c8848f5 fix: let a seat enrol in 2FA from the console, and hold it once required
The Security tab only rendered the 2FA card for accounts with a confirmed
email, so a seat — provisioned without one by design — had no way to
enrol from the UI at all. It was the GUI twin of the enable-route check
fixed earlier: the rule could be required of accounts that could not
satisfy it. The card now shows for a seat too, giving it the ordinary
authenticator flow: QR, manual secret, six-digit code, recovery codes.

Disabling is refused while the seat's team requires 2FA. Without that a
member could switch off the factor their team mandates and lock
themselves out of everything until they enrolled again.
2026-09-25 14:31:58 -04:00
Juan Castro beaf4a9a7c docs: record the team 2FA rule where the other member-facing gates are
No method pages: requiring 2FA and resetting a member's factor are team
administration, which the docs deliberately do not publish — only
`list`, `listMembers` and `listDirectory` are app-callable.

But an app acting for a gated seat does meet `two_factor_required`, and
an undocumented 403 leaves it nothing to go on. It goes beside
`password_change_required` in rate-limits-and-quotas, which already
explains what a team's administration does to a member's requests.
2026-09-25 14:18:30 -04:00
404oopsandClaude Opus 5.5 d36da7522b fix(ai-ocr): address review: playground examples, Textract options, inline size
Adds the Mistral and document-annotation img2txt examples to the
playground, states that Textract takes no options, and drops the
normalize mentions from the img2txt docs and types.

The SDK now checks Mistral inline sources against the size a base64
upload can reach inside the 50 MB JSON body (36 MB) instead of the
10 MB Textract limit; 50 MB let 38-50 MB inputs through to a body-parser
413. Retired-model messages no longer cite vendor dates.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-25 18:56:29 +02:00
Daniel Salazar e6cfe8c492 fix: codeowners (#3948) 2026-09-25 09:23:26 -07:00
404oopsandClaude Opus 5.5 eef8258539 fix(ai-ocr): sync OCR models and prices, route by model, cap input size
Model sync against Mistral's listing, docs and a live call per id:
mistral-ocr-latest now serves OCR 4.1, so the catalog pins it (with
mistral-ocr-4 and the retired-but-redirected 2503) to mistral-ocr-4-1,
adds mistral-ocr-4-0 and OCR 3 (mistral-ocr-2512 / -3 / -3-0), and
rejects the retired mistral-ocr-2505 before any upstream call. Mistral
usage is metered per model at vendor rates: $4/1000 pages for OCR 4.x
with $5 annotations, $2 and $3 for OCR 3, replacing the flat $1 and $3.

Textract now calls DetectDocumentText instead of AnalyzeDocument with
LAYOUT. The layout feature cost $4/1000 pages while we billed the
$1.50 detect rate, and it only added text-less LAYOUT_* blocks that put
blank lines into img2txt output. Multi-page PDFs and unsupported
formats become a 400 that names Mistral as the alternative.

Request shape: a model name alone now selects its provider, so calls
differ only by model; an explicit provider still picks its default and
must agree with the model. Mistral options gain tableFormat,
extractHeader, extractFooter and documentAnnotationPrompt; annotation
formats accept the REST (json_schema.schema) or SDK
(jsonSchema.schemaDefinition) spelling and are validated before
billing. Non-image documents such as DOCX go as document_url.

Response shape: both providers return model, text and blocks with a
0-based page; Mistral also returns document_annotation, which was
billed but dropped. img2txt resolves to that annotation when
documentAnnotationFormat is set and to the text otherwise; the return
is a string for every model, so normalize has no effect on it.

Input size: OCR input is capped at the provider limit (10 MB Textract,
50 MB Mistral), and web inputs in loadFileInput are streamed under the
cap instead of buffered whole, which also bounds speech2txt URLs.

Docs: img2txt models, options, input limits, return value and errors;
OCR limits in the quotas page.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-25 18:19:54 +02:00
Juan Castro 3105378fdc fix: let a team seat enrol in 2FA, and fit the rule's note on its card
Found by driving the flow against a live instance: a provisioned seat
could reach `configure-2fa/setup` but never `enable`, which demands a
confirmed email. Seats are provisioned without one on purpose (PUT-1792,
the team being the trust anchor), so requiring 2FA of a team would have
locked out precisely the accounts the rule is for — permanently, since a
seat cannot enrol and the rule never lifts.

The check now passes for an account a team provisioned; every other
account still needs its address confirmed, with tests either way.

The card's note was also a line too long and rendered clipped, so it says
the same thing in the space the layout gives it.
2026-09-25 11:58:32 -04:00
Juan Castro 695a26b272 feat: surface the team 2FA rule in the console and SDK (PUT-1988)
An owner toggle beside the directory one, confirmed on the way on since
it locks out every account without 2FA until they enrol, and unconfirmed
on the way off since that only lifts a requirement.

A per-member Reset 2FA action, shown only while the rule is on — with it
off a member can clear their own factor and needs nobody's help. The
confirmation says what it does and what it leaves alone: signed out
everywhere, told about it, password unchanged.

`require2fa` rides the Team shape and `update()`, and `resetTwoFactor()`
joins the module beside `resetPassword()`, whose docblock already says
2FA is left alone — the two now read as the pair they are.
2026-09-25 11:28:59 -04:00
Juan Castro cab70254b1 feat: let a team owner reset a seat's second factor (PUT-1988)
Forcing 2FA makes lockout materially likelier, and nothing could clear a
factor except the account holding it — `resetMemberPassword` leaves 2FA
alone on purpose, and admin only displays it. A seat that lost both its
authenticator and its recovery codes had no way back.

Its own route and its own audit action rather than folded into the
password reset, so that method keeps the property its comment claims: a
password reset alone is not takeover. The two together are, and for an
account the team created and pays for that is the team's to do — but it
has to be chosen and recorded, not arrived at sideways.

Bounded to `org_owned = 1`: a joined member's own login is not a team's
to reach into. The member is told, and signed out everywhere.
2026-09-25 11:15:23 -04:00
Juan Castro db1a95260a feat: let a team require 2FA of the accounts it provisioned (PUT-1988)
`group.require_2fa`, toggled by the owner through `PUT /teams/:uid` and
recorded in the audit log like `directory_enabled` — it changes what
every seat must hold to sign in, so it is not something a team should be
able to alter silently.

Enforced by deriving, not by stamping a flag on the user: a stamp left
behind by a member leaving, or by the rule being turned off, would lock
someone out of their own account with no team left to clear it. The seat
row already joins `group` and is already read per request, so it carries
the flag and the gate costs nothing extra. Toggling busts those rows, or
turning the rule off would leave members locked out until the TTL lapsed.

The gate rides the same `allowUnconfirmed` opt-out as the other pending-
requirement gates, so the routes a gated member must still reach stay
reachable — `configure-2fa` among them, which would otherwise be walled
off behind the very requirement it clears.

Scoped to `org_owned = 1`: the owner is not a seat of their own team, and
a joined member is an ordinary self-paying account whose personal login
is not a team's to dictate. The owner is held to the rule at the point it
matters — they cannot turn it on without 2FA of their own.
2026-09-25 10:35:24 -04:00
Juan Castro ca48477db9 Merge branch 'main' into juancastro/put-1879-oidc-popup-return-proof-binding 2026-09-24 12:19:59 -04:00
Juan Castro d73933b8a5 fix(auth): bind the OIDC popup-return proof to its purpose, browser, and popup
Tokens under the `oidc-state` scope share one signing key, so the payload is
what says which job a token belongs to. The popup-return proof now carries a
`purpose` claim, and each verifier accepts exactly one kind of token.

The return leg also sets a single-use companion cookie, mirroring the nonce
`/start` already uses, and stamps the action its redirect lands on. Redeeming
a proof requires the matching cookie — consumed on success only, so a rejected
call can't invalidate an in-flight return — and the popup honors only a proof
minted for its own action.

The cookie is host-only, so `/auth/oidc/verify-popup-return` is also served on
the GUI origin and the popup redeems it same-origin. The return leg always
lands the popup on `config.origin`, where both the cookie and the route live.
2026-09-23 17:21:35 -04:00
569 changed files with 42457 additions and 10334 deletions

No files matched your search

+5
View File
@@ -16,3 +16,8 @@ MARIADB_PASSWORD=replace-with-strong-password
S3_ACCESS_KEY=puter
S3_SECRET_KEY=replace-with-strong-secret
S3_BUCKET=puter-local
# ── Inbound SMTP (compose profile `smtp`) ------------------------------
# Host port mapped to the receiver's 2525. 25 is what an MX record means;
# many hosting providers block it.
SMTP_PORT=25
+57 -46
View File
@@ -1,61 +1,72 @@
# Default owner, including backend and SDK areas without a specialist below.
# The last matching rule takes precedence.
* @Salazareo
- @Salazareo
# GUI
/src/gui/ @jelveh @Salazareo
# Documentation, developer site, and CLI
/doc/ @reynaldichernando @Salazareo
/src/docs/ @reynaldichernando @Salazareo
/src/dev-center/ @reynaldichernando @Salazareo
/src/cli/ @reynaldichernando @Salazareo
/doc/ @reynaldichernando @Salazareo
/src/docs/ @reynaldichernando @Salazareo
/src/dev-center/ @reynaldichernando @Salazareo
/src/cli/ @reynaldichernando @Salazareo
# Backend AI
/src/backend/controllers/puterai/ @404oops @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/ai-\*/ @404oops @ProgrammerIn-wonderland @Salazareo
# Backend: AI, hosting, workers, networking, and WebDAV
/src/backend/controllers/puterai/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/hosting/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/webdav/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/wisp/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/ai-*/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/util/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/workers/ @ProgrammerIn-wonderland @Salazareo
/src/backend/services/localworker/ @ProgrammerIn-wonderland @Salazareo
/src/backend/stores/subdomain/ @ProgrammerIn-wonderland @Salazareo
/extensions/workerSandbox* @ProgrammerIn-wonderland @Salazareo
/src/worker/ @ProgrammerIn-wonderland @Salazareo
/src/worker-types/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/peer/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/hosting/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/webdav/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/wisp/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/util/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/workers/ @ProgrammerIn-wonderland @Salazareo
/src/backend/services/localworker/ @ProgrammerIn-wonderland @Salazareo
/src/backend/stores/subdomain/ @ProgrammerIn-wonderland @Salazareo
/extensions/workerSandbox\* @ProgrammerIn-wonderland @Salazareo
/src/worker/ @ProgrammerIn-wonderland @Salazareo
/src/worker-types/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/peer/ @ProgrammerIn-wonderland @Salazareo
# Backend: sharing, permissions, and teams
/src/backend/controllers/share/ @jfcastro92 @Salazareo
/src/backend/controllers/team/ @jfcastro92 @Salazareo
/src/backend/services/share/ @jfcastro92 @Salazareo
/src/backend/services/permission/ @jfcastro92 @Salazareo
/src/backend/services/acl/ @jfcastro92 @Salazareo
/src/backend/services/team/ @jfcastro92 @Salazareo
/src/backend/services/apps/AppPermissionService* @jfcastro92 @Salazareo
/src/backend/stores/share/ @jfcastro92 @Salazareo
/src/backend/stores/permission/ @jfcastro92 @Salazareo
/src/backend/stores/group/ @jfcastro92 @Salazareo
/src/backend/stores/team/ @jfcastro92 @Salazareo
/src/backend/controllers/share/ @jfcastro92 @Salazareo
/src/backend/controllers/team/ @jfcastro92 @Salazareo
/src/backend/services/share/ @jfcastro92 @Salazareo
/src/backend/services/permission/ @jfcastro92 @Salazareo
/src/backend/services/acl/ @jfcastro92 @Salazareo
/src/backend/services/team/ @jfcastro92 @Salazareo
/src/backend/services/apps/AppPermissionService\* @jfcastro92 @Salazareo
/src/backend/stores/share/ @jfcastro92 @Salazareo
/src/backend/stores/permission/ @jfcastro92 @Salazareo
/src/backend/stores/group/ @jfcastro92 @Salazareo
/src/backend/stores/team/ @jfcastro92 @Salazareo
# SDK: AI, hosting, workers, and networking
/src/puter-js/src/modules/ai/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/hosting/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Workers* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/networking/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/lib/networkUtils* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/ai.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/hosting.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/workers.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/net.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Peer* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/ai/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/hosting/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Workers* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/networking/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/lib/networkUtils* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/ai.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/hosting.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/workers.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/net.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Peer\* @ProgrammerIn-wonderland @Salazareo
# SDK: sharing and permissions
/src/puter-js/src/modules/perms/ @jfcastro92
/src/puter-js/src/modules/FileSystem/operations/share* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/unshare* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/getShares* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/listShared* @jfcastro92 @Salazareo
/src/puter-js/tests/api/suites/perms.suite.ts @jfcastro92 @Salazareo
/src/puter-js/tests/api/suites/sharing.suite.ts @jfcastro92 @Salazareo
/src/puter-js/src/modules/perms/ @jfcastro92
/src/puter-js/src/modules/FileSystem/operations/share* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/unshare* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/getShares* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/listShared* @jfcastro92 @Salazareo
/src/puter-js/tests/api/suites/perms.suite.ts @jfcastro92 @Salazareo
/src/puter-js/tests/api/suites/sharing.suite.ts @jfcastro92 @Salazareo
+6 -4
View File
@@ -47,6 +47,8 @@ Before opening a PR, scan the diff for:
When in doubt, return less. Auth-, permission-, or data-export-related changes deserve an explicit callout in the PR description.
Commit messages and PR descriptions never credit anyone or mention a bug report. For security fixes, state what the change does mechanically ("scoped tokens do xyz now"), not the vulnerability or how it could be abused.
### Working rules of thumb
- **Run it, don't just compile it.** "It type-checks" is not "it works." Exercise the code path end-to-end at least once.
@@ -65,10 +67,10 @@ root (or merge into the existing file):
```yaml
services:
puter:
pull_policy: never
build:
context: .
puter:
pull_policy: never
build:
context: .
```
Compose merges it automatically. Run `docker compose up -d --build` to build and
+2
View File
@@ -79,6 +79,8 @@ ENV PUTER_CONFIG_PATH=/etc/puter/config.json
ENV NODE_OPTIONS=--enable-source-maps
EXPOSE 4100
# The inbound SMTP receiver, when run from this image with the `smtp` profile.
EXPOSE 2525
USER node
+32
View File
@@ -255,6 +255,38 @@
}
},
// ── Inbound mail (separate process) ─────────────────────────────────
// Mailboxes addressed at one of your own domains. `secret` gates the
// ingress endpoint that files a message into the recipient's ~/.mail.
//
// `localServer` additionally runs an SMTP listener that accepts mail from
// other mail servers and hands each message to that endpoint. It runs as
// its own process - `npm run start:smtp`, or the `smtp` compose profile -
// and stores nothing itself. Without it, nothing delivers to the endpoint
// unless you point something else at it.
//
// The listener is unauthenticated by nature: anything on the internet may
// hand it a message for an address in `localDomains`. It performs no spam,
// SPF, DKIM or DMARC checks. Read the self-hosting guide before exposing
// it.
// "userEmail": {
// "secret": "a long random string",
// "localServer": true,
// "localDomains": ["example.com"],
// // Container-internal by default; publish 25 onto it from outside.
// "localPort": 2525,
// // Defaults to <api_base_url>/email/ingress. The endpoint is served
// // on the `api` subdomain, so when this points somewhere internal,
// // `localIngressHost` has to name that virtual host.
// "localIngressUrl": "http://puter:4100/email/ingress",
// "localIngressHost": "api.example.com",
// "localHostname": "mx.example.com",
// "localMaxRecipients": 50,
// // Concurrent connections. Each message is held in memory while it
// // is delivered, so this also bounds memory use.
// "localMaxClients": 20
// },
// ── Sharing ─────────────────────────────────────────────────────────
// Email a recipient who already has an account about a new share. On
// unless set to false; they opt out via the unsubscribe link, or by
+105
View File
@@ -35,6 +35,12 @@ Optional services (compose profile `ai`, opt-in):
| `puter-ollama` | `ollama/ollama` | Local LLM provider (CPU; GPU passthrough opt-in) |
| `puter-ollama-init` | `ollama/ollama` | One-shot — pulls the default model (`tinyllama`) on first boot |
Optional services (compose profile `smtp`, opt-in):
| Container | Image | Role |
| ------------ | ------------------------ | -------------------------------------------------------- |
| `puter-smtp` | `ghcr.io/heyputer/puter` | Receives mail over SMTP and files it into user mailboxes |
State lives under `./puter/data/<service>/`.
---
@@ -144,6 +150,16 @@ Why these knobs:
- `providers.ollama.enabled: false` — Puter auto-probes a local Ollama at `127.0.0.1:11434` by default; without one running you'd see `ECONNREFUSED` on every boot. To run a bundled Ollama, see [Optional: local LLM (Ollama)](#optional-local-llm-ollama) below.
- `s3.s3Config.forcePathStyle: true` — RustFS / MinIO / fauxqs need path-style URLs (`<endpoint>/<bucket>`). Real AWS S3 wants virtual-hosted (`<bucket>.<endpoint>`) — drop this flag (or set `false`) when you swap to real S3.
- `s3.s3Config.publicEndpoint` — `endpoint` (`http://s3:9000`) only resolves inside the docker network; presigned upload/download URLs handed to the browser need a host-reachable URL. Caddy routes the `s3.<domain>` subdomain to RustFS internally and preserves the Host header end-to-end (required for S3 signature validation), so the browser hits the same port/protocol as the rest of the app — no separate published port, no mixed-content surprises when you turn on TLS. Switch to `https://s3.<your-domain>` once you enable TLS in Step 3. Real AWS S3 doesn't need this — its endpoint is already public; drop the field entirely.
- **Bucket CORS** — the browser uploads file bytes straight to RustFS with a cross-origin `PUT` to the presigned URL (Dev Center deploy, `puter.fs.upload`). That only works if the bucket answers the `OPTIONS` preflight with `Access-Control-Allow-*` headers. The `s3-init` container applies those rules on every boot (`put-bucket-cors`, origins `*` — the signed URL is the credential, and the SDK sends no cookies). If deploys fail with `No 'Access-Control-Allow-Origin' header is present`, re-run it: `docker compose run --rm s3-init`. Verify with:
```bash
curl -sk -X OPTIONS "https://s3.<domain>/<bucket>/test" \
-H "Origin: https://<domain>" \
-H "Access-Control-Request-Method: PUT" \
-H "Access-Control-Request-Headers: content-type" -D - -o /dev/null | grep -i access-control
```
You should see `access-control-allow-origin`, `allow-methods: GET, HEAD, PUT, POST, DELETE`, `allow-headers: *`, and `max-age: 3600`.
- `trust_proxy: 1` — Caddy terminates TLS and forwards `X-Forwarded-For`. Without this, `req.ip` is the docker-network address of the Caddy container instead of the real client IP, which breaks rate limiting and IP-based audit logs. `1` = one trusted hop (Caddy). Bump to `2` if you put Cloudflare in front of Caddy; never set `true` (it trusts every hop and makes XFF forgeable).
> If you ever change `MARIADB_PASSWORD` after first boot, `.env` alone won't update MariaDB — its credentials are baked into `./puter/data/mariadb/` on first init. Either rotate the password inside MariaDB by hand or `docker compose down && rm -rf ./puter/data/mariadb` to start fresh.
@@ -503,6 +519,95 @@ Without `--profile ai`, the `ollama` containers stay down and Puter (with `enabl
For GPU acceleration (NVIDIA), uncomment the `deploy:` block under the `ollama` service in [docker-compose.yml](../docker-compose.yml). Requires `nvidia-container-toolkit` on the host.
## Optional: receiving mail over SMTP
Puter can give each account a mailbox at a domain you control. Two pieces are
involved, and they are configured together under `userEmail`:
- an **ingress endpoint** (`POST /email/ingress`) that files a message into the
recipient's `~/.mail`. It exists whenever `userEmail.secret` is set.
- the **`puter-smtp` receiver**, which accepts mail from other mail servers and
hands each message to that endpoint. It runs as its own process, behind the
`smtp` compose profile.
> **Read this before you expose it.** An inbound mail server is unauthenticated
> by nature: anyone on the internet may hand it a message for any address in
> `localDomains`. Puter performs **no** spam filtering and **no** SPF, DKIM or
> DMARC checks — whatever is delivered is stored in the recipient's mailbox and
> counts against their storage. If that is not acceptable, put a filtering relay
> in front of it and point `localDomains` at that instead.
1. Configure it in `puter/config/config.json`:
```json
"userEmail": {
"secret": "a long random string",
"localServer": true,
"localDomains": ["example.com"]
}
```
`secret` is shared between the two halves; generate it with
`openssl rand -hex 32`. `localDomains` lists the domains this server accepts
mail for — anything else is refused at `RCPT TO`, which is what stops it
being used as an open relay. It has no default, and the receiver refuses to
start without it.
The receiver posts to `<api_base_url>/email/ingress` unless you set
`localIngressUrl`. The endpoint is served on the `api` subdomain, so if you
point it at an internal address to skip the reverse proxy, name that virtual
host too or the request will 404:
```json
"localIngressUrl": "http://puter:4100/email/ingress",
"localIngressHost": "api.example.com"
```
`localIngressHost` defaults to the host of `api_base_url`, which is usually
what you want.
2. Bring it up with the `smtp` profile:
```bash
docker compose --profile smtp up -d
docker compose logs -f puter-smtp
```
3. Point DNS at it:
```dns
mx.example.com. A 203.0.113.10
example.com. MX 10 mx.example.com.
```
Also set a `PTR` record for the address that resolves back to
`mx.example.com` — many senders refuse mail from a server without one.
Inbound TCP port 25 must reach the host. **Most cloud providers block port
25 by default and residential ISPs block it permanently**; if you cannot get
it unblocked, this will not work no matter how it is configured. The
container itself binds 2525 (it runs unprivileged) and compose publishes
`${SMTP_PORT:-25}` onto it, so you can move the host port if you need to.
A message is held in memory while it is being delivered, so peak memory is
bounded by `localMaxClients` (default 20) times the 25 MiB message ceiling.
Lower `localMaxClients` on a small host. There is no queue and no retry
schedule — if delivery fails, the sending server is told to try again later and
retrying is its job.
Mail is stored as `message/rfc822` objects under `~/.mail/objects/<date>/` and
read back through `puter.email` in the SDK.
Every setting is documented in
[config.template.jsonc](../config.template.jsonc). Two deliberate limits, both
in the name of keeping the self-hosted path simple: it does not offer STARTTLS,
so mail arrives in the clear, and it applies no per-sender rate limiting. Put a
filtering relay in front if either matters to you.
Note this is a *receiving* server only: it never sends, and it must not be used
for authenticated submission. Outbound transactional mail is the separate
`email` block above.
## Building from source instead of pulling
To run a local build against the full stack, use a source checkout and complete
+50 -2
View File
@@ -139,8 +139,10 @@ services:
start_period: 5s
s3-init:
# One-shot container that creates the `puter-local` bucket on first
# boot. Exits 0 once the bucket exists; stays exited 0 thereafter.
# One-shot container that creates the `puter-local` bucket and
# applies its CORS rules on every boot (put-bucket-cors is
# idempotent). Without CORS, browser uploads to presigned URLs
# fail the preflight check.
image: amazon/aws-cli:latest
container_name: puter-s3-init
depends_on:
@@ -163,6 +165,26 @@ services:
echo "creating bucket $$bucket"
aws --endpoint-url "$$endpoint" s3 mb "s3://$$bucket"
fi
# Browser uploads (Dev Center deploy, puter.fs.upload) PUT to
# presigned URLs cross-origin, so the bucket must answer the
# OPTIONS preflight. Origins are open because the presigned URL
# itself is the credential and the SDK sends no cookies.
cat > /tmp/cors.json <<'EOF'
{
"CORSRules": [
{
"AllowedOrigins": ["*"],
"AllowedMethods": ["GET", "HEAD", "PUT", "POST", "DELETE"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["ETag", "x-amz-request-id"],
"MaxAgeSeconds": 3600
}
]
}
EOF
aws --endpoint-url "$$endpoint" s3api put-bucket-cors \
--bucket "$$bucket" --cors-configuration file:///tmp/cors.json
echo "bucket CORS configured"
restart: "no"
# ── Optional: local LLM ───────────────────────────────────────────
@@ -264,6 +286,32 @@ services:
retries: 3
start_period: 30s
# Inbound mail. Behind the `smtp` profile, so it only starts when asked for:
# docker compose --profile smtp up -d
# Accepts SMTP from other mail servers and hands each message to the ingress
# endpoint, which files it into the recipient's mailbox. Needs `userEmail`
# configured — see doc/self-hosting.md#optional-receiving-mail-over-smtp.
puter-smtp:
profiles: ["smtp"]
image: ghcr.io/heyputer/puter:main
pull_policy: always
container_name: puter-smtp
restart: unless-stopped
command: ["node", "-r", "./dist/src/backend/telemetry.js", "./dist/src/backend/smtp/index.js"]
depends_on:
puter:
condition: service_healthy
ports:
# The container runs unprivileged and binds 2525; 25 is what an MX record
# means, and many providers block it.
- "${SMTP_PORT:-25}:2525"
environment:
PUID: 1000
PGID: 1000
volumes:
# The same config.json the backend reads; only `userEmail` applies here.
- ./puter/config:/etc/puter:z
caddy:
image: caddy:2.11-alpine
container_name: puter-caddy
+165
View File
@@ -8,6 +8,7 @@ import {
it,
vi,
} from 'vitest';
import { makeActor, type Actor } from '../src/backend/core/actor.ts';
import { runWithContext } from '../src/backend/core/context.ts';
import { PuterServer } from '../src/backend/server.ts';
import { setupTestServer } from '../src/backend/testUtil.ts';
@@ -88,6 +89,80 @@ describe('metering extension — handleMeteringUsage', () => {
(captured.body as Record<string, unknown>).allowanceInfo,
).toBeDefined();
});
describe('usage split across apps', () => {
const seedUsage = async () => {
const user = await seedUser();
const owner = { uuid: user.uuid, id: user.id as number };
const metering = server.services.metering;
await metering.incrementUsage(
makeActor({ user: owner, app: { uid: 'app-mine' } }),
'kv:read',
1,
10,
);
await metering.incrementUsage(
makeActor({ user: owner, app: { uid: 'app-other' } }),
'kv:write',
1,
20,
);
await metering.incrementUsage(
makeActor({ user: owner }),
'kv:list',
1,
40,
);
await server.stores.meteringBuffer.flushCycle();
return owner;
};
const usageAs = async (actor: Actor) => {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleMeteringUsage(makeReq(), res),
);
return captured.body as {
usage: Record<string, unknown> & { total: number };
appTotals: Record<string, unknown>;
allowanceInfo: Record<string, unknown>;
};
};
it('reports only the calling app’s own usage to an app actor', async () => {
const owner = await seedUsage();
const body = await usageAs(
makeActor({ user: owner, app: { uid: 'app-mine' } }),
);
expect(body.usage.total).toBe(10);
expect(Object.keys(body.usage).sort()).toEqual([
'kv:read',
'total',
]);
expect(body.appTotals).toEqual({
'app-mine': { total: 10, count: 1 },
});
expect(body.allowanceInfo).toMatchObject({
remaining: expect.any(Number),
monthUsageAllowance: expect.any(Number),
});
});
it('still reports the whole account to the user', async () => {
const owner = await seedUsage();
const body = await usageAs(makeActor({ user: owner }));
expect(body.usage.total).toBe(70);
expect(body.usage).toHaveProperty('kv:read');
expect(body.usage).toHaveProperty('kv:write');
expect(body.usage).toHaveProperty('kv:list');
expect(body.appTotals).toMatchObject({
'app-mine': { total: 10, count: 1 },
'app-other': { total: 20, count: 1 },
});
});
});
});
describe('metering extension — handleMeteringUsageForApp', () => {
@@ -132,6 +207,96 @@ describe('metering extension — handleMeteringUsageForApp', () => {
),
).rejects.toMatchObject({ statusCode: 404 });
});
describe('scoped to the calling app', () => {
const appUsageAs = async (actor: Actor, appIdOrName: string) => {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleMeteringUsageForApp(
makeReq({ params: { appIdOrName } }),
res,
),
);
return captured.body as { total: number };
};
const seedApps = async () => {
const user = await seedUser();
const owner = { uuid: user.uuid, id: user.id as number };
const mine = `app-${uuidv4()}`;
const other = `app-${uuidv4()}`;
await server.services.metering.incrementUsage(
makeActor({ user: owner, app: { uid: mine } }),
'kv:read',
1,
10,
);
await server.services.metering.incrementUsage(
makeActor({ user: owner, app: { uid: other } }),
'kv:write',
1,
20,
);
await server.stores.meteringBuffer.flushCycle();
return {
owner,
mine,
other,
asMine: makeActor({ user: owner, app: { uid: mine } }),
};
};
it('lets an app read its own usage', async () => {
const { mine, asMine } = await seedApps();
expect((await appUsageAs(asMine, mine)).total).toBe(10);
});
it('refuses an app another app’s usage', async () => {
const { other, asMine } = await seedApps();
await expect(appUsageAs(asMine, other)).rejects.toMatchObject({
statusCode: 403,
});
});
it('refuses an app another app’s usage by name', async () => {
const { owner, other, asMine } = await seedApps();
const name = `other-${uuidv4()}`;
await server.clients.db.write(
'INSERT INTO `apps` (`uid`, `name`, `title`, `index_url`, `owner_user_id`) VALUES (?, ?, ?, ?, ?)',
[other, name, name, `https://${name}.example/`, owner.id],
);
await expect(appUsageAs(asMine, name)).rejects.toMatchObject({
statusCode: 403,
});
});
it('refuses an app the usage outside any app', async () => {
const { asMine } = await seedApps();
await expect(appUsageAs(asMine, 'os-global')).rejects.toMatchObject(
{ statusCode: 403 },
);
});
it('still lets the user read any of their apps', async () => {
const { owner, mine, other } = await seedApps();
const asUser = makeActor({ user: owner });
expect((await appUsageAs(asUser, mine)).total).toBe(10);
expect((await appUsageAs(asUser, other)).total).toBe(20);
});
it('lets the user read their usage outside any app', async () => {
const { owner } = await seedApps();
const asUser = makeActor({ user: owner });
await server.services.metering.incrementUsage(
asUser,
'kv:read',
1,
5,
);
await server.stores.meteringBuffer.flushCycle();
expect((await appUsageAs(asUser, 'os-global')).total).toBe(5);
});
});
});
describe('metering extension — handleMeteringGlobalUsage', () => {
+4 -2
View File
@@ -6,6 +6,7 @@ import {
servicesContainers,
} from '@heyputer/backend/src/exports';
import { extension } from '@heyputer/backend/src/extensions';
import { GLOBAL_APP_KEY } from '@heyputer/backend/src/services/metering/consts';
import {
creditMultiplierFrom,
toCredits,
@@ -151,8 +152,9 @@ export const handleMeteringUsageForApp = async (
let appId = String(req.params.appIdOrName ?? '');
if (!appId) throw new HttpError(400, 'appId parameter is required');
// If not a UUID-shaped app UID, look up by name
if (!appId.startsWith('app-')) {
// If not a UUID-shaped app UID or the global sentinel, look up by name.
// Which apps an actor may read is MeteringService's call, not this route's.
if (!appId.startsWith('app-') && appId !== GLOBAL_APP_KEY) {
const appRows = (await clients.db.read(
'SELECT `uid` FROM `apps` WHERE `name` = ? LIMIT 1',
[appId],
+496 -173
View File
@@ -1,8 +1,10 @@
import {
GetObjectCommand,
HeadObjectCommand,
PutObjectCommand,
type S3Client,
} from '@aws-sdk/client-s3';
import type { Request, Response } from 'express';
import crypto from 'node:crypto';
import {
afterAll,
@@ -12,8 +14,17 @@ import {
it,
vi,
} from 'vitest';
import type { FSController } from '../src/backend/controllers/fs/FSController.ts';
import type {
ClientSignedWriteResponse,
CompleteWriteRequest,
SignedWriteRequest,
} from '../src/backend/controllers/fs/requestTypes.ts';
import type { Actor } from '../src/backend/core/actor.ts';
import { runWithContext } from '../src/backend/core/context.ts';
import { PuterServer } from '../src/backend/server.ts';
import { setupTestServer } from '../src/backend/testUtil.ts';
import { generateDefaultFsentries } from '../src/backend/util/userProvisioning.ts';
import {
handleFsCopyNodeThumbnail,
handleFsRemoveNodeThumbnail,
@@ -27,9 +38,33 @@ const TINY_PNG_BASE64 =
'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII=';
const BUCKET = 'puter-local';
const BUCKET_ENDPOINT = 'http://127.0.0.1:4566/puter-local/';
// Keys the extension will accept back: its own `thumbnails/<uuid>` namespace.
const mintedKey = () => `thumbnails/${crypto.randomUUID()}`;
// Keys the extension mints: `thumbnails/<entry uuid>/<uuid>`.
const mintedKey = (entryUuid: string = crypto.randomUUID()) =>
`thumbnails/${entryUuid}/${crypto.randomUUID()}`;
// Minted before keys were bound to an entry.
const legacyKey = () => `thumbnails/${crypto.randomUUID()}`;
const putObject = async (s3: S3Client, key: string, body: Buffer) => {
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: key,
Body: body,
ContentType: 'image/png',
}),
);
};
const objectExists = async (s3: S3Client, key: string) => {
try {
await s3.send(new HeadObjectCommand({ Bucket: BUCKET, Key: key }));
return true;
} catch {
return false;
}
};
const streamToBuffer = async (
body: { transformToByteArray: () => Promise<Uint8Array> } | undefined,
@@ -40,28 +75,42 @@ const streamToBuffer = async (
describe('thumbnails extension — handleThumbnailCreated', () => {
let server: PuterServer;
let s3: S3Client;
beforeAll(async () => {
server = await setupTestServer();
s3 = server.clients.s3.get();
});
afterAll(async () => {
await server?.shutdown();
});
it('uploads a valid data: URL thumbnail to S3 and rewrites event.url to an s3:// pointer', async () => {
const s3 = server.clients.s3.get();
const event: Record<string, unknown> = {
url: `data:image/png;base64,${TINY_PNG_BASE64}`,
};
const created = async (url: unknown, uuid?: unknown) => {
const event: Record<string, unknown> = { url, uuid };
await handleThumbnailCreated(event, {
s3,
bucketName: BUCKET,
bucketEndpoint: BUCKET_ENDPOINT,
});
return event.url;
};
await handleThumbnailCreated(event, { s3, bucketName: BUCKET });
it('uploads a valid data: URL thumbnail under a key bound to the entry', async () => {
const entryUuid = crypto.randomUUID();
const newUrl = await created(
`data:image/png;base64,${TINY_PNG_BASE64}`,
entryUuid,
);
expect(typeof event.url).toBe('string');
const newUrl = event.url as string;
expect(newUrl.startsWith(`s3://${BUCKET}/`)).toBe(true);
expect(typeof newUrl).toBe('string');
expect(
(newUrl as string).startsWith(
`s3://${BUCKET}/thumbnails/${entryUuid}/`,
),
).toBe(true);
const key = newUrl.slice(`s3://${BUCKET}/`.length);
const key = (newUrl as string).slice(`s3://${BUCKET}/`.length);
const obj = await s3.send(
new GetObjectCommand({ Bucket: BUCKET, Key: key }),
);
@@ -73,34 +122,77 @@ describe('thumbnails extension — handleThumbnailCreated', () => {
});
it('sets event.url to null when the data: URL does not decode to a valid image', async () => {
const s3 = server.clients.s3.get();
const event: Record<string, unknown> = {
url: `data:image/png;base64,${Buffer.from('not an image').toString('base64')}`,
};
await handleThumbnailCreated(event, { s3, bucketName: BUCKET });
expect(event.url).toBeNull();
const url = await created(
`data:image/png;base64,${Buffer.from('not an image').toString('base64')}`,
crypto.randomUUID(),
);
expect(url).toBeNull();
});
it('leaves event.url untouched when the URL is not a data: URL', async () => {
const s3 = server.clients.s3.get();
it('sets event.url to null for a data: URL with no entry to bind it to', async () => {
const url = await created(`data:image/png;base64,${TINY_PNG_BASE64}`);
expect(url).toBeNull();
});
// A documented SDK input: `thumbnail` / `thumbnailGenerator` may return
// a plain URL. It never names our storage, so it is stored as given.
it('leaves an external image URL untouched', async () => {
const original = 'https://example.com/thumb.png';
const event: Record<string, unknown> = { url: original };
await handleThumbnailCreated(event, { s3, bucketName: BUCKET });
expect(event.url).toBe(original);
expect(await created(original, crypto.randomUUID())).toBe(original);
});
it('returns without writing to S3 when event.url is missing', async () => {
const s3 = server.clients.s3.get();
const event: Record<string, unknown> = {};
await handleThumbnailCreated(event, { s3, bucketName: BUCKET });
await handleThumbnailCreated(event, {
s3,
bucketName: BUCKET,
bucketEndpoint: BUCKET_ENDPOINT,
});
expect(event.url).toBeUndefined();
});
it('keeps an uploaded pointer minted for the entry, normalised to our bucket', async () => {
const entryUuid = crypto.randomUUID();
const key = mintedKey(entryUuid);
await putObject(s3, key, Buffer.from(TINY_PNG_BASE64, 'base64'));
expect(
await created(`s3://client-named-bucket/${key}`, entryUuid),
).toBe(`s3://${BUCKET}/${key}`);
});
// A thumbnail key is visible to anyone who can list the entry holding it.
// Planting it on a file of one's own used to hand its deletion to them.
it.each([
['a key minted for another entry', () => mintedKey()],
['a key that predates entry binding', legacyKey],
])('drops a pointer naming %s', async (_label, makeKey) => {
const key = makeKey();
await putObject(s3, key, Buffer.from(TINY_PNG_BASE64, 'base64'));
expect(
await created(`s3://${BUCKET}/${key}`, crypto.randomUUID()),
).toBeNull();
expect(await objectExists(s3, key)).toBe(true);
});
it('drops a pointer whose object was never uploaded', async () => {
const entryUuid = crypto.randomUUID();
expect(
await created(`s3://${BUCKET}/${mintedKey(entryUuid)}`, entryUuid),
).toBeNull();
});
it('drops and removes an uploaded object over the size bound', async () => {
const entryUuid = crypto.randomUUID();
const key = mintedKey(entryUuid);
await putObject(s3, key, Buffer.alloc(2 * 1024 * 1024 + 1));
expect(await created(`s3://${BUCKET}/${key}`, entryUuid)).toBeNull();
expect(await objectExists(s3, key)).toBe(false);
});
});
describe('thumbnails extension — handleThumbnailUploadPrepare', () => {
@@ -148,10 +240,15 @@ describe('thumbnails extension — handleThumbnailUploadPrepare', () => {
expect(item.thumbnailUrl).toBeUndefined();
});
it('skips items whose size exceeds the max thumbnail bytes', async () => {
it.each([
['exceeds the max thumbnail bytes', { size: 999_999_999 }],
['is not declared', {}],
['is not a whole number', { size: 10.5 }],
])('skips items whose size %s', async (_label, sizeField) => {
const item: Record<string, unknown> = {
contentType: 'image/png',
size: 999_999_999,
item_uid: crypto.randomUUID(),
...sizeField,
};
await handleThumbnailUploadPrepare(
{ items: [item] },
@@ -161,10 +258,26 @@ describe('thumbnails extension — handleThumbnailUploadPrepare', () => {
expect(item.thumbnailUrl).toBeUndefined();
});
it('mints a presigned uploadUrl and an s3:// thumbnailUrl for valid items', async () => {
it('skips items with no entry to bind the key to', async () => {
const item: Record<string, unknown> = {
contentType: 'image/png',
size: 1024,
item_uid: 'not-a-uuid',
};
await handleThumbnailUploadPrepare(
{ items: [item] },
{ s3Presign, bucketName: BUCKET },
);
expect(item.uploadUrl).toBeUndefined();
expect(item.thumbnailUrl).toBeUndefined();
});
it('mints a size-bound upload URL for a key bound to the entry', async () => {
const entryUuid = crypto.randomUUID();
const item: Record<string, unknown> = {
contentType: 'image/png',
size: 1024,
item_uid: entryUuid,
};
await handleThumbnailUploadPrepare(
{ items: [item] },
@@ -174,8 +287,15 @@ describe('thumbnails extension — handleThumbnailUploadPrepare', () => {
expect((item.uploadUrl as string).startsWith('http')).toBe(true);
expect(typeof item.thumbnailUrl).toBe('string');
expect(
(item.thumbnailUrl as string).startsWith(`s3://${BUCKET}/`),
(item.thumbnailUrl as string).startsWith(
`s3://${BUCKET}/thumbnails/${entryUuid}/`,
),
).toBe(true);
// A signed `content-length` makes the store refuse any other body size.
const signedHeaders = new URL(
item.uploadUrl as string,
).searchParams.get('X-Amz-SignedHeaders');
expect(signedHeaders).toContain('content-length');
});
});
@@ -196,34 +316,41 @@ describe('thumbnails extension — handleThumbnailRead', () => {
await server?.shutdown();
});
it('rewrites an s3:// thumbnail into a presigned https URL', async () => {
// Seed an object so the presigned URL points at something real
// (the signer itself doesn't validate existence, but this keeps
// the test honest).
const key = mintedKey();
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: key,
Body: Buffer.from(TINY_PNG_BASE64, 'base64'),
ContentType: 'image/png',
}),
);
// Thumbnails stored before keys were bound to entries keep displaying.
it.each([
['an entry-bound key', () => mintedKey()],
['a key that predates entry binding', legacyKey],
])(
'rewrites an s3:// thumbnail naming %s into a presigned https URL',
async (_label, makeKey) => {
// Seed an object so the presigned URL points at something real
// (the signer itself doesn't validate existence, but this keeps
// the test honest).
const key = makeKey();
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: key,
Body: Buffer.from(TINY_PNG_BASE64, 'base64'),
ContentType: 'image/png',
}),
);
const entry: Record<string, unknown> = {
thumbnail: `s3://${BUCKET}/${key}`,
};
await handleThumbnailRead(entry, {
s3,
s3Presign,
bucketName: BUCKET,
bucketEndpoint: 'http://127.0.0.1:4566/puter-local/',
db: stubDb,
});
const entry: Record<string, unknown> = {
thumbnail: `s3://${BUCKET}/${key}`,
};
await handleThumbnailRead(entry, {
s3,
s3Presign,
bucketName: BUCKET,
bucketEndpoint: 'http://127.0.0.1:4566/puter-local/',
db: stubDb,
});
expect(typeof entry.thumbnail).toBe('string');
expect((entry.thumbnail as string).startsWith('http')).toBe(true);
});
expect(typeof entry.thumbnail).toBe('string');
expect((entry.thumbnail as string).startsWith('http')).toBe(true);
},
);
// `fsentries.thumbnail` is writable through the FS API, so a stored
// pointer is attacker input. Signing one the extension didn't mint would
@@ -245,6 +372,10 @@ describe('thumbnails extension — handleThumbnailRead', () => {
`s3://${BUCKET}/thumbnails/../${crypto.randomUUID()}`,
],
['a non-uuid inside the namespace', `s3://${BUCKET}/thumbnails/etc`],
[
'a key nested deeper than the namespace allows',
`s3://${BUCKET}/${mintedKey()}/${crypto.randomUUID()}`,
],
])('refuses to presign a pointer naming %s', async (_label, thumbnail) => {
const entry: Record<string, unknown> = { thumbnail };
await handleThumbnailRead(entry, {
@@ -301,8 +432,9 @@ describe('thumbnails extension — handleThumbnailRead', () => {
});
it('migrates an inline data: URL by uploading to S3 and updating the DB row', async () => {
const entryUuid = crypto.randomUUID();
const entry: Record<string, unknown> = {
uuid: 'fs-entry-uuid',
uuid: entryUuid,
thumbnail: `data:image/png;base64,${TINY_PNG_BASE64}`,
};
@@ -322,9 +454,27 @@ describe('thumbnails extension — handleThumbnailRead', () => {
await Promise.resolve();
expect(stubDb.write).toHaveBeenCalledWith(
'UPDATE `fsentries` SET `thumbnail` = ? WHERE `uuid` = ?',
[expect.stringMatching(/^s3:\/\//), 'fs-entry-uuid'],
[
expect.stringMatching(
new RegExp(`^s3://${BUCKET}/thumbnails/${entryUuid}/`),
),
entryUuid,
],
);
});
it('leaves an inline data: URL as is when there is no entry to bind it to', async () => {
const thumbnail = `data:image/png;base64,${TINY_PNG_BASE64}`;
const entry: Record<string, unknown> = { thumbnail };
await handleThumbnailRead(entry, {
s3,
s3Presign,
bucketName: BUCKET,
bucketEndpoint: BUCKET_ENDPOINT,
db: stubDb,
});
expect(entry.thumbnail).toBe(thumbnail);
});
});
describe('thumbnails extension — handleFsRemoveNodeThumbnail', () => {
@@ -340,66 +490,63 @@ describe('thumbnails extension — handleFsRemoveNodeThumbnail', () => {
await server?.shutdown();
});
it('deletes the S3 object referenced by an s3:// thumbnail URL', async () => {
const key = mintedKey();
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: key,
Body: Buffer.from(TINY_PNG_BASE64, 'base64'),
ContentType: 'image/png',
}),
const remove = (target: { thumbnail?: string | null; uuid?: string }) =>
handleFsRemoveNodeThumbnail(
{ target },
{ s3, bucketName: BUCKET, bucketEndpoint: BUCKET_ENDPOINT },
);
await handleFsRemoveNodeThumbnail(
{ target: { thumbnail: `s3://${BUCKET}/${key}` } },
{ s3, bucketName: BUCKET },
);
it('deletes the object minted for the removed entry', async () => {
const entryUuid = crypto.randomUUID();
const key = mintedKey(entryUuid);
await putObject(s3, key, Buffer.from(TINY_PNG_BASE64, 'base64'));
// GetObject should now error because the key was deleted.
await expect(
s3.send(new GetObjectCommand({ Bucket: BUCKET, Key: key })),
).rejects.toThrow();
await remove({ thumbnail: `s3://${BUCKET}/${key}`, uuid: entryUuid });
expect(await objectExists(s3, key)).toBe(false);
});
// The destructive half of the same confused deputy: the stored pointer
// decides which object is deleted, so a key the extension didn't mint
// must never reach DeleteObject.
it('does not delete an object the pointer names but we did not mint', async () => {
const victimKey = crypto.randomUUID(); // shaped like an fs object key
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: victimKey,
Body: Buffer.from(TINY_PNG_BASE64, 'base64'),
ContentType: 'image/png',
}),
);
// decides which object is deleted, so only a key minted for the removed
// entry may reach DeleteObject.
it.each([
// Shaped like an fs object key.
[
'an object outside the thumbnails namespace',
() => crypto.randomUUID(),
],
// Visible to anyone who can list the entry it belongs to.
["another entry's thumbnail", () => mintedKey()],
// Can't be told apart from a planted or shared pointer.
['a thumbnail that predates entry binding', legacyKey],
])('does not delete %s', async (_label, makeKey) => {
const victimKey = makeKey();
await putObject(s3, victimKey, Buffer.from(TINY_PNG_BASE64, 'base64'));
await handleFsRemoveNodeThumbnail(
{ target: { thumbnail: `s3://${BUCKET}/${victimKey}` } },
{ s3, bucketName: BUCKET },
);
await remove({
thumbnail: `s3://${BUCKET}/${victimKey}`,
uuid: crypto.randomUUID(),
});
const survivor = await s3.send(
new GetObjectCommand({ Bucket: BUCKET, Key: victimKey }),
);
expect(survivor.ContentType).toBe('image/png');
expect(await objectExists(s3, victimKey)).toBe(true);
});
it('does not delete through an external URL that names our key', async () => {
const entryUuid = crypto.randomUUID();
const key = mintedKey(entryUuid);
await putObject(s3, key, Buffer.from(TINY_PNG_BASE64, 'base64'));
await remove({
thumbnail: `https://cdn.example.com/${BUCKET}/${key}`,
uuid: entryUuid,
});
expect(await objectExists(s3, key)).toBe(true);
});
it('is a no-op when the target has no thumbnail', async () => {
// Should not throw or attempt a delete.
await handleFsRemoveNodeThumbnail(
{ target: {} },
{ s3, bucketName: BUCKET },
);
});
it('is a no-op when the thumbnail URL is not an s3:// pointer', async () => {
await handleFsRemoveNodeThumbnail(
{ target: { thumbnail: 'https://cdn.example.com/x.png' } },
{ s3, bucketName: BUCKET },
);
await remove({});
});
});
@@ -416,64 +563,58 @@ describe('thumbnails extension — handleFsCopyNodeThumbnail', () => {
await server?.shutdown();
});
it('duplicates the shared thumbnail object and repoints the copied row', async () => {
const sourceKey = mintedKey();
const body = Buffer.from(TINY_PNG_BASE64, 'base64');
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: sourceKey,
Body: body,
ContentType: 'image/png',
}),
);
const copyUuid = crypto.randomUUID();
const copyNode = async (thumbnail: string | null, copyUuid: string) => {
const db = { write: vi.fn().mockResolvedValue(undefined) };
await handleFsCopyNodeThumbnail(
{
copy: {
thumbnail: `s3://${BUCKET}/${sourceKey}`,
uuid: copyUuid,
},
},
{ s3, bucketName: BUCKET, db },
{ copy: { thumbnail, uuid: copyUuid } },
{ s3, bucketName: BUCKET, bucketEndpoint: BUCKET_ENDPOINT, db },
);
return db;
};
// The copied row was repointed at a fresh object...
expect(db.write).toHaveBeenCalledTimes(1);
const [, params] = db.write.mock.calls[0] as [string, [string, string]];
const [newPointer, updatedUuid] = params;
expect(updatedUuid).toBe(copyUuid);
expect(newPointer.startsWith(`s3://${BUCKET}/thumbnails/`)).toBe(true);
expect(newPointer).not.toBe(`s3://${BUCKET}/${sourceKey}`);
it.each([
['an entry-bound key', () => mintedKey()],
['a key that predates entry binding', legacyKey],
])(
'duplicates a thumbnail naming %s under a key bound to the copy',
async (_label, makeKey) => {
const sourceKey = makeKey();
const body = Buffer.from(TINY_PNG_BASE64, 'base64');
await putObject(s3, sourceKey, body);
// ...whose content matches, while the source object survives — so
// deleting either entry can no longer break the other's thumbnail.
const newKey = newPointer.slice(`s3://${BUCKET}/`.length);
const duplicated = await s3.send(
new GetObjectCommand({ Bucket: BUCKET, Key: newKey }),
);
expect(
(await streamToBuffer(duplicated.Body as never)).equals(body),
).toBe(true);
const original = await s3.send(
new GetObjectCommand({ Bucket: BUCKET, Key: sourceKey }),
);
expect(original.ContentType).toBe('image/png');
});
const copyUuid = crypto.randomUUID();
const db = await copyNode(`s3://${BUCKET}/${sourceKey}`, copyUuid);
// The copied row was repointed at a fresh object of its own...
expect(db.write).toHaveBeenCalledTimes(1);
const [, params] = db.write.mock.calls[0] as [
string,
[string, string],
];
const [newPointer, updatedUuid] = params;
expect(updatedUuid).toBe(copyUuid);
expect(
newPointer.startsWith(`s3://${BUCKET}/thumbnails/${copyUuid}/`),
).toBe(true);
// ...whose content matches, while the source object survives — so
// deleting either entry can no longer break the other's thumbnail.
const newKey = newPointer.slice(`s3://${BUCKET}/`.length);
const duplicated = await s3.send(
new GetObjectCommand({ Bucket: BUCKET, Key: newKey }),
);
expect(
(await streamToBuffer(duplicated.Body as never)).equals(body),
).toBe(true);
expect(await objectExists(s3, sourceKey)).toBe(true);
},
);
it('drops the pointer when the shared object is already gone', async () => {
const copyUuid = crypto.randomUUID();
const db = { write: vi.fn().mockResolvedValue(undefined) };
await handleFsCopyNodeThumbnail(
{
copy: {
thumbnail: `s3://${BUCKET}/${mintedKey()}`, // never uploaded
uuid: copyUuid,
},
},
{ s3, bucketName: BUCKET, db },
const db = await copyNode(
`s3://${BUCKET}/${mintedKey()}`, // never uploaded
copyUuid,
);
expect(db.write).toHaveBeenCalledTimes(1);
@@ -482,27 +623,209 @@ describe('thumbnails extension — handleFsCopyNodeThumbnail', () => {
expect(params).toEqual([copyUuid]);
});
// Only reachable for objects uploaded before upload URLs were size-bound;
// copying one would multiply bytes no allowance counts.
it('drops the pointer instead of duplicating an object over the size bound', async () => {
const sourceKey = legacyKey();
await putObject(s3, sourceKey, Buffer.alloc(2 * 1024 * 1024 + 1));
const copyUuid = crypto.randomUUID();
const db = await copyNode(`s3://${BUCKET}/${sourceKey}`, copyUuid);
expect(db.write).toHaveBeenCalledTimes(1);
const [sql, params] = db.write.mock.calls[0] as [string, [string]];
expect(sql).toContain('NULL');
expect(params).toEqual([copyUuid]);
});
it('does not duplicate an object the pointer names but we did not mint', async () => {
const foreignKey = crypto.randomUUID(); // shaped like an fs object key
const db = { write: vi.fn().mockResolvedValue(undefined) };
await handleFsCopyNodeThumbnail(
{
copy: {
thumbnail: `s3://${BUCKET}/${foreignKey}`,
uuid: crypto.randomUUID(),
},
},
{ s3, bucketName: BUCKET, db },
const db = await copyNode(
`s3://${BUCKET}/${foreignKey}`,
crypto.randomUUID(),
);
expect(db.write).not.toHaveBeenCalled();
});
it('leaves an external image URL to ride along as is', async () => {
const sourceKey = legacyKey();
await putObject(s3, sourceKey, Buffer.from(TINY_PNG_BASE64, 'base64'));
const db = await copyNode(
`https://cdn.example.com/${BUCKET}/${sourceKey}`,
crypto.randomUUID(),
);
expect(db.write).not.toHaveBeenCalled();
});
it('is a no-op when the copy has no thumbnail', async () => {
const db = { write: vi.fn().mockResolvedValue(undefined) };
await handleFsCopyNodeThumbnail(
{ copy: { thumbnail: null, uuid: crypto.randomUUID() } },
{ s3, bucketName: BUCKET, db },
);
const db = await copyNode(null, crypto.randomUUID());
expect(db.write).not.toHaveBeenCalled();
});
});
// The flow `puter.fs.upload()` runs — start, PUT to the signed URLs, complete
// with the returned pointer — through the real controller and the listeners
// this extension registers.
describe('thumbnails extension — signed batch upload through /fs', () => {
let server: PuterServer;
let controller: FSController;
const png = Buffer.from(TINY_PNG_BASE64, 'base64');
beforeAll(async () => {
server = await setupTestServer();
controller = server.controllers.fs as unknown as FSController;
});
afterAll(async () => {
await server?.shutdown();
});
const makeActor = async (): Promise<{ actor: Actor; username: string }> => {
const username = `thumb-${Math.random().toString(36).slice(2, 10)}`;
const user = await server.stores.user.create({
username,
uuid: crypto.randomUUID(),
password: null,
email: `${username}@test.local`,
free_storage: 100 * 1024 * 1024,
requires_email_confirmation: false,
});
await generateDefaultFsentries(
server.clients.db,
server.stores.user,
user,
);
return {
username,
actor: {
user: {
id: user.id,
uuid: user.uuid,
username,
email: user.email ?? null,
email_confirmed: true,
} as Actor['user'],
},
};
};
const call = async <B>(
actor: Actor,
handler: (req: Request, res: Response) => Promise<void>,
body: B,
): Promise<unknown> => {
let captured: unknown;
const res = {
json: (value: unknown) => {
captured = value;
return res;
},
status: () => res,
setHeader: () => res,
};
const req = {
body,
query: {},
headers: { 'content-type': 'application/json' },
actor,
user: { id: actor.user!.id, username: actor.user!.username },
};
await runWithContext({ actor }, () =>
handler.call(
controller,
req as unknown as Request,
res as unknown as Response,
),
);
return captured;
};
const upload = async (
actor: Actor,
path: string,
thumbnailData: (
started: ClientSignedWriteResponse,
) => string | undefined,
) => {
const [started] = (await call<SignedWriteRequest[]>(
actor,
controller.startBatchWrites,
[
{
fileMetadata: { path, size: 4 },
thumbnailMetadata: {
contentType: 'image/png',
size: png.length,
},
},
],
)) as ClientSignedWriteResponse[];
await fetch(started!.url!, { method: 'PUT', body: 'abcd' });
if (started!.thumbnailUploadUrl) {
await fetch(started!.thumbnailUploadUrl, {
method: 'PUT',
body: png,
headers: { 'content-type': 'image/png' },
});
}
const thumbnail = thumbnailData(started!);
await call<CompleteWriteRequest[]>(
actor,
controller.completeBatchWrites,
[
{
uploadId: started!.sessionId,
...(thumbnail === undefined
? {}
: { thumbnailData: thumbnail }),
},
],
);
return {
started: started!,
entry: (await server.stores.fsEntry.getEntryByPath(path))!,
};
};
it('stores the pointer minted for the uploaded entry', async () => {
const { actor, username } = await makeActor();
const { started, entry } = await upload(
actor,
`/${username}/Documents/photo.png`,
(s) => s.thumbnailUrl,
);
expect(started.thumbnailUrl).toMatch(
new RegExp(`^s3://${BUCKET}/thumbnails/${entry.uuid}/`),
);
expect(entry.thumbnail).toBe(started.thumbnailUrl);
});
it("drops a pointer minted for another user's entry", async () => {
const victim = await makeActor();
const { entry: victimEntry } = await upload(
victim.actor,
`/${victim.username}/Documents/photo.png`,
(s) => s.thumbnailUrl,
);
const attacker = await makeActor();
const { entry } = await upload(
attacker.actor,
`/${attacker.username}/Documents/decoy.png`,
() => victimEntry.thumbnail!,
);
expect(entry.thumbnail).toBeNull();
});
it('keeps an external image URL as given', async () => {
const { actor, username } = await makeActor();
const { entry } = await upload(
actor,
`/${username}/Documents/linked.png`,
() => 'https://example.com/thumb.png',
);
expect(entry.thumbnail).toBe('https://example.com/thumb.png');
});
});
+144 -51
View File
@@ -2,11 +2,13 @@ import {
CopyObjectCommand,
DeleteObjectCommand,
GetObjectCommand,
HeadObjectCommand,
PutObjectCommand,
S3Client,
} from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { extension } from '@heyputer/backend/src/extensions';
import { isMissingObjectError } from '@heyputer/backend/src/stores/fs/S3ObjectStore';
import crypto from 'node:crypto';
import sharp from 'sharp';
const clients = extension.import('client');
@@ -22,8 +24,13 @@ const THUMBNAIL_KEY_PREFIX = 'thumbnails/';
const UUID_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
const mintThumbnailKey = (): string =>
`${THUMBNAIL_KEY_PREFIX}${crypto.randomUUID()}`;
const isUuid = (value: unknown): value is string =>
typeof value === 'string' && UUID_PATTERN.test(value);
// `thumbnails/<entry uuid>/<random uuid>`: the entry segment binds the object
// to the entry it was minted for, which is what writes and deletes check.
const mintThumbnailKey = (entryUuid: string): string =>
`${THUMBNAIL_KEY_PREFIX}${entryUuid}/${crypto.randomUUID()}`;
/**
* Extract the object key from a stored thumbnail pointer, or null when the
@@ -31,12 +38,13 @@ const mintThumbnailKey = (): string =>
*
* `fsentries.thumbnail` is writable through the FS API, so neither half of the
* stored string is trusted: the bucket is discarded (callers always pass their
* own) and the key must sit under {@link THUMBNAIL_KEY_PREFIX} with a random
* uuid. Honouring an arbitrary key would lend this extension's storage
* credentials to whatever object the caller named — in the shared-bucket layout
* that is every user's file, since an fs object's key is its fsentry uuid.
* Legacy bare-uuid thumbnails fail the check and are treated as absent; they
* are indistinguishable from a planted pointer, so there is nothing safer to do
* own) and the key must sit under {@link THUMBNAIL_KEY_PREFIX} as either
* `<uuid>` (minted before keys were bound to entries) or `<entry uuid>/<uuid>`.
* Honouring an arbitrary key would lend this extension's storage credentials to
* whatever object the caller named — in the shared-bucket layout that is every
* user's file, since an fs object's key is its fsentry uuid. Bare-uuid keys
* from before the prefix fail the check and are treated as absent; they are
* indistinguishable from a planted pointer, so there is nothing safer to do
* with them than stop signing them.
*/
const resolveThumbnailKey = (pointer: string): string | null => {
@@ -58,10 +66,26 @@ const resolveThumbnailKey = (pointer: string): string | null => {
key = segments.join('/');
}
if (!key.startsWith(THUMBNAIL_KEY_PREFIX)) return null;
const id = key.slice(THUMBNAIL_KEY_PREFIX.length);
return UUID_PATTERN.test(id) ? key : null;
const segments = key.slice(THUMBNAIL_KEY_PREFIX.length).split('/');
return segments.length <= 2 && segments.every(isUuid) ? key : null;
};
// The entry a resolved key was minted for. Null for an unbound key, whose
// holder can't be told apart from a row that copied or planted the pointer.
const thumbnailKeyOwner = (key: string): string | null => {
const segments = key.slice(THUMBNAIL_KEY_PREFIX.length).split('/');
return segments.length === 2 ? segments[0] : null;
};
// Whether a stored thumbnail names an object in our storage. Anything else is
// an external image URL the client supplied: shown as given, never copied or
// deleted.
const isStoragePointer = (pointer: string, bucketEndpoint: string): boolean =>
pointer.startsWith('s3://') ||
// Legacy format — remove after full migration
(pointer.startsWith('https') &&
pointer.includes(new URL(bucketEndpoint).hostname));
// S3 client + bucket config — lazily resolved after boot from config.
let s3Client: S3Client | null = null;
let s3PresignClient: S3Client | null = null;
@@ -144,16 +168,73 @@ async function decodeAndValidateThumbnail(
return { mimeType, data };
}
// A storage pointer a client hands back after a presigned upload is kept only
// if it names a key minted for this entry and an object within the size bound.
async function vetUploadedThumbnail(
pointer: string,
entryUuid: unknown,
deps: { s3: S3Client; bucketName: string },
): Promise<string | null> {
const key = resolveThumbnailKey(pointer);
if (!key || thumbnailKeyOwner(key) !== entryUuid) return null;
let size: number | undefined;
try {
const head = await deps.s3.send(
new HeadObjectCommand({ Bucket: deps.bucketName, Key: key }),
);
size = head.ContentLength;
} catch (err) {
if (!isMissingObjectError(err)) {
console.warn(
'[thumbnails] failed to check uploaded thumbnail',
err,
);
}
return null;
}
if (typeof size !== 'number') return null;
if (size > MAX_THUMBNAIL_BYTES) {
// Only reachable on a store that doesn't enforce the signed length.
// The key is this entry's own, so removing it is safe.
try {
await deps.s3.send(
new DeleteObjectCommand({ Bucket: deps.bucketName, Key: key }),
);
} catch (err) {
console.warn(
'[thumbnails] failed to remove oversized thumbnail',
err,
);
}
return null;
}
return `s3://${deps.bucketName}/${key}`;
}
// -- thumbnail.created -----------------------------------------------
// Intercept data-URL thumbnails before they hit the DB: upload to S3
// and replace the URL with an s3:// pointer.
// and replace the URL with an s3:// pointer bound to the entry.
export async function handleThumbnailCreated(
event: Record<string, unknown>,
deps: { s3: S3Client; bucketName: string },
deps: { s3: S3Client; bucketName: string; bucketEndpoint: string },
): Promise<void> {
const url = event.url;
if (typeof url !== 'string' || !url.startsWith('data:')) return;
if (typeof url !== 'string') return;
if (!url.startsWith('data:')) {
if (isStoragePointer(url, deps.bucketEndpoint)) {
event.url = await vetUploadedThumbnail(url, event.uuid, deps);
}
return;
}
const entryUuid = event.uuid;
if (!isUuid(entryUuid)) {
event.url = null;
return;
}
const decoded = await decodeAndValidateThumbnail(url);
if (!decoded) {
@@ -161,7 +242,7 @@ export async function handleThumbnailCreated(
return;
}
const key = mintThumbnailKey();
const key = mintThumbnailKey(entryUuid);
event.url = `s3://${deps.bucketName}/${key}`;
await deps.s3.send(
@@ -190,21 +271,24 @@ export const handleThumbnailUploadPrepare = async (
typeof item.contentType === 'string' ? item.contentType.trim() : '';
if (!contentType) continue;
if (item.size !== undefined) {
const size = Number(item.size);
if (
!Number.isFinite(size) ||
size < 0 ||
size > MAX_THUMBNAIL_BYTES
)
continue;
}
const size = item.size;
if (
typeof size !== 'number' ||
!Number.isInteger(size) ||
size < 0 ||
size > MAX_THUMBNAIL_BYTES
)
continue;
if (!isUuid(item.item_uid)) continue;
const key = mintThumbnailKey();
const key = mintThumbnailKey(item.item_uid);
const command = new PutObjectCommand({
Bucket: deps.bucketName,
Key: key,
ContentType: contentType,
// Signing covers `content-length`, so a body of any other size
// fails the signature instead of landing.
ContentLength: size,
});
item.uploadUrl = await getSignedUrl(presignClient, command, {
expiresIn: 900,
@@ -227,12 +311,7 @@ export const handleThumbnailRead = async (
if (typeof thumb !== 'string' || !thumb) return;
const presignClient = deps.s3Presign;
if (
thumb.startsWith('s3://') ||
// Legacy format — remove after full migration
(thumb.startsWith('https') &&
thumb.includes(new URL(deps.bucketEndpoint).hostname))
) {
if (isStoragePointer(thumb, deps.bucketEndpoint)) {
const key = resolveThumbnailKey(thumb);
if (!key) {
// Not a pointer we minted — refuse to sign it rather than hand
@@ -247,7 +326,9 @@ export const handleThumbnailRead = async (
);
} else if (thumb.startsWith('data')) {
// Inline data-URL migration: upload to S3 and update the DB entry.
const key = mintThumbnailKey();
const uuid = entry.uuid ?? entry.uid;
if (!isUuid(uuid)) return;
const key = mintThumbnailKey(uuid);
const { mimeType, data } = base64ParseDataUrl(thumb);
const newUrl = `s3://${deps.bucketName}/${key}`;
@@ -261,7 +342,6 @@ export const handleThumbnailRead = async (
);
// Best-effort async DB update
const uuid = entry.uuid ?? entry.uid;
if (uuid) {
deps.db
.write(
@@ -286,24 +366,36 @@ export const handleFsCopyNodeThumbnail = async (
deps: {
s3: S3Client;
bucketName: string;
bucketEndpoint: string;
db: { write: (sql: string, params: unknown[]) => Promise<unknown> };
},
): Promise<void> => {
const copy = payload.copy;
const thumbnailUrl = copy?.thumbnail;
if (!copy || !copy.uuid || typeof thumbnailUrl !== 'string') return;
if (!copy || !isUuid(copy.uuid) || typeof thumbnailUrl !== 'string') return;
if (!isStoragePointer(thumbnailUrl, deps.bucketEndpoint)) return;
// Same trust rule as the read and remove paths: only touch objects this
// extension minted.
const sourceKey = resolveThumbnailKey(thumbnailUrl);
if (!sourceKey) return;
// The copied row points at the SAME S3 object as its source, and
// fs.remove.node deletes the pointed-to object — so the first removal
// among the sharers (an overwrite, a trash purge) would break every
// other sharer's thumbnail. Give the copy an object of its own.
const newKey = mintThumbnailKey();
// The copied row points at the SAME object as its source. Give the copy
// an object of its own, bound to it, so removing either entry leaves the
// other's thumbnail alone.
const newKey = mintThumbnailKey(copy.uuid);
try {
// Nothing larger than an upload may produce gets duplicated; such an
// object came in through an unbounded upload URL.
const head = await deps.s3.send(
new HeadObjectCommand({ Bucket: deps.bucketName, Key: sourceKey }),
);
if (
typeof head.ContentLength !== 'number' ||
head.ContentLength > MAX_THUMBNAIL_BYTES
) {
throw new Error('thumbnail exceeds the size bound');
}
await deps.s3.send(
new CopyObjectCommand({
Bucket: deps.bucketName,
@@ -312,10 +404,8 @@ export const handleFsCopyNodeThumbnail = async (
}),
);
} catch (err) {
// The shared object is already gone (e.g. a sharer was removed
// before this fix existed) — the pointer is dead either way, so
// drop it rather than leave the row advertising a thumbnail it
// doesn't have.
// The shared object is gone or oversized — drop the pointer rather
// than leave the row advertising a thumbnail it doesn't have.
await deps.db.write(
'UPDATE `fsentries` SET `thumbnail` = NULL WHERE `uuid` = ?',
[copy.uuid],
@@ -331,18 +421,18 @@ export const handleFsCopyNodeThumbnail = async (
};
export const handleFsRemoveNodeThumbnail = async (
payload: { target: { thumbnail?: string | null } },
deps: { s3: S3Client; bucketName: string },
payload: { target: { thumbnail?: string | null; uuid?: string } },
deps: { s3: S3Client; bucketName: string; bucketEndpoint: string },
): Promise<void> => {
const thumbnailUrl = payload.target.thumbnail;
if (!thumbnailUrl) return;
const { thumbnail: thumbnailUrl, uuid } = payload.target;
if (!thumbnailUrl || !isStoragePointer(thumbnailUrl, deps.bucketEndpoint))
return;
// Same trust rule as the read path, and load-bearing for the same reason:
// the pointer decides which object gets deleted, so a key we didn't mint
// would let the owner of one file destroy an object belonging to someone
// else just by naming it here.
// The pointer decides which object gets deleted, and a key is visible to
// anyone who can list the entry holding it. Only a key minted for this
// entry is deleted; an unbound one may be shared or planted, so it stays.
const key = resolveThumbnailKey(thumbnailUrl);
if (!key) return;
if (!key || thumbnailKeyOwner(key) !== uuid) return;
await deps.s3.send(
new DeleteObjectCommand({ Bucket: deps.bucketName, Key: key }),
@@ -355,6 +445,7 @@ extension.on(
await handleThumbnailCreated(event, {
s3: getClient(),
bucketName: thumbnailBucketName,
bucketEndpoint: extensionBucketEndpoint,
});
},
);
@@ -397,6 +488,7 @@ extension.on('fs.copy.node', async (_key, payload) => {
{
s3: getClient(),
bucketName: thumbnailBucketName,
bucketEndpoint: extensionBucketEndpoint,
db: clients.db,
},
);
@@ -409,5 +501,6 @@ extension.on('fs.remove.node', async (_key, payload) => {
await handleFsRemoveNodeThumbnail(payload, {
s3: getClient(),
bucketName: thumbnailBucketName,
bucketEndpoint: extensionBucketEndpoint,
});
});
+355
View File
@@ -0,0 +1,355 @@
import { EventEmitter } from 'node:events';
import { Readable } from 'node:stream';
import type { Request, Response } from 'express';
import { beforeEach, describe, expect, test, vi } from 'vitest';
const SECRET = 'ingress-secret';
// Only `extension` is stubbed; the secret comparison and the routing are the
// real code paths, and the fake service/store layer is the boundary.
const state = vi.hoisted(() => ({
config: { userEmail: { secret: 'ingress-secret' } } as Record<
string,
unknown
>,
routes: [] as Array<{
path: string;
handler: (req: unknown, res: unknown) => unknown;
}>,
users: new Map<string, Record<string, unknown>>(),
writes: [] as Array<{
userId: number;
path: string;
homeRegion?: string;
content?: unknown;
}>,
feedback: [] as Array<{ to: string; from: string | null; raw: Buffer }>,
takeFeedback: true,
}));
vi.mock('@heyputer/backend/src/extensions', () => ({
extension: {
get config() {
return state.config;
},
post(
path: string,
_opts: unknown,
handler: (req: unknown, res: unknown) => unknown,
) {
state.routes.push({ path, handler });
},
import(layer: string) {
if (layer === 'client') {
return {
event: {
async emitAndWait(
key: string,
data: {
to: string;
from: string | null;
raw: Buffer;
handled: boolean;
},
) {
if (key !== 'email.ingress.feedback') return;
state.feedback.push(data);
if (state.takeFeedback) data.handled = true;
},
},
};
}
if (layer === 'store') {
return {
user: {
getByUsername: async (username: string) =>
state.users.get(username) ?? null,
},
};
}
return {
fs: {
async write(
userId: number,
{
fileMetadata,
fileContent,
}: {
fileMetadata: { path: string };
fileContent: unknown;
},
_uploadTracker?: unknown,
_storageAllowanceMax?: number,
homeRegion?: string,
) {
state.writes.push({
userId,
path: fileMetadata.path,
homeRegion,
content: fileContent,
});
return { fsEntry: { path: fileMetadata.path } };
},
},
};
},
},
}));
import './index.js';
const permanent = (id: number, username: string) => ({
id,
username,
email: `${username}@example.com`,
password: '$2b$10$hash',
});
/** No password and no email is what makes an account temporary. */
const temp = (id: number, username: string) => ({
id,
username,
email: null,
password: null,
});
const ingress = async (
to: string,
over: { secret?: string; contentLength?: string | null } = {},
) => {
const route = state.routes.find((r) => r.path === '/email/ingress');
if (!route) throw new Error('route not registered');
const body = Buffer.from('From: [email protected]\r\n\r\nhello\r\n');
const req = Readable.from([body], { objectMode: false }) as unknown as {
headers: Record<string, string>;
query: Record<string, string>;
};
// `contentLength: null` sends no length header at all, which is the one
// framing the HTTP parser does not bound for us.
req.headers =
over.contentLength === null
? {}
: {
'content-length': over.contentLength ?? String(body.length),
};
req.query = {
SECRET: over.secret ?? SECRET,
to,
from: '[email protected]',
subject: 'hi',
};
let status = 200;
let ended = false;
// `end` takes (chunk, encoding) and no callback, the way the compression
// middleware leaves it, and emits `finish` — which is what the refusal
// paths hang their drain/destroy on. A stub that accepts a callback
// instead hides the bug that took the 413 path to production as a 500.
const emitter = new EventEmitter();
const res = {
status(code: number) {
status = code;
return res;
},
end(chunk?: unknown, encoding?: BufferEncoding) {
if (chunk) Buffer.byteLength(chunk as string, encoding);
ended = true;
emitter.emit('finish');
},
once(event: string, listener: () => void) {
emitter.once(event, listener);
return res;
},
} as unknown as Response;
await route.handler(req as unknown as Request, res);
return {
status,
ended,
drained: (req as unknown as Readable).readableFlowing,
destroyed: (req as unknown as Readable).destroyed,
};
};
beforeEach(() => {
state.users.clear();
state.writes.length = 0;
state.feedback.length = 0;
state.takeFeedback = true;
});
describe('temporary accounts cannot receive', () => {
test('mail for a temp account is accepted and dropped', async () => {
state.users.set('tempy', temp(1, 'tempy'));
const { status, ended, drained } = await ingress('[email protected]');
// Accepted, so the relay neither retries nor bounces to the sender.
expect(status).toBe(200);
expect(ended).toBe(true);
// The body is read off the wire rather than left unconsumed.
expect(drained).toBe(true);
expect(state.writes).toEqual([]);
});
test('mail for a permanent account still lands in the mailbox', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status } = await ingress('[email protected]');
expect(status).toBe(200);
expect(state.writes).toHaveLength(1);
expect(state.writes[0].userId).toBe(2);
expect(state.writes[0].path).toMatch(/^\/bob\/\.mail\/objects\//);
});
test('an unknown address is still rejected outright', async () => {
const { status } = await ingress('[email protected]');
expect(status).toBe(404);
expect(state.writes).toEqual([]);
});
});
describe('inbound mail is placed in the recipient home region', () => {
/** A permanent account with explicit placement columns. */
const placed = (
id: number,
username: string,
over: { home?: string | null; signup_server?: string | null },
) => ({
...permanent(id, username),
home: null,
signup_server: null,
...over,
});
test("uses the recipient's own home region when they have one", async () => {
state.users.set('bob', placed(2, 'bob', { home: 'frankfurt' }));
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('frankfurt');
});
test('falls back to the server that served their signup', async () => {
// Every account that predates the `home` column is this case.
state.users.set(
'bob',
placed(2, 'bob', { home: null, signup_server: 'london' }),
);
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('london');
});
test('falls back to the primary region when neither is set', async () => {
state.users.set('bob', placed(2, 'bob', {}));
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('oregon');
});
test('home wins over signup_server when both are set', async () => {
state.users.set(
'bob',
placed(2, 'bob', { home: 'sydney', signup_server: 'london' }),
);
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('sydney');
});
});
describe('refusals dispose of the body nobody read', () => {
test('an unknown recipient drains it and keeps the connection', async () => {
// The secret checked out and the length is under the cap, so the body
// was ours to read: discard it rather than resetting a usable socket.
const { status, drained, destroyed } =
await ingress('[email protected]');
expect(status).toBe(404);
expect(drained).toBe(true);
expect(destroyed).toBe(false);
});
test('a bad secret closes rather than reading the upload', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status, destroyed } = await ingress('[email protected]', {
secret: 'not-the-secret',
});
expect(status).toBe(403);
// Nothing authenticated this caller; their bytes are not worth reading.
expect(destroyed).toBe(true);
expect(state.writes).toEqual([]);
});
test('no content-length is refused with 411', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status, destroyed } = await ingress('[email protected]', {
contentLength: null,
});
expect(status).toBe(411);
expect(destroyed).toBe(true);
expect(state.writes).toEqual([]);
});
test('a declared length over the cap is refused with 413', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status, destroyed } = await ingress('[email protected]', {
contentLength: String(25 * 1024 * 1024 + 1),
});
expect(status).toBe(413);
expect(destroyed).toBe(true);
expect(state.writes).toEqual([]);
});
});
describe('feedback addresses', () => {
test('are handed to a listener and reach no mailbox', async () => {
const { status, ended } = await ingress('[email protected]');
expect(status).toBe(200);
expect(ended).toBe(true);
expect(state.feedback).toHaveLength(1);
expect(state.feedback[0].to).toBe('[email protected]');
expect(state.feedback[0].from).toBe('[email protected]');
expect(state.feedback[0].raw.toString()).toContain('hello');
expect(state.writes).toEqual([]);
});
test('match the local part case-insensitively', async () => {
await ingress('[email protected]');
expect(state.feedback).toHaveLength(1);
});
test('fall through to normal routing when nobody takes them', async () => {
state.takeFeedback = false;
state.users.set('abuse', permanent(3, 'abuse'));
const { status } = await ingress('[email protected]');
expect(status).toBe(200);
expect(state.feedback).toHaveLength(1);
// The body was already read, so the buffered copy is what is stored.
expect(state.writes).toHaveLength(1);
expect(Buffer.isBuffer(state.writes[0].content)).toBe(true);
});
test('an unclaimed one with no such account is still a 404', async () => {
state.takeFeedback = false;
const { status } = await ingress('[email protected]');
expect(status).toBe(404);
});
test('only count on a Puter mail domain', async () => {
state.users.set('abuse', permanent(3, 'abuse'));
await ingress('[email protected]');
expect(state.feedback).toEqual([]);
});
test('are refused over the feedback size cap without being read', async () => {
const { status, destroyed } = await ingress('[email protected]', {
contentLength: String(5 * 1024 * 1024 + 1),
});
expect(status).toBe(413);
expect(destroyed).toBe(true);
expect(state.feedback).toEqual([]);
});
});
+161
View File
@@ -0,0 +1,161 @@
import { extension } from '@heyputer/backend/src/extensions';
import { createHmac, randomBytes, timingSafeEqual } from 'node:crypto';
import type { Request } from 'express';
import {
MAX_MESSAGE_BYTES,
isPuterEmailAddress,
isTempUser,
puterEmailUsername,
storeInboxMessage,
} from '@heyputer/backend/src/services/email/mailbox.js';
// copied form peer secrets checker
const COMPARE_KEY = randomBytes(32);
const secretsEqual = (a: string, b: string): boolean =>
timingSafeEqual(
createHmac('sha256', COMPARE_KEY).update(a).digest(),
createHmac('sha256', COMPARE_KEY).update(b).digest(),
);
const USER_EMAIL_CONFIG = (extension.config as Record<string, unknown>)
.userEmail as { secret?: string; feedbackAddresses?: unknown } | undefined;
const INGRESS_SECRET = USER_EMAIL_CONFIG?.secret;
/**
* Local parts that take complaint feedback rather than naming a mailbox. The
* same names are reserved at signup, so no account can be handed them.
*/
const DEFAULT_FEEDBACK_ADDRESSES = ['fbl', 'abuse', 'postmaster'];
const FEEDBACK_ADDRESSES = new Set(
(Array.isArray(USER_EMAIL_CONFIG?.feedbackAddresses)
? USER_EMAIL_CONFIG.feedbackAddresses
: DEFAULT_FEEDBACK_ADDRESSES
).map((local) => String(local).toLowerCase()),
);
/**
* Feedback is handed over whole, so it is read into memory; a complaint report
* quotes one message and has no business near the mailbox ceiling.
*/
const MAX_FEEDBACK_BYTES = 5 * 1024 * 1024;
const isFeedbackAddress = (to: string): boolean =>
isPuterEmailAddress(to) &&
FEEDBACK_ADDRESSES.has(puterEmailUsername(to).toLowerCase());
const readBody = async (req: Request): Promise<Buffer> => {
const chunks: Buffer[] = [];
for await (const chunk of req) chunks.push(chunk as Buffer);
return Buffer.concat(chunks);
};
if (!INGRESS_SECRET) {
console.warn(
'[useremailrx] config.userEmail.secret unset - email ingress disabled',
);
} else {
extension.post(
'/email/ingress',
{
subdomain: 'api',
},
async (req, res) => {
// Answer first, then deal with the body nobody read. Touching the
// request before the response has flushed takes the response down
// with the connection and the caller sees a reset with no status,
// so this hangs off `finish` — and it has to be the event rather
// than `res.end(cb)`, because the compression middleware replaces
// `res.end` with an `(chunk, encoding)` signature that has no
// callback parameter, so a callback lands in `chunk` and dies in
// `Buffer.byteLength`.
//
// `drain` discards the rest of a body we were entitled to read and
// leaves the connection reusable. `close` is for bodies we do not
// want to read at all: an unauthenticated caller's, or one whose
// declared length is missing or over the cap.
const refuse = (status: number, how: 'drain' | 'close') => {
res.once('finish', () =>
how === 'drain' ? req.resume() : req.destroy(),
);
res.status(status).end();
};
const presented = req.query.SECRET;
if (
typeof presented !== 'string' ||
!secretsEqual(presented, INGRESS_SECRET)
) {
// Nothing has authenticated this caller, so refuse to spend
// bandwidth reading whatever they were uploading.
refuse(403, 'close');
return;
}
const size = Number(req.headers['content-length']);
if (!Number.isInteger(size) || size <= 0) {
refuse(411, 'close');
return;
}
if (size > MAX_MESSAGE_BYTES) {
refuse(413, 'close');
return;
}
const stores = extension.import('store');
const services = extension.import('service');
const to = String(req.query.to ?? '');
let content: Buffer | Request = req;
if (isFeedbackAddress(to)) {
if (size > MAX_FEEDBACK_BYTES) {
refuse(413, 'close');
return;
}
const raw = await readBody(req);
const event = {
to,
from:
typeof req.query.from === 'string'
? req.query.from
: null,
raw,
handled: false,
};
await extension
.import('client')
.event.emitAndWait('email.ingress.feedback', event, {});
if (event.handled) {
res.end();
return;
}
// Nobody took it: route it like any other message.
content = raw;
}
const user = await stores.user.getByUsername(
puterEmailUsername(to),
);
if (!user) {
refuse(404, 'drain');
return;
}
if (isTempUser(user)) {
if (content === req) req.resume();
res.end();
return;
}
await storeInboxMessage(services.fs, user, {
subject: req.query.subject,
content,
size: Buffer.isBuffer(content) ? content.length : size,
});
const { SECRET: _secret, ..._loggableQuery } = req.query;
// console.log('got email: ', puterUser, loggableQuery, size);
res.end();
},
);
}
+12
View File
@@ -0,0 +1,12 @@
{
"name": "emailrx",
"version": "1.0.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "emailrx",
"version": "1.0.0"
}
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"name": "emailrx",
"version": "1.0.0",
"main": "index.ts",
"description": "Internal service which handles routing puter emails to users"
}
+99 -1
View File
@@ -8,7 +8,7 @@ import {
it,
vi,
} from 'vitest';
import { makeActor } from '../src/backend/core/actor.ts';
import { makeActor, type Actor } from '../src/backend/core/actor.ts';
import { runWithContext } from '../src/backend/core/context.ts';
import { configContainer } from '../src/backend/exports.ts';
import { PuterServer } from '../src/backend/server.ts';
@@ -285,6 +285,104 @@ describe('whoami extension — handleWhoami', () => {
expect(body.directories).toBeUndefined();
});
describe('fields added by whoami.details listeners', () => {
const addAccountDetails = (
_key: unknown,
event: { details: Record<string, unknown> },
) => {
Object.assign(event.details, {
subscribed: true,
paid_storage: 1024,
subscription: { tier: 'pro', status: 'trialing' },
accountEligible: true,
});
};
const whoamiWithListener = async (
actor: Actor,
listener: typeof addAccountDetails = addAccountDetails,
) => {
const { res, captured } = makeRes();
server.clients.event.on('whoami.details', listener);
try {
await runWithContext({ actor }, () =>
handleWhoami(makeReq(), res),
);
} finally {
server.clients.event.off('whoami.details', listener);
}
return captured.body as Record<string, unknown>;
};
it('reach an app actor only as the documented subscribed / paid_storage', async () => {
const user = await seedUser();
const body = await whoamiWithListener(
makeActor({
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
);
expect(body.subscribed).toBe(true);
expect(body.paid_storage).toBe(1024);
expect(body).not.toHaveProperty('subscription');
expect(body).not.toHaveProperty('accountEligible');
// Fields the handler built itself still ship.
expect(body.username).toBe(user.username);
expect(body.app_name).toBe('app-test-actor');
});
it('are withheld from an app reached through an access token it issued', async () => {
const user = await seedUser();
const owner = { uuid: user.uuid, id: user.id as number };
const body = await whoamiWithListener(
makeActor({
user: owner,
accessToken: {
uid: 'tok-1',
issuer: makeActor({
user: owner,
app: { uid: 'app-issuer' },
}),
},
}),
);
expect(body.subscribed).toBe(true);
expect(body).not.toHaveProperty('subscription');
expect(body).not.toHaveProperty('accountEligible');
});
it('all reach a user actor', async () => {
const user = await seedUser();
const body = await whoamiWithListener(
makeActor({ user: { uuid: user.uuid, id: user.id as number } }),
);
expect(body).toMatchObject({
subscribed: true,
paid_storage: 1024,
subscription: { tier: 'pro', status: 'trialing' },
accountEligible: true,
});
});
it('cannot add taskbar_items for an app actor, which core omits', async () => {
const user = await seedUser();
const body = await whoamiWithListener(
makeActor({
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
(_key, event) => {
event.details.taskbar_items = [{ name: 'injected' }];
},
);
expect(body).not.toHaveProperty('taskbar_items');
});
});
it('redacts tmp_password from metadata for user actors', async () => {
const user = await seedUser();
await server.stores.user.updateMetadata(user.id as number, {
+35 -13
View File
@@ -71,6 +71,13 @@ const SENSITIVE_KEYS: ReadonlySet<string> = new Set([
'audit_metadata',
]);
// What an app actor may receive from `whoami.details` listeners. Anything
// else a listener adds describes the account, and only ships to user actors.
const APP_VISIBLE_LISTENER_KEYS: ReadonlySet<string> = new Set([
'subscribed',
'paid_storage',
]);
// Depth-limited, cycle-safe walk deleting every SENSITIVE_KEYS entry it finds
// at any level (`metadata` and `taskbar_items` are both nested structures).
const scrubSensitive = (
@@ -186,19 +193,6 @@ export const handleWhoami = async (
// this endpoint is polled, and a mint is a write.
referral_code: user.referral_code,
oidc_only: oidcOnly,
taskbar_items: isUser
? await getTaskbarItems(
user,
{
clients,
stores,
services,
apiBaseUrl: String(extension.config.api_base_url ?? ''),
config: extension.config,
},
{ iconSize, noIcons },
)
: undefined,
otp: !!user.otp_enabled,
feature_flags,
created_ts: toUnixSeconds(user.timestamp),
@@ -227,6 +221,21 @@ export const handleWhoami = async (
}
}
// Taskbar items — only sent to user actors
if (isUser) {
details.taskbar_items = await getTaskbarItems(
user,
{
clients,
stores,
services,
apiBaseUrl: String(extension.config.api_base_url ?? ''),
config: extension.config,
},
{ iconSize, noIcons },
);
}
// Directories — only sent to user actors
if (isUser) {
const directories: Record<string, unknown> = {};
@@ -292,6 +301,12 @@ export const handleWhoami = async (
details.app_name = app.uid;
}
// A key core left undefined isn't built, so a listener can't fill it for an app.
const builtKeys = isUser
? null
: new Set(
Object.keys(details).filter((key) => details[key] !== undefined),
);
try {
await clients.event.emitAndWait(
'whoami.details',
@@ -301,6 +316,13 @@ export const handleWhoami = async (
} catch {
/* best-effort */
}
if (builtKeys) {
for (const key of Object.keys(details)) {
if (!builtKeys.has(key) && !APP_VISIBLE_LISTENER_KEYS.has(key)) {
delete details[key];
}
}
}
const subscription = details.subscription as
{ offering?: Record<string, unknown> } | undefined;
+110 -79
View File
@@ -16,7 +16,7 @@
"@aws-sdk/s3-request-presigner": "^3.1028.0",
"dedent": "^1.5.3",
"javascript-time-ago": "^2.5.11",
"libphonenumber-js": "1.13.13",
"libphonenumber-js": "1.13.14",
"miniflare": "^4.20260617.1",
"open": "^10.1.0"
},
@@ -2043,9 +2043,9 @@
"license": "MIT"
},
"node_modules/@eslint/config-array/node_modules/brace-expansion": {
"version": "1.1.18",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.18.tgz",
"integrity": "sha512-Edep/X9fGqVNmzKBVsDYIOtD+z1tuezV70LBjdCst9Tqu76lsnvRiZ6oTic1n+/BIwX6QDGAO94PN4N2SADvtw==",
"version": "1.1.21",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.21.tgz",
"integrity": "sha512-9zeA+KLZNNzglF2TPKRQEDyx6Yby7daAkuy8MiPzpXPsYDWi/DRM8jmwUDxokQjYqBpv5DgPiwD4h4ZZSy1Ujw==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -2124,9 +2124,9 @@
"license": "MIT"
},
"node_modules/@eslint/eslintrc/node_modules/brace-expansion": {
"version": "1.1.18",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.18.tgz",
"integrity": "sha512-Edep/X9fGqVNmzKBVsDYIOtD+z1tuezV70LBjdCst9Tqu76lsnvRiZ6oTic1n+/BIwX6QDGAO94PN4N2SADvtw==",
"version": "1.1.21",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.21.tgz",
"integrity": "sha512-9zeA+KLZNNzglF2TPKRQEDyx6Yby7daAkuy8MiPzpXPsYDWi/DRM8jmwUDxokQjYqBpv5DgPiwD4h4ZZSy1Ujw==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -2412,9 +2412,9 @@
}
},
"node_modules/@grpc/grpc-js": {
"version": "1.14.4",
"resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-1.14.4.tgz",
"integrity": "sha512-k9Dj3DV/itK9D06Y8f190Qgop7/Ui+D0njFV3LHMPwPT75DpXLQohE9Wmz0QElrJnzsjB7KPWiKJbOl7IPDArQ==",
"version": "1.14.5",
"resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-1.14.5.tgz",
"integrity": "sha512-7VZM+SVdEcUUqSQeNI3zM8Qs/BhQKZndPo2h5VkYkAM8Iz0wJIa8mKV5ekQGqG8UUsnkQ0NMxIxwkIHYvj0qOw==",
"license": "Apache-2.0",
"dependencies": {
"@grpc/proto-loader": "^0.8.0",
@@ -6153,6 +6153,17 @@
"@types/node": "*"
}
},
"node_modules/@types/smtp-server": {
"version": "3.5.13",
"resolved": "https://registry.npmjs.org/@types/smtp-server/-/smtp-server-3.5.13.tgz",
"integrity": "sha512-S3rGl2KbViH+98/CgHipPIWgtAFnjxLsppxIRbgJHNuZL7Y+py+7kZjT7xS+wOmCxYTn4O7IqLqkvekg99MDVQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/node": "*",
"@types/nodemailer": "*"
}
},
"node_modules/@types/tedious": {
"version": "4.0.14",
"resolved": "https://registry.npmjs.org/@types/tedious/-/tedious-4.0.14.tgz",
@@ -7185,13 +7196,13 @@
}
},
"node_modules/axios": {
"version": "1.18.1",
"resolved": "https://registry.npmjs.org/axios/-/axios-1.18.1.tgz",
"integrity": "sha512-3nTvFlvpn9Zu/RkHUqtc7/+al4UpRW5az71ap5zccp6e8RAYEzhMTecX8Dz1wWDYrPpUoB1HAQEGEAEvUr7S9g==",
"version": "1.20.0",
"resolved": "https://registry.npmjs.org/axios/-/axios-1.20.0.tgz",
"integrity": "sha512-r8aOh8j9cGKpgQAqpzrUHnSIc6a59Y3Xf/cv8sy1DrHCkZHzQGEuoq1tARk6qSyDdtQGSDgpb9kFlruzPvrgwg==",
"license": "MIT",
"dependencies": {
"follow-redirects": "^1.16.0",
"form-data": "^4.0.5",
"form-data": "^4.0.6",
"https-proxy-agent": "^5.0.1",
"proxy-from-env": "^2.1.0"
}
@@ -7272,15 +7283,6 @@
],
"license": "MIT"
},
"node_modules/base64id": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/base64id/-/base64id-2.0.0.tgz",
"integrity": "sha512-lGe34o6EHj9y3Kts9R4ZYs/Gr+6N7MCaMlIFA3F1R2O5/m7K06AxfSeO5530PEERE6/WyEg3lsuyw4GHlPZHog==",
"license": "MIT",
"engines": {
"node": "^4.5.0 || >= 5.9"
}
},
"node_modules/baseline-browser-mapping": {
"version": "2.11.21",
"resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.21.tgz",
@@ -7506,9 +7508,9 @@
"license": "MIT"
},
"node_modules/brace-expansion": {
"version": "5.0.9",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.9.tgz",
"integrity": "sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg==",
"version": "5.0.12",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.12.tgz",
"integrity": "sha512-YovQ3rzhaLMIrDjNDMkNS01tea93qhEhG5xy8f6+R0l+dw3Ki+5sCoIoI942iuLZTHWogWktgwVDhU09iNEimQ==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -8756,16 +8758,15 @@
}
},
"node_modules/engine.io": {
"version": "6.6.9",
"resolved": "https://registry.npmjs.org/engine.io/-/engine.io-6.6.9.tgz",
"integrity": "sha512-clKkw4C7nJ22mGgoVcCg6V/W/TxdNyIOTr89k2ONZu81qqkddPFDF0LXcbAwhzPD8DjkiRCjzuiO6Y+fkpD4vg==",
"version": "6.6.11",
"resolved": "https://registry.npmjs.org/engine.io/-/engine.io-6.6.11.tgz",
"integrity": "sha512-z34mGpLoY/82ZbkX1LqGMIaf3wjVMPL6gVC681QDqnmzJ26synI8qAE32P+a7UazmpJZ8UGrFIeAavgRfXzYrg==",
"license": "MIT",
"dependencies": {
"@types/cors": "^2.8.12",
"@types/node": ">=10.0.0",
"@types/ws": "^8.5.12",
"accepts": "~1.3.4",
"base64id": "2.0.0",
"cookie": "~0.7.2",
"cors": "~2.8.5",
"debug": "~4.4.1",
@@ -9181,9 +9182,9 @@
"license": "MIT"
},
"node_modules/eslint/node_modules/brace-expansion": {
"version": "1.1.18",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.18.tgz",
"integrity": "sha512-Edep/X9fGqVNmzKBVsDYIOtD+z1tuezV70LBjdCst9Tqu76lsnvRiZ6oTic1n+/BIwX6QDGAO94PN4N2SADvtw==",
"version": "1.1.21",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.21.tgz",
"integrity": "sha512-9zeA+KLZNNzglF2TPKRQEDyx6Yby7daAkuy8MiPzpXPsYDWi/DRM8jmwUDxokQjYqBpv5DgPiwD4h4ZZSy1Ujw==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -9542,9 +9543,9 @@
"license": "Unlicense"
},
"node_modules/fast-uri": {
"version": "3.1.7",
"resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.7.tgz",
"integrity": "sha512-dOvZVzjdZdz7phd9v6jCbwxrBW3fK6n8Rc0CtdmM4bumzMnxywBYhuph6J819RRw/ku+rLbelwfMunktuzVVHg==",
"version": "3.1.8",
"resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.8.tgz",
"integrity": "sha512-GZMtZUTNRpOVIECoXwLNZS5xUGE+mVNbTB8h/7Rwh2TFWcBQiPzTgyZi05BF9UMZKkLJv8XBRJTlU7zg8+ZfMg==",
"funding": [
{
"type": "github",
@@ -9568,9 +9569,9 @@
}
},
"node_modules/fastify": {
"version": "5.12.3",
"resolved": "https://registry.npmjs.org/fastify/-/fastify-5.12.3.tgz",
"integrity": "sha512-reZ8wce5VNCcufIt9AVtzZa3L4u1j8esikn7OEgHWLVpRpL5R7Y2+Xzj70OUkv5zDfzUAxXZT6cu4Rt0zr3EKA==",
"version": "5.12.5",
"resolved": "https://registry.npmjs.org/fastify/-/fastify-5.12.5.tgz",
"integrity": "sha512-OB2k1dlxs5/NAABqeKV2FUHkSD2BbENsCak8yULVcymn3fHIPDVa9TI3SDnJSWYSllZmSYuZXy2gTnsT+Sut1A==",
"funding": [
{
"type": "github",
@@ -10797,6 +10798,12 @@
"node": ">= 10"
}
},
"node_modules/ipv6-normalize": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/ipv6-normalize/-/ipv6-normalize-1.0.1.tgz",
"integrity": "sha512-Bm6H79i01DjgGTCWjUuCjJ6QDo1HB96PT/xCYuyJUP9WFbVDrLSbG4EZCvOCun2rNswZb0c3e4Jt/ws795esHA==",
"license": "MIT"
},
"node_modules/is-binary-path": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/is-binary-path/-/is-binary-path-2.1.0.tgz",
@@ -11492,9 +11499,9 @@
}
},
"node_modules/libphonenumber-js": {
"version": "1.13.13",
"resolved": "https://registry.npmjs.org/libphonenumber-js/-/libphonenumber-js-1.13.13.tgz",
"integrity": "sha512-hV6SmWnyQ8U1QXhhqbqYBHIStNnscG7A5XbGodgdHhf/8N5PwhYpJA5U/yHDjYwPYTd8uNGXqq416XctuvK2PA==",
"version": "1.13.14",
"resolved": "https://registry.npmjs.org/libphonenumber-js/-/libphonenumber-js-1.13.14.tgz",
"integrity": "sha512-llihgCcx0BFLksecLP+x1J+6JDE1GsXS1RN/LoPF6qcwpeQcnjj0lcvZxY8AzbEpYwyZWPZW/nDuqkqzm3amiw==",
"license": "MIT"
},
"node_modules/light-my-request": {
@@ -14225,9 +14232,9 @@
}
},
"node_modules/postal-mime": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/postal-mime/-/postal-mime-3.0.0.tgz",
"integrity": "sha512-Z4a9ar2Bv3YpK3IXag+Yda30k7bMZfpRuUGyqtHnZ2pjHG8Bl62EhZIk4n1dzv00gfzP9g+94e9kd8+XmjVWLA==",
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/postal-mime/-/postal-mime-3.0.1.tgz",
"integrity": "sha512-N86EnYOxZjW1uHQIgSXnnRdnJz+wxmtDsnk3C4R/9rQ1utj8A2znauXZ3Q3+stCHxG5ckY4/NrO09WiPU6rcdQ==",
"license": "MIT-0"
},
"node_modules/postcss": {
@@ -14336,9 +14343,9 @@
}
},
"node_modules/prettier": {
"version": "3.9.8",
"resolved": "https://registry.npmjs.org/prettier/-/prettier-3.9.8.tgz",
"integrity": "sha512-WRFq3Wn3WId7LLROfMLdH7xaFr2jR62wU8nLO6rQUOLOxNZUviyJQs1M0iIhLexSFy+L+w0ch66wtoO2jRjG0A==",
"version": "3.9.9",
"resolved": "https://registry.npmjs.org/prettier/-/prettier-3.9.9.tgz",
"integrity": "sha512-Z/CJHIkdujO/OtN7nXUii0Rf3VT5SRuhjBA82Xvu2XhBUgX3nhP67T0LHceBdQLex7OOFGTox+Q5Yg8Jk2Qivg==",
"dev": true,
"license": "MIT",
"bin": {
@@ -14488,6 +14495,15 @@
"node": ">=6"
}
},
"node_modules/punycode.js": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/punycode.js/-/punycode.js-2.3.1.tgz",
"integrity": "sha512-uxFIHU0YlHYhDQtV4R9J6a52SLx28BCjT+4ieh7IGbgwVJWO+km431c4yRlREUAsAmt/uMjQUyQHNEPf0M39CA==",
"license": "MIT",
"engines": {
"node": ">=6"
}
},
"node_modules/puter-mcp-connector": {
"resolved": "src/mcp-connector",
"link": true
@@ -14845,9 +14861,9 @@
"license": "MIT"
},
"node_modules/rimraf/node_modules/brace-expansion": {
"version": "2.1.4",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.4.tgz",
"integrity": "sha512-hGfVzPxthbf3+2yjg/RBs60cB0FhqBS/zvdV/4wn4/BmN0bNMMHPc4V/BbFieqf1TKAGGAHnY4eSjajCl0f2Xg==",
"version": "2.1.7",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.7.tgz",
"integrity": "sha512-uZbew1NqdmPDTMJ8ah1y+b+9QEJrfkXFk3RcTQw3X0jW/xRUvFKsg1CfQdSYGdTbXZWExtU3J3ccxtnfw1Fi0g==",
"license": "MIT",
"dependencies": {
"balanced-match": "^1.0.0"
@@ -15530,14 +15546,36 @@
"integrity": "sha512-bLGGlR1QxBcynn2d5YmDX4MGjlZvy2MRBDRNHLJ8VI6l6+9FUiyTFNJ0IveOSP0bcXgVDPRcfGqA0pjaqUpfVg==",
"license": "MIT"
},
"node_modules/smtp-server": {
"version": "3.19.13",
"resolved": "https://registry.npmjs.org/smtp-server/-/smtp-server-3.19.13.tgz",
"integrity": "sha512-W1CjNcrPqW4+1gUOYRyjPvTOxu7GgKR9+7l50yQBF0GM8v8jpYwheHFCHX7s+p7pYI8iXKQQZiF2Rv2Na/3i2g==",
"license": "MIT-0",
"dependencies": {
"ipv6-normalize": "1.0.1",
"nodemailer": "10.0.10",
"punycode.js": "2.3.1"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/smtp-server/node_modules/nodemailer": {
"version": "10.0.10",
"resolved": "https://registry.npmjs.org/nodemailer/-/nodemailer-10.0.10.tgz",
"integrity": "sha512-He9XskOFms62SyAKkLu8CcGcYHAo+BSHSsORI8s4PJH8qZgZY4L4lDfvyN0twvmbpyG+eGrxpyBlskj9d9rJ8A==",
"license": "MIT-0",
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/socket.io": {
"version": "4.8.3",
"resolved": "https://registry.npmjs.org/socket.io/-/socket.io-4.8.3.tgz",
"integrity": "sha512-2Dd78bqzzjE6KPkD5fHZmDAKRNe3J15q+YHDrIsy9WEkqttc7GY+kT9OBLSMaPbQaEd0x1BjcmtMtXkfpc+T5A==",
"version": "4.8.4",
"resolved": "https://registry.npmjs.org/socket.io/-/socket.io-4.8.4.tgz",
"integrity": "sha512-9ZV9zg8vN8hO3Y25DhR7nUkJ1VDnBhq5O/JrXhFT9s6+F0gqytSASTHVfCFc2mJXaTfARroErplXlzFWwETZqQ==",
"license": "MIT",
"dependencies": {
"accepts": "~1.3.4",
"base64id": "~2.0.0",
"cors": "~2.8.5",
"debug": "~4.4.1",
"engine.io": "~6.6.0",
@@ -15576,9 +15614,9 @@
}
},
"node_modules/socket.io-client": {
"version": "4.8.3",
"resolved": "https://registry.npmjs.org/socket.io-client/-/socket.io-client-4.8.3.tgz",
"integrity": "sha512-uP0bpjWrjQmUt5DTHq9RuoCBdFJF10cdX9X+a368j/Ft0wmaVgxlrjvK3kjvgCODOMMOz9lcaRzxmso0bTWZ/g==",
"version": "4.8.4",
"resolved": "https://registry.npmjs.org/socket.io-client/-/socket.io-client-4.8.4.tgz",
"integrity": "sha512-7wjKAvVXYOJsbmUgkWbnhPD4bFpkqrUR0++XOjpGFmqDqsqV1CI93VTraeExLxpjxEqrWgFoSRZu1ihjLWfkAA==",
"license": "MIT",
"dependencies": {
"@socket.io/component-emitter": "~3.1.0",
@@ -16133,9 +16171,9 @@
"license": "MIT"
},
"node_modules/test-exclude/node_modules/brace-expansion": {
"version": "2.1.4",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.4.tgz",
"integrity": "sha512-hGfVzPxthbf3+2yjg/RBs60cB0FhqBS/zvdV/4wn4/BmN0bNMMHPc4V/BbFieqf1TKAGGAHnY4eSjajCl0f2Xg==",
"version": "2.1.7",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.7.tgz",
"integrity": "sha512-uZbew1NqdmPDTMJ8ah1y+b+9QEJrfkXFk3RcTQw3X0jW/xRUvFKsg1CfQdSYGdTbXZWExtU3J3ccxtnfw1Fi0g==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -16538,9 +16576,9 @@
"license": "MIT"
},
"node_modules/undici": {
"version": "7.29.1",
"resolved": "https://registry.npmjs.org/undici/-/undici-7.29.1.tgz",
"integrity": "sha512-RYONW2MeafgYlkVOKYKkA/Ag7BmXqgIWCa8t1m0JcxrQg9pI9lEqRhAOruOBCbAohOa/gkCF+iPi9hrgvTzu6Q==",
"version": "7.30.0",
"resolved": "https://registry.npmjs.org/undici/-/undici-7.30.0.tgz",
"integrity": "sha512-dkrQXeHSaoamnItlYbmzG0wFYrM0ZwDxCIg0A7aKjTyyhh9svRzCNFEzV+Vm05/yehjCzjDZ31KXfGEjYSztDQ==",
"license": "MIT",
"engines": {
"node": ">=20.18.1"
@@ -17338,16 +17376,6 @@
"dev": true,
"license": "MIT"
},
"node_modules/wrangler/node_modules/undici": {
"version": "7.29.0",
"resolved": "https://registry.npmjs.org/undici/-/undici-7.29.0.tgz",
"integrity": "sha512-IDxfleLmmbSskfWSUATiN1nfn2rDuvnMOqb5CWR92iIfojA0Ud+ulOAAEQ57LPr9rWmsreUyf5lwyao+7GNNVw==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=20.18.1"
}
},
"node_modules/wrangler/node_modules/workerd": {
"version": "1.20260915.1",
"resolved": "https://registry.npmjs.org/workerd/-/workerd-1.20260915.1.tgz",
@@ -17624,7 +17652,7 @@
"@pagerduty/pdjs": "^2.2.4",
"@smithy/node-http-handler": "^2.5.0",
"@socket.io/redis-streams-adapter": "^0.3.1",
"axios": "^1.15.0",
"axios": "^1.20.0",
"bcrypt": "^6.0.0",
"better-sqlite3": "^12.6.0",
"busboy": "^1.6.0",
@@ -17650,9 +17678,11 @@
"openai": "^6.34.0",
"otpauth": "^9.5.2",
"pg": "^8.21.0",
"postal-mime": "^3.0.1",
"replicate": "^1.0.0",
"sharp": "^0.35.4",
"socket.io": "^4.8.3",
"smtp-server": "^3.19.13",
"socket.io": "^4.8.4",
"svg-captcha": "^1.4.0",
"together-ai": "^0.33.0",
"ua-parser-js": "^1.0.41",
@@ -17670,6 +17700,7 @@
"@types/node": "^24.13.6",
"@types/nodemailer": "^8.0.1",
"@types/pg": "^8.6.1",
"@types/smtp-server": "^3.5.13",
"@types/validator": "^13.15.10",
"pgmock": "^1.0.3",
"typescript": "^5.9.3",
@@ -18437,14 +18468,14 @@
},
"src/puter-js": {
"name": "@heyputer/puter.js",
"version": "2.6.3",
"version": "2.6.4",
"license": "Apache-2.0",
"dependencies": {
"@heyputer/kv.js": "^0.2.1",
"open": "^10.2.0",
"path-browserify": "1.0.1",
"postal-mime": "^3.0.0",
"socket.io-client": "^4.8.3"
"postal-mime": "^3.0.1",
"socket.io-client": "^4.8.4"
},
"devDependencies": {
"@playwright/test": "^1.49.0",
+3 -2
View File
@@ -50,6 +50,7 @@
"build:workerLib:coverage": "cd src/puter-js && npm run build:coverage && cd ../worker && npm run build",
"start:gui": "nodemon --exec \"node dev-server.js\" ",
"start": "node ./tools/start.mjs",
"start:smtp": "node ./tools/start.mjs --smtp",
"dev": "npm start",
"build": "npm run setupExtensions && npm run build:ts && cd src/gui && node ./build.js && cd ../puter-js && npm run build",
"build:workerLib": "cd src/puter-js && npm run build && cd ../worker && npm run build",
@@ -76,7 +77,7 @@
"@aws-sdk/s3-request-presigner": "^3.1028.0",
"dedent": "^1.5.3",
"javascript-time-ago": "^2.5.11",
"libphonenumber-js": "1.13.13",
"libphonenumber-js": "1.13.14",
"miniflare": "^4.20260617.1",
"open": "^10.1.0"
},
@@ -96,6 +97,6 @@
"workerd": true
},
"overrides": {
"undici": "^7.29.0"
"undici": "^7.30.0"
}
}
@@ -184,7 +184,8 @@ describe('compareMigrationFilenames', () => {
//
// `read()` normally goes to the replica batcher; when the replica side is
// degraded (batcher load-shed or a transient connection error) and a real
// replica is configured, the read retries once on the primary batcher.
// replica is configured, the read retries once on the primary read batcher.
// The write batcher is left unset, so a failover that reached it would throw.
type Batcher = { execute: ReturnType<typeof vi.fn> };
@@ -200,7 +201,7 @@ const makeClient = (opts: {
// inject the batchers directly. Configuration enum: SINGLE=0, REPLICA=1.
Object.assign(client as unknown as Record<string, unknown>, {
dbReplica: opts.replica,
db: opts.primary,
dbPrimaryRead: opts.primary,
configuration: opts.multiNode === false ? 0 : 1,
});
return client;
@@ -284,6 +285,10 @@ const startClient = async (
// must answer with one row-set per statement plus one for the SELECT 1.
const rowSets = (...sets: unknown[][]) => [[...sets, [{ 1: 1 }]], undefined];
// Writes go out one statement at a time, so the driver answers with just the
// result header.
const writeHeader = (header: Record<string, unknown>) => [header, undefined];
beforeEach(() => {
createdPools.length = 0;
createPoolMock.mockClear();
@@ -303,6 +308,9 @@ describe('MySQLDatabaseClient — pool construction', () => {
expect(createdPools[0].poolConfig).toEqual({
maxPreparedStatements: 900,
connectionLimit: 30,
maxIdle: 29,
idleTimeout: 60_000,
gracefulEnd: true,
enableKeepAlive: true,
host: 'db.internal',
port: 3307,
@@ -313,6 +321,32 @@ describe('MySQLDatabaseClient — pool construction', () => {
});
});
// mysql2 skips its idle reaper unless maxIdle < connectionLimit, and an
// unreaped connection is closed by the server under the pool's feet.
it('keeps the idle reaper on when a pool sets its own connection limit', async () => {
await startClient({
replica: { host: 'replica.internal', connectionLimit: 10 },
});
expect(createdPools[1].poolConfig).toMatchObject({
connectionLimit: 10,
maxIdle: 9,
idleTimeout: 60_000,
});
});
// mysql2 reads 0 as unlimited; a maxIdle of -1 would crash its reaper.
it('leaves maxIdle to mysql2 for an unlimited pool', async () => {
await startClient({
replica: { host: 'replica.internal', connectionLimit: 0 },
});
expect(createdPools[1].poolConfig).toMatchObject({
connectionLimit: 0,
});
expect(createdPools[1].poolConfig).not.toHaveProperty('maxIdle');
});
it('falls back to loopback defaults when the endpoint is unspecified', async () => {
await startClient();
@@ -401,10 +435,30 @@ describe('MySQLDatabaseClient — query interface', () => {
]);
});
it('sends a write on its own, outside a transaction', async () => {
const client = await startClient();
const pool = createdPools[0];
pool.respond = () => writeHeader({ affectedRows: 1 });
await client.write('UPDATE `user` SET `username` = ? WHERE `id` = ?', [
'ada',
1,
]);
expect(pool.calls).toEqual([
{
sql: 'UPDATE `user` SET `username` = ? WHERE `id` = ?',
values: ['ada', 1],
},
]);
expect(pool.connections[0].beginTransaction).not.toHaveBeenCalled();
expect(pool.connections[0].commit).not.toHaveBeenCalled();
});
it('maps the mysql result header onto the shared write result', async () => {
const client = await startClient();
createdPools[0].respond = () =>
rowSets({ insertId: 42, affectedRows: 2 } as unknown as unknown[]);
writeHeader({ insertId: 42, affectedRows: 2 });
await expect(
client.write('UPDATE `user` SET `username` = ? WHERE `id` = ?', [
@@ -420,7 +474,7 @@ describe('MySQLDatabaseClient — query interface', () => {
it('reports no rows affected when the header omits the counters', async () => {
const client = await startClient();
createdPools[0].respond = () => rowSets({} as unknown as unknown[]);
createdPools[0].respond = () => writeHeader({});
await expect(client.write('DELETE FROM `user`')).resolves.toEqual({
insertId: 0,
@@ -432,17 +486,14 @@ describe('MySQLDatabaseClient — query interface', () => {
it('builds an INSERT with quoted identifiers and positional params', async () => {
const client = await startClient();
const pool = createdPools[0];
pool.respond = () =>
rowSets({ insertId: 7, affectedRows: 1 } as unknown as unknown[]);
pool.respond = () => writeHeader({ insertId: 7, affectedRows: 1 });
await expect(
client.insert('user', { username: 'ada', email: null }),
).resolves.toMatchObject({ insertId: 7 });
expect(pool.calls[0]).toEqual({
sql:
'INSERT INTO `user` (`username`, `email`) VALUES (?, ?); ' +
'SELECT 1',
sql: 'INSERT INTO `user` (`username`, `email`) VALUES (?, ?)',
values: ['ada', null],
});
});
@@ -30,6 +30,8 @@ import { compareMigrationFilenames } from './migrationFilenames.js';
import type { IConfig } from '../../types';
const DEFAULT_SELECT_TIMEOUT_MS = 30_000;
const DEFAULT_CONNECTION_LIMIT = 30;
const IDLE_TIMEOUT_MS = 60_000;
const replicaFailoverCounter = metrics
.getMeter('puter-backend')
@@ -54,7 +56,7 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
private replicaPool!: Pool;
private db!: SQLBatcher;
private dbReplica!: SQLBatcher;
/** Primary pool, SELECT-only: same rows as `db`, without the transaction. */
/** Primary pool, SELECT-only: same rows as `db`, coalesced like the replica. */
private dbPrimaryRead!: SQLBatcher;
private configuration = Configuration.SINGLE;
private shutdownStarted = false;
@@ -136,7 +138,7 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
throw error;
}
replicaFailoverCounter.add(1);
result = await this.db.execute(query, params);
result = await this.dbPrimaryRead.execute(query, params);
}
if (!result) return [];
return (result[0] as Record<string, unknown>[]) ?? [];
@@ -300,9 +302,19 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
// ------------------------------------------------------------------
private createPool(poolConfig: PoolConfig): Pool {
const connectionLimit =
poolConfig.connectionLimit ?? DEFAULT_CONNECTION_LIMIT;
const pool = createPool({
maxPreparedStatements: 900,
connectionLimit: 30,
connectionLimit,
// mysql2 only reaps idle connections when maxIdle < connectionLimit.
// Unreaped, they sit until the server's wait_timeout closes them,
// and the next statement sent on one fails. 0 means unlimited,
// and a negative maxIdle would crash the reaper.
...(connectionLimit > 1 && { maxIdle: connectionLimit - 1 }),
idleTimeout: IDLE_TIMEOUT_MS,
// Reaped connections say goodbye instead of dropping the socket.
gracefulEnd: true,
enableKeepAlive: true,
...poolConfig,
multipleStatements: true,
@@ -330,21 +342,20 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
return pool;
}
/**
* One write per flush. Coalesced writes share a transaction, so unrelated
* callers hold each other's row locks until COMMIT and deadlock; a lone
* statement needs no BEGIN/COMMIT round trips.
*/
private createPrimaryBatcher(pool: Pool): SQLBatcher {
return new SQLBatcher(pool, {
maxTimeInQueue: 30,
maxBatchSize: 5,
maxBatchSize: 1,
poolLabel: 'primary',
acquireTimeoutMs: this.config.database?.acquireTimeoutMs,
});
}
/**
* Reads that must see the primary still only read, so they skip the
* batcher's transaction wrapper. Worth a separate batcher because BEGIN and
* COMMIT are round trips: against a primary in another region they cost
* more than the query.
*/
/** Reads that must see the primary, still coalesced. */
private createPrimaryReadBatcher(pool: Pool): SQLBatcher {
return new SQLBatcher(pool, {
maxTimeInQueue: 30,
+58 -24
View File
@@ -258,16 +258,11 @@ export class SQLBatcher {
}
if (batch.length === 0) return;
const query = `${batch.map((b) => b.sql.replace(/;+\s*$/, '')).join(';')}; SELECT 1`; // SELECT 1 forces mysql2 to return array
const values = batch.map((b) => b.values ?? []).flat();
let connection;
try {
connection = await this.#acquireConnection();
} catch (error) {
this.#consecutiveFailures++;
this.#lastFailureAt = Date.now();
flushFailureCounter.add(1, this.#metricAttrs);
this.#recordFlushFailure();
console.warn(
'SQLBatcher could not acquire connection for flush:',
error,
@@ -278,6 +273,14 @@ export class SQLBatcher {
return;
}
if (batch.length === 1 && !this.readOnly) {
await this.#flushLoneWrite(batch[0], connection);
return;
}
const query = `${batch.map((b) => b.sql.replace(/;+\s*$/, '')).join(';')}; SELECT 1`; // SELECT 1 forces mysql2 to return array
const values = batch.map((b) => b.values ?? []).flat();
// Run the coalesced multi-statement inside an explicit transaction so
// a single bad statement (e.g. a duplicate-key INSERT) rolls back the
// whole batch atomically, leaving us free to re-run each item
@@ -333,7 +336,7 @@ export class SQLBatcher {
async () => {
while (cursor < batch.length) {
const i = cursor++;
settled[i] = await this.#runFallbackItem(batch[i]);
settled[i] = await this.#runItem(batch[i]);
}
},
);
@@ -367,7 +370,34 @@ export class SQLBatcher {
}
}
// Run one fallback item, retrying transient failures with backoff.
// A single statement is atomic under autocommit, so it skips BEGIN/COMMIT
// and their round trips. With no transaction to roll back, a failure is
// not re-run wholesale; only the per-item retry policy applies.
async #flushLoneWrite(item, connection) {
const r = await this.#runItem(item, connection);
if (r.ok) {
this.#consecutiveFailures = 0;
item.resolve(r.value);
return;
}
// An error the server answered with (duplicate key, bad SQL) means it
// is up; only failing to reach it counts toward the breaker.
if (isRetriableError(r.error)) {
this.#recordFlushFailure();
} else {
this.#consecutiveFailures = 0;
flushFailureCounter.add(1, this.#metricAttrs);
}
item.reject(r.error);
}
#recordFlushFailure() {
this.#consecutiveFailures++;
this.#lastFailureAt = Date.now();
flushFailureCounter.add(1, this.#metricAttrs);
}
// Run one item on its own, retrying transient failures with backoff.
// A read-only batcher may retry anything transient; a batcher that
// carries writes only retries failures where the statement provably
// did not apply — either it never reached the server, or the server
@@ -378,33 +408,37 @@ export class SQLBatcher {
// than surfacing: an item is a single statement, so a deadlock victim has
// been fully undone, and the caller sees an unhandled 500 for what the
// database is telling us to just run again.
async #runFallbackItem(b) {
async #runItem(b, connection) {
let attempt = 0;
while (true) {
let connection;
try {
connection = await this.#acquireConnection();
} catch (error) {
return { ok: false, error };
if (!connection) {
try {
connection = await this.#acquireConnection();
} catch (error) {
return { ok: false, error };
}
}
let error;
try {
return {
ok: true,
value: await connection.query(b.sql, b.values ?? []),
};
} catch (error) {
const canRetry = this.readOnly
? isRetriableError(error) || isRolledBackError(error)
: isNeverSentError(error) || isRolledBackError(error);
if (!canRetry || attempt >= ITEM_RETRY_ATTEMPTS) {
return { ok: false, error };
}
attempt++;
fallbackItemRetriesCounter.add(1, this.#metricAttrs);
await sleep(RETRY_BASE_BACKOFF_MS * attempt);
} catch (e) {
error = e;
} finally {
connection.release();
connection = undefined;
}
const canRetry = this.readOnly
? isRetriableError(error) || isRolledBackError(error)
: isNeverSentError(error) || isRolledBackError(error);
if (!canRetry || attempt >= ITEM_RETRY_ATTEMPTS) {
return { ok: false, error };
}
attempt++;
fallbackItemRetriesCounter.add(1, this.#metricAttrs);
await sleep(RETRY_BASE_BACKOFF_MS * attempt);
}
}
}
@@ -126,7 +126,7 @@ describe('SQLBatcher', () => {
});
it('drops the oldest item with reason queueOverflow at the high-water mark', async () => {
const conn = makeConnection(happyBatch);
const conn = makeConnection(() => [[{ n: 0 }], undefined]);
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxTimeInQueue: 5,
@@ -277,6 +277,65 @@ describe('SQLBatcher', () => {
);
});
it('sends a lone write as-is, outside a transaction', async () => {
const conn = makeConnection((sql) => {
if (isBatchQuery(sql)) throw new Error('unexpected batch query');
return [{ affectedRows: 1 }, undefined];
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, { maxBatchSize: 1 });
const result = await batcher.query('UPDATE t SET x = ?', [1]);
expect(result[0]).toEqual({ affectedRows: 1 });
expect(conn.query).toHaveBeenCalledTimes(1);
expect(conn.query).toHaveBeenCalledWith('UPDATE t SET x = ?', [1]);
expect(conn.beginTransaction).not.toHaveBeenCalled();
expect(conn.commit).not.toHaveBeenCalled();
expect(conn.release).toHaveBeenCalledTimes(1);
});
it('runs a lone write the server rejects once, without tripping the breaker', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
const conn = makeConnection((sql) => {
if (sql === 'INSERT dup') throw makeError('ER_DUP_ENTRY');
return [{ affectedRows: 1 }, undefined];
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxBatchSize: 1,
failureThreshold: 1,
cooldownMs: 60_000,
});
await expect(batcher.query('INSERT dup', [])).rejects.toMatchObject({
code: 'ER_DUP_ENTRY',
});
expect(conn.query).toHaveBeenCalledTimes(1);
expect(conn.rollback).not.toHaveBeenCalled();
expect(warnSpy).not.toHaveBeenCalled();
// The server answered, so it is up: the next write goes through.
await expect(batcher.query('INSERT fine', [])).resolves.toBeTruthy();
warnSpy.mockRestore();
});
it('retries a lone write on a connection the server closed for inactivity', async () => {
let attempts = 0;
const conn = makeConnection(() => {
attempts++;
if (attempts === 1) {
throw makeError('ER_CLIENT_INTERACTION_TIMEOUT');
}
return [{ affectedRows: 1 }, undefined];
});
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, { maxBatchSize: 1 });
const result = await batcher.query('INSERT x', []);
expect(result[0]).toEqual({ affectedRows: 1 });
expect(attempts).toBe(2);
expect(getConnection).toHaveBeenCalledTimes(2);
});
it('never retries deterministic row-level errors and does not escalate the breaker', async () => {
let fallbackAttempts = 0;
const conn = makeConnection((sql) => {
@@ -27,7 +27,7 @@ import { DatabaseClientFactory } from './index.js';
import { SqliteDatabaseClient } from './SqliteDatabaseClient.js';
/** Highest schema version the migration table can reach. */
const CURRENT_SCHEMA_VERSION = 82;
const CURRENT_SCHEMA_VERSION = 86;
/**
* These suites migrate real files on disk. Idle they finish in well under a
@@ -116,6 +116,10 @@ const AVAILABLE_MIGRATIONS: [number, string[]][] = [
[79, ['0084_share-anyone-with-link.sql']],
[80, ['0085_event-subscriptions-include-value.sql']],
[81, ['0086_subdomains-app-owner-cascade.sql']],
[82, ['0087_feedback-attachments.sql']],
[83, ['0088_apps-index-url.sql']],
[84, ['0089_team-require-2fa.sql']],
[85, ['0090_user-app-permission-index.sql']],
];
export class SqliteDatabaseClient extends AbstractDatabaseClient {
@@ -0,0 +1,39 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0087: Contact Us attachment metadata, a JSON array
-- of `{name, type, size}` (NULL when none). Guarded by a throwaway procedure,
-- as in mysql_mig_21.
DROP PROCEDURE IF EXISTS _puter_add_feedback_attachments;
DELIMITER //
CREATE PROCEDURE _puter_add_feedback_attachments()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'feedback'
AND COLUMN_NAME = 'attachments'
) THEN
ALTER TABLE `feedback` ADD COLUMN `attachments` text DEFAULT NULL;
END IF;
END//
DELIMITER ;
CALL _puter_add_feedback_attachments();
DROP PROCEDURE IF EXISTS _puter_add_feedback_attachments;
@@ -0,0 +1,44 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0088: index `apps.index_url` for origin lookups.
-- TEXT needs a prefix; 191 characters keeps the key under 767 bytes in
-- utf8mb4 whatever the table's row format, and URLs diverge well before it.
--
-- Idempotent: the guarded procedure, as mig_36. The 1061 handler covers two
-- nodes booting at once, where the loser finds the index already built.
DROP PROCEDURE IF EXISTS _puter_add_apps_index_url_index;
DELIMITER //
CREATE PROCEDURE _puter_add_apps_index_url_index()
BEGIN
DECLARE CONTINUE HANDLER FOR 1061 BEGIN END;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'apps'
AND INDEX_NAME = 'idx_apps_index_url'
) THEN
ALTER TABLE `apps`
ADD INDEX `idx_apps_index_url` (`index_url`(191)),
ALGORITHM=INPLACE, LOCK=NONE;
END IF;
END//
DELIMITER ;
CALL _puter_add_apps_index_url_index();
DROP PROCEDURE IF EXISTS _puter_add_apps_index_url_index;
@@ -0,0 +1,21 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0089: a team may require 2FA of the accounts it
-- provisioned. Added through `_puter_add_col` as mysql_mig_32 does, so a
-- replay finds the column and does nothing.
CALL _puter_add_col('group', 'require_2fa', '`require_2fa` tinyint(1) NOT NULL DEFAULT 0');
@@ -0,0 +1,39 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0090. Guarded like mysql_mig_22, because there is
-- no per-file applied-state tracking and a replay must do nothing.
DROP PROCEDURE IF EXISTS _puter_add_user_app_permission_index;
DELIMITER //
CREATE PROCEDURE _puter_add_user_app_permission_index()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user_to_app_permissions'
AND INDEX_NAME = 'idx_user_to_app_permissions_permission'
) THEN
ALTER TABLE `user_to_app_permissions`
ADD INDEX `idx_user_to_app_permissions_permission` (`permission`);
END IF;
END//
DELIMITER ;
CALL _puter_add_user_app_permission_index();
DROP PROCEDURE IF EXISTS _puter_add_user_app_permission_index;
@@ -0,0 +1,21 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0087: Contact Us attachment metadata, a JSON array
-- of `{name, type, size}` (NULL when none).
ALTER TABLE feedback ADD COLUMN IF NOT EXISTS attachments text DEFAULT NULL;
@@ -0,0 +1,24 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0088: index `apps.index_url` for origin lookups.
-- Hash rather than btree: `index_url` allows 3000 characters, past btree's
-- ~2.7 KB row limit, and every lookup is equality (`=` / `IN`).
--
-- Idempotent via IF NOT EXISTS.
CREATE INDEX IF NOT EXISTS idx_apps_index_url ON apps USING hash (index_url);
@@ -0,0 +1,21 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0089: a team may require 2FA of the accounts it
-- provisioned. Idempotent -- there is no per-file applied-state tracking, so
-- a replay must find the column already present and do nothing.
ALTER TABLE "group" ADD COLUMN IF NOT EXISTS require_2fa smallint NOT NULL DEFAULT 0;
@@ -0,0 +1,24 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0090, as postgres_mig_11 did for user_to_user.
-- text_pattern_ops is what makes the left-anchored LIKE a range scan under a
-- non-C collation -- the plain index dev_to_app already has does not.
CREATE INDEX IF NOT EXISTS idx_user_to_app_permissions_permission
ON user_to_app_permissions (permission text_pattern_ops);
CREATE INDEX IF NOT EXISTS idx_dev_to_app_permissions_permission_pattern
ON dev_to_app_permissions (permission text_pattern_ops);
@@ -0,0 +1,21 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Contact Us attachment metadata: a JSON array of `{name, type, size}`, NULL
-- when none. The files themselves go out with the support email.
ALTER TABLE `feedback` ADD COLUMN "attachments" TEXT DEFAULT NULL;
@@ -0,0 +1,22 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Resolving an origin to its app looks `apps` up by `index_url` on every
-- origin-based app token request (`AuthService#findCanonicalAppUidForOrigin`),
-- plus `AppStore.existsByIndexUrl`. With no index each lookup scans the table.
CREATE INDEX IF NOT EXISTS `idx_apps_index_url` ON `apps` (`index_url`);
@@ -0,0 +1,21 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Whether every account this team provisioned must hold 2FA before it can
-- use Puter. Off until the owner turns it on, and the owner must hold 2FA
-- themselves first, or enabling it would lock out the account that did so.
ALTER TABLE `group` ADD COLUMN `require_2fa` INTEGER NOT NULL DEFAULT 0;
@@ -0,0 +1,26 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Withdrawing an app's cross-app data grants looks them up by permission text,
-- and no existing index on either table leads with `permission`. Same fix as
-- `idx_user_to_user_permissions_permission` in 0067.
CREATE INDEX IF NOT EXISTS `idx_user_to_app_permissions_permission`
ON `user_to_app_permissions` (`permission`);
-- The sweep reads both tables; MySQL and Postgres already index this one.
CREATE INDEX IF NOT EXISTS `idx_dev_to_app_permissions_permission`
ON `dev_to_app_permissions` (`permission`);
@@ -32,6 +32,7 @@ const RETRIABLE_ERROR_CODES = new Set([
'ENETUNREACH',
'EAI_AGAIN',
POOL_ACQUIRE_TIMEOUT,
'ER_CLIENT_INTERACTION_TIMEOUT',
]);
const RETRIABLE_ERROR_MESSAGES = [
@@ -52,6 +53,9 @@ const NEVER_SENT_ERROR_CODES = new Set([
'ENETUNREACH',
'EAI_AGAIN',
POOL_ACQUIRE_TIMEOUT,
// The server closed the pooled connection for inactivity before the
// statement arrived, so nothing ran.
'ER_CLIENT_INTERACTION_TIMEOUT',
]);
/**
@@ -142,6 +142,42 @@ describe('DDBClient — item operations', () => {
expect(result.Item).toBeUndefined();
});
it('deletes only while its condition holds', async () => {
await client.put(TABLE, { pk: 'u6', sk: 'guarded', v: 1 });
await expect(
client.del(
TABLE,
{ pk: 'u6', sk: 'guarded' },
{
condition: {
expression: '#v = :wrong',
names: { '#v': 'v' },
values: { ':wrong': 2 },
},
},
),
).rejects.toMatchObject({ name: 'ConditionalCheckFailedException' });
expect(
(await client.get(TABLE, { pk: 'u6', sk: 'guarded' })).Item,
).toBeDefined();
await client.del(
TABLE,
{ pk: 'u6', sk: 'guarded' },
{
condition: {
expression: '#v = :right',
names: { '#v': 'v' },
values: { ':right': 1 },
},
},
);
expect(
(await client.get(TABLE, { pk: 'u6', sk: 'guarded' })).Item,
).toBeUndefined();
});
it('fetches keys from several tables in one batch', async () => {
await client.put(TABLE, { pk: 'b1', sk: 'a', v: 1 });
await client.put(TABLE, { pk: 'b1', sk: 'b', v: 2 });
@@ -364,6 +400,59 @@ describe('DDBClient — expired item sweep', () => {
expect((await client.get(table, { pk: 'fresh' })).Item).toBeDefined();
expect((await client.get(table, { pk: 'eternal' })).Item).toBeDefined();
});
it('keeps an item whose ttl is 0, which never expires', async () => {
const client = new DDBClient(localConfig());
const table = 'ttl-zero-items';
const schema = {
TableName: table,
KeySchema: [{ AttributeName: 'pk', KeyType: 'HASH' as const }],
AttributeDefinitions: [
{ AttributeName: 'pk', AttributeType: 'S' as const },
],
ProvisionedThroughput: {
ReadCapacityUnits: 5,
WriteCapacityUnits: 5,
},
};
await client.createTableIfNotExists(schema, 'expireAt');
await awaitTable(client, table, { pk: 'ready' });
await client.put(table, { pk: 'zero', expireAt: 0 });
// Second call finds the table in use, so the sweep runs.
await client.createTableIfNotExists(schema, 'expireAt');
expect((await client.get(table, { pk: 'zero' })).Item).toBeDefined();
});
it('keeps an item whose ttl is not a number, which the sweep can never purge', async () => {
const client = new DDBClient(localConfig());
const table = 'ttl-nan-items';
const schema = {
TableName: table,
KeySchema: [{ AttributeName: 'pk', KeyType: 'HASH' as const }],
AttributeDefinitions: [
{ AttributeName: 'pk', AttributeType: 'S' as const },
],
ProvisionedThroughput: {
ReadCapacityUnits: 5,
WriteCapacityUnits: 5,
},
};
await client.createTableIfNotExists(schema, 'expireAt');
await awaitTable(client, table, { pk: 'ready' });
const past = Math.floor(Date.now() / 1000) - 60;
await client.put(table, { pk: 'stringTtl', expireAt: String(past) });
// Second call finds the table in use, so the sweep runs.
await client.createTableIfNotExists(schema, 'expireAt');
expect(
(await client.get(table, { pk: 'stringTtl' })).Item,
).toBeDefined();
});
});
describe('DDBClient — expired item sweep on an empty table', () => {
+27 -5
View File
@@ -481,7 +481,15 @@ export class DDBClient extends PuterClient {
async del<T extends Record<string, unknown>>(
table: string,
key: T,
opts?: { returnOld?: boolean },
opts?: {
returnOld?: boolean;
/** Refused (`ConditionalCheckFailedException`) unless it holds. */
condition?: {
expression: string;
names?: Record<string, string>;
values?: Record<string, unknown>;
};
},
) {
const command = new DeleteCommand({
TableName: table,
@@ -490,6 +498,20 @@ export class DDBClient extends PuterClient {
// ALL_OLD makes the delete an atomic claim: exactly one caller
// gets the attributes back.
...(opts?.returnOld ? { ReturnValues: 'ALL_OLD' as const } : {}),
...(opts?.condition
? {
ConditionExpression: opts.condition.expression,
...(opts.condition.names
? { ExpressionAttributeNames: opts.condition.names }
: {}),
...(opts.condition.values
? {
ExpressionAttributeValues:
opts.condition.values,
}
: {}),
}
: {}),
});
const client = await this.#getDocumentClient();
@@ -761,14 +783,15 @@ export class DDBClient extends PuterClient {
const scan = await client.send(
new ScanCommand({
TableName: table,
FilterExpression: '#ttl < :now',
// 0 means no expiry, so it must never sweep.
FilterExpression: '#ttl < :now AND #ttl <> :none',
ExpressionAttributeNames: {
'#ttl': ttlAttribute,
...Object.fromEntries(
keyNames.map((key) => [`#k_${key}`, key]),
),
},
ExpressionAttributeValues: { ':now': now },
ExpressionAttributeValues: { ':now': now, ':none': 0 },
ProjectionExpression: keyNames
.map((key) => `#k_${key}`)
.join(', '),
@@ -779,8 +802,7 @@ export class DDBClient extends PuterClient {
);
lastEvaluatedKey = scan.LastEvaluatedKey as
| Record<string, unknown>
| undefined;
Record<string, unknown> | undefined;
const items = scan.Items;
if (!items || items.length === 0) continue;
+16
View File
@@ -404,6 +404,22 @@ will be asked to choose your own the next time you sign in.</p>
<p>If you did not expect this, the record of who reset it and when is in your
team activity, along with every sign-in to your account.</p>
<p>Sincerely,</p>
<p>Puter</p>
`,
},
team_2fa_reset: {
subject:
'Two-factor authentication was reset on your {{team_name}} account',
html: `
<p>Hi there,</p>
<p>An administrator of {{team_name}} reset two-factor authentication on your
Puter account <b>{{username}}</b>, so you can set it up again from a device you
still have. You have been signed out everywhere.</p>
<p>Your password is unchanged. If your team requires two-factor authentication,
you will be asked to enrol the next time you sign in.</p>
<p>If you did not expect this, the record of who reset it and when is in your
team activity, along with every sign-in to your account.</p>
<p>Sincerely,</p>
<p>Puter</p>
`,
},
+28 -5
View File
@@ -267,6 +267,17 @@ export type EventMap = {
message: string | null;
[key: string]: unknown;
};
// Inbound mail to a feedback address (`abuse@`, `fbl@`, ...), emitted via
// `emitAndWait`. A listener that takes it sets `handled` and it reaches no
// mailbox; otherwise it is routed like any other message.
'email.ingress.feedback': {
/** The envelope recipient, as the relay presented it. */
to: string;
from: string | null;
/** The whole message, exactly as received. */
raw: Buffer;
handled: boolean;
};
'user.save_account': {
user_id: number;
old_username?: string;
@@ -463,6 +474,8 @@ export type EventMap = {
values?: unknown[];
/** Keys among `keys` private to the namespace's app. */
noShareKeys?: string[];
/** Events handler runs behind the write; see `Actor.handlerDepth`. */
handlerDepth?: number;
};
/**
* A whole namespace was emptied. Namespace-level on purpose: `flush`'s own
@@ -486,6 +499,8 @@ export type EventMap = {
appUid: string | null;
value: Record<string, unknown>;
createdAt: number;
/** Events handler runs behind whatever created it. */
handlerDepth?: number;
};
/**
* One notification, addressed at whatever sockets each region holds for the
@@ -589,6 +604,11 @@ export type EventMap = {
generation: number;
/** Whether the table changed; only then does a peer need to re-read it. */
durable: boolean;
/**
* Session subscriptions this bump ended, so siblings stop queueing for
* them.
*/
ended?: string[];
};
/**
* A user's sockets moved between regions, so every node must drop what it
@@ -660,7 +680,8 @@ export type EventMap = {
uid?: string;
thumbnail?: string | null;
};
'thumbnail.created': { url: string };
/** `uuid` is the entry the thumbnail belongs to. */
'thumbnail.created': { url: string; uuid: string };
'thumbnail.upload.prepare': {
items: { index: number; item_uid: string }[];
uploadUrl?: string;
@@ -708,6 +729,7 @@ export type EventMap = {
email?: string;
app_name?: string;
};
/** False for an app actor: add only `subscribed` / `paid_storage` then. */
isUser: boolean;
};
'wisp.get-policy': {
@@ -852,10 +874,11 @@ export type EventKey = keyof EventMap & string;
// Generates a wildcard for every non-final dot-separated prefix of K.
export type WildcardPrefixes<K extends string> =
K extends `${infer Head}.${infer Tail}`
? | `${Head}.*`
| (Tail extends `${string}.${string}`
? `${Head}.${WildcardPrefixes<Tail>}`
: never)
?
| `${Head}.*`
| (Tail extends `${string}.${string}`
? `${Head}.${WildcardPrefixes<Tail>}`
: never)
: never;
export type ListenKey = EventKey | WildcardPrefixes<EventKey>;
+159
View File
@@ -0,0 +1,159 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { existsSync, readFileSync } from 'node:fs';
import path from 'node:path';
import type { IConfig } from './types';
// Config resolution order:
// 1. `process.env.PUTER_CONFIG_PATH` — absolute path to a config file. Used
// by deployments whose outer bootstrap writes a merged config to a known
// location.
// 2. `<PACKAGE_ROOT>/config.json` — user's runtime override (gitignored),
// deep-merged over config.default.json so users can omit keys they
// don't care to override (e.g. gui_assets_root, database).
// 3. `<PACKAGE_ROOT>/config.default.json` — bundled OSS defaults.
//
// Post-flatten depth: compiled file is at `packages/puter/dist/src/backend/index.js`,
// so three `..`s land at `packages/puter/`.
const PACKAGE_ROOT = path.resolve(__dirname, '../../..');
// Root of the running code tree. Matches PACKAGE_ROOT for a source run, but
// points at `dist/` for a compiled run — so config-declared paths like
// `./extensions` resolve to `dist/extensions` at runtime without the config
// having to know about the build layout.
const RUNTIME_ROOT = path.resolve(__dirname, '../..');
const isPlainObject = (v: unknown): v is Record<string, unknown> =>
typeof v === 'object' && v !== null && !Array.isArray(v);
const deepMerge = <T extends Record<string, unknown>>(
base: T,
override: Record<string, unknown>,
): T => {
const out: Record<string, unknown> = { ...base };
for (const [k, v] of Object.entries(override)) {
out[k] =
isPlainObject(v) && isPlainObject(out[k])
? deepMerge(out[k] as Record<string, unknown>, v)
: v;
}
return out as T;
};
export const loadConfig = (): IConfig => {
const envPath = process.env.PUTER_CONFIG_PATH;
const runtimePath = path.join(PACKAGE_ROOT, 'config.json');
const defaultPath = path.join(PACKAGE_ROOT, 'config.default.json');
const defaults = existsSync(defaultPath)
? (JSON.parse(readFileSync(defaultPath, 'utf8')) as Record<
string,
unknown
>)
: {};
// Runtime override path: env wins, then config.json, else no override
// (we still return defaults so single-file installs work).
const overridePath =
envPath && existsSync(envPath)
? envPath
: existsSync(runtimePath)
? runtimePath
: null;
console.log(`[config] defaults from ${defaultPath}`);
if (overridePath) console.log(`[config] override from ${overridePath}`);
const override = overridePath
? (JSON.parse(readFileSync(overridePath, 'utf8')) as Record<
string,
unknown
>)
: {};
const config = deepMerge(defaults, override) as IConfig;
if (!config.version) {
const pkgPath = path.join(PACKAGE_ROOT, 'package.json');
if (existsSync(pkgPath)) {
try {
const pkg = JSON.parse(readFileSync(pkgPath, 'utf8')) as {
version?: string;
};
if (pkg.version) config.version = pkg.version;
} catch {
// fall through — /version returns 'unknown'
}
}
}
// Computed defaults. `origin` and `pub_port` are the externally-visible
// URL+port — what the browser sees. Separate from `port`, which is the
// bind port (can differ when behind a reverse proxy). Code paths that
// build self-referential URLs (GUI bootstrap, email links, OIDC callbacks)
// depend on `origin` having the right port baked in.
if (config.pub_port === undefined) config.pub_port = config.port;
const protocol = config.protocol ?? 'http';
const domain = config.domain ?? 'localhost';
const portSuffix =
config.pub_port === 80 || config.pub_port === 443
? ''
: `:${config.pub_port}`;
if (config.origin === undefined) {
config.origin = `${protocol}://${domain}${portSuffix}`;
}
// API lives on the `api.` subdomain on the same host+port as the main
// origin (see PuterRouter subdomain handling in server.ts). Compute it
// from pub_port/domain so a single-port override (e.g. port: 5101) flows
// through to the GUI bootstrap without users having to restate the URL.
if (config.api_base_url === undefined) {
config.api_base_url = `${protocol}://api.${domain}${portSuffix}`;
}
// Resolve path-valued config fields. Two different roots:
// - `extensions` uses RUNTIME_ROOT because extensions ship inside the
// build output (dist/extensions) and the loader's dynamic import()
// resolves relative paths against the *importing* module file.
// - GUI/puter-js/builtin-apps use PACKAGE_ROOT because those assets
// live only in the source tree (not copied into dist/) and are served
// via express.static at runtime.
const resolveRuntime = (p: string): string =>
path.isAbsolute(p) ? p : path.resolve(RUNTIME_ROOT, p);
const resolvePackage = (p: string): string =>
path.isAbsolute(p) ? p : path.resolve(PACKAGE_ROOT, p);
if (Array.isArray(config.extensions)) {
config.extensions = config.extensions.map(resolveRuntime);
}
if (typeof config.gui_assets_root === 'string') {
config.gui_assets_root = resolvePackage(config.gui_assets_root);
}
if (typeof config.puterjs_root === 'string') {
config.puterjs_root = resolvePackage(config.puterjs_root);
}
if (isPlainObject(config.builtin_apps)) {
for (const [k, v] of Object.entries(config.builtin_apps)) {
if (typeof v === 'string') {
(config.builtin_apps as Record<string, string>)[k] =
resolvePackage(v);
}
}
}
return config;
};
+18 -6
View File
@@ -69,6 +69,11 @@ const APP_ICON_LIMIT = {
key: 'ip',
};
// Shared by `GET /apps/:name` (pipe-separated) and `POST /query/app`, which
// both fan out one lookup per name.
const APP_LOOKUP_MAX_ENTRIES = 200;
const APP_LOOKUP_MAX_SELECTOR_LEN = 200;
export class AppController extends PuterController {
get appStore() {
return this.stores.app;
@@ -266,11 +271,21 @@ export class AppController extends PuterController {
async (req, res) => {
const raw = req.params.name;
const names = raw.split('|').filter(Boolean);
if (names.length > APP_LOOKUP_MAX_ENTRIES) {
throw new HttpError(
400,
`at most ${APP_LOOKUP_MAX_ENTRIES} app names per request`,
{ legacyCode: 'bad_request' },
);
}
const userUid = req.actor?.user?.uuid ?? null;
const results = await Promise.all(
names.map(async (name) => {
if (name.length > APP_LOOKUP_MAX_SELECTOR_LEN) {
return null;
}
const app = await this.appStore.getByName(name);
if (!app) return null;
let shaped;
@@ -327,9 +342,6 @@ export class AppController extends PuterController {
// included for public/owned apps only, consistent with
// marketplace semantics.
const QUERY_APP_MAX_ENTRIES = 200;
const QUERY_APP_MAX_SELECTOR_LEN = 200;
router.post(
'/query/app',
{
@@ -339,10 +351,10 @@ export class AppController extends PuterController {
},
async (req, res) => {
const appList = Array.isArray(req.body) ? req.body : [];
if (appList.length > QUERY_APP_MAX_ENTRIES) {
if (appList.length > APP_LOOKUP_MAX_ENTRIES) {
throw new HttpError(
400,
`request body must contain at most ${QUERY_APP_MAX_ENTRIES} selectors`,
`request body must contain at most ${APP_LOOKUP_MAX_ENTRIES} selectors`,
{ legacyCode: 'bad_request' },
);
}
@@ -354,7 +366,7 @@ export class AppController extends PuterController {
if (
typeof selector !== 'string' ||
selector.length === 0 ||
selector.length > QUERY_APP_MAX_SELECTOR_LEN
selector.length > APP_LOOKUP_MAX_SELECTOR_LEN
) {
continue;
}
@@ -538,6 +538,57 @@ describe('AppController GET /apps/:name', () => {
});
});
it('throws 400 when the pipe-batch exceeds the 200-name cap', async () => {
const { actor } = await makeUser();
const { res } = makeRes();
const names = new Array(201).fill('x').join('|');
await expect(
withActor(actor, () =>
callRoute(
'get',
'/apps/:name',
makeReq({ params: { name: names }, actor }),
res,
),
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('accepts a pipe-batch at exactly the 200-name cap', async () => {
const owner = await makeUser();
const app = await createApp(owner.actor);
const names = [app.name, ...new Array(199).fill('missing-app')].join(
'|',
);
const { res, captured } = makeRes();
await withActor(owner.actor, () =>
callRoute(
'get',
'/apps/:name',
makeReq({ params: { name: names }, actor: owner.actor }),
res,
),
);
expect(Array.isArray(captured.body)).toBe(true);
expect(captured.body as unknown[]).toHaveLength(200);
});
it('throws 404 for a single name over the per-name length cap', async () => {
const { actor } = await makeUser();
const { res } = makeRes();
await expect(
withActor(actor, () =>
callRoute(
'get',
'/apps/:name',
makeReq({ params: { name: 'x'.repeat(201) }, actor }),
res,
),
),
).rejects.toMatchObject({ statusCode: 404 });
});
it('omits the stale index_url for a non-owner', async () => {
const owner = await makeUser();
const stranger = await makeUser();
@@ -17,11 +17,18 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { TOTP } from 'otpauth';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { makeActor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { createSecret as otpCreateSecret } from '../../services/auth/OTPUtil.js';
import { FULL_API_ACCESS } from '../../services/permission/consts.js';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
import {
createTestUser,
setupPuterTestEnv,
type PuterTestEnv,
} from '../../testUtil.js';
import type { IConfig } from '../../types';
/**
* HTTP-level coverage for `/auth/revoke-own-access-token` — the route that
@@ -343,4 +350,265 @@ describe('revoke-own-access-token over HTTP', () => {
}
expect(await tokenReadStatus(file.uid, readToken)).not.toBe(200);
});
// These three run last in the file's shared `env.users.user`: each one
// revokes a session belonging to `owner.token` / `owner.workerToken`, so
// later tests can't rely on those credentials still being live.
it('a revoked access token yields 401 with no reauth_token or auth_id', async () => {
const owner = env.users.user;
const file = await makeFile(owner);
const readToken = await mintReadToken(owner.token, file.uid);
const revoke = await call(
'POST',
'/auth/revoke-own-access-token',
owner.token,
{ token: readToken },
);
expect(revoke.status).toBe(200);
const res = await call('GET', '/whoami', readToken);
expect(res.status).toBe(401);
const body = (await res.json()) as Record<string, unknown>;
expect(body.code).toBe('reauth_required');
expect(body.reauth_token).toBeUndefined();
expect(body.auth_id).toBeUndefined();
});
it('a revoked worker session gets no reauth_token even though it rides the session token type', async () => {
const owner = env.users.user;
const decoded = env.server.services.token.verify(
'auth',
owner.workerToken,
) as { session_uid: string };
await env.server.stores.session.removeByUuid(decoded.session_uid);
const res = await call('GET', '/whoami', owner.workerToken);
expect(res.status).toBe(401);
const body = (await res.json()) as Record<string, unknown>;
expect(body.code).toBe('reauth_required');
expect(body.reauth_token).toBeUndefined();
expect(body.auth_id).toBeUndefined();
});
it('a revoked GUI session still gets a reauth_token', async () => {
const owner = env.users.user;
const decoded = env.server.services.token.verify('auth', owner.token) as {
session_uid: string;
};
await env.server.stores.session.removeByUuid(decoded.session_uid);
const res = await call('GET', '/whoami', owner.token);
expect(res.status).toBe(401);
const body = (await res.json()) as Record<string, unknown>;
expect(body.code).toBe('reauth_required');
expect(typeof body.reauth_token).toBe('string');
const user = await env.server.stores.user.getByUsername(
owner.username,
);
expect(
env.server.services.auth.verifyReauthToken(
body.reauth_token as string,
).authId,
).toBe(user!.uuid);
});
});
/**
* Enrolling a second factor: setup sits behind the user-protected gate, and
* enable needs a live code for the secret setup issued.
*/
describe('2FA enrollment over HTTP', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv({ teams_enabled: true } as IConfig);
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const PASSWORD = 'tfa-http-password';
const uniq = () => Math.random().toString(36).slice(2, 10);
/** A signed-in password account with a confirmed address. */
const makeAccount = async () => {
const username = `tfa_${uniq()}`;
const { token } = await createTestUser(env.server, {
username,
password: PASSWORD,
});
const row = (await env.server.stores.user.getByUsername(username))!;
await env.server.stores.user.update(row.id, {
email: `${username}@test.local`,
email_confirmed: 1,
});
await env.server.stores.user.invalidateById(row.id);
return { username, token, userId: row.id };
};
/** The user-protected gate takes the session cookie on the GUI origin. */
const protectedPost = (
path: string,
token: string,
body: Record<string, unknown>,
credential: 'cookie' | 'bearer' = 'cookie',
) =>
fetch(new URL(path, env.origin), {
method: 'POST',
headers: {
'content-type': 'application/json',
...(credential === 'cookie'
? { cookie: `puter_auth_token=${token}` }
: { authorization: `Bearer ${token}` }),
},
body: JSON.stringify(body),
});
const setup = (
token: string,
body: Record<string, unknown>,
credential: 'cookie' | 'bearer' = 'cookie',
) => protectedPost('/user-protected/setup-2fa', token, body, credential);
const configure = (
token: string,
action: string,
body: Record<string, unknown> = {},
) =>
fetch(new URL(`/auth/configure-2fa/${action}`, env.apiOrigin), {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${token}`,
},
body: JSON.stringify(body),
});
const otpState = async (userId: number) => {
const row = await env.server.stores.user.getByProperty('id', userId, {
force: true,
});
return {
secret: row?.otp_secret ?? null,
enabled: Boolean(row?.otp_enabled),
};
};
const liveTotp = (username: string, secret: string) =>
new TOTP({
issuer: 'puter.com',
label: username,
algorithm: 'SHA1',
digits: 6,
secret,
}).generate();
it('refuses setup to a session that has not re-proved the password', async () => {
const account = await makeAccount();
const noPassword = await setup(account.token, {});
expect(noPassword.status).toBe(403);
expect(await noPassword.text()).toContain('password_required');
const wrongPassword = await setup(account.token, { password: 'nope' });
expect(wrongPassword.status).toBe(400);
expect(await wrongPassword.text()).toContain('password_mismatch');
// Right password, but a bearer token is not the session cookie.
const bearer = await setup(
account.token,
{ password: PASSWORD },
'bearer',
);
expect(bearer.status).toBe(401);
// The old ungated path no longer hands out a secret.
expect((await configure(account.token, 'setup')).status).toBe(400);
expect((await otpState(account.userId)).secret).toBeNull();
});
it('enrolls after the password, and enables only on a live code', async () => {
const account = await makeAccount();
const res = await setup(account.token, { password: PASSWORD });
expect(res.status).toBe(200);
const { secret, codes } = (await res.json()) as {
secret: string;
codes: string[];
};
expect(codes).toHaveLength(10);
expect((await otpState(account.userId)).secret).toBe(secret);
expect((await configure(account.token, 'enable')).status).toBe(400);
const foreign = otpCreateSecret(account.username).secret;
const wrong = await configure(account.token, 'enable', {
code: liveTotp(account.username, foreign),
});
expect(wrong.status).toBe(400);
expect(await wrong.text()).toContain('code_mismatch');
expect((await otpState(account.userId)).enabled).toBe(false);
const enabled = await configure(account.token, 'enable', {
code: liveTotp(account.username, secret),
});
expect(enabled.status).toBe(200);
expect((await otpState(account.userId)).enabled).toBe(true);
});
it('lets a seat its team holds for 2FA enroll through the same gate', async () => {
const owner = await makeAccount();
// The rule can only be turned on by an owner who has 2FA.
await env.server.stores.user.update(owner.userId, { otp_enabled: 1 });
await env.server.stores.user.invalidateById(owner.userId);
const team = await env.server.services.team.createTeam(owner.userId, {
name: 'Acme',
});
const seatName = `seat_${uniq()}`;
const { userId, temporaryPassword } =
await env.server.services.team.provisionAccount(
team.uid,
owner.userId,
{ username: seatName },
);
await env.server.services.team.updateTeam(team.uid, owner.userId, {
require2fa: true,
});
const { token } = await env.server.services.auth.createSessionToken(
(await env.server.stores.user.getById(userId))!,
);
// First sign-in replaces the temporary password.
const changed = await protectedPost(
'/user-protected/change-password',
token,
{ password: temporaryPassword, new_pass: 'seat-own-password' },
);
expect(changed.status).toBe(200);
const held = await fetch(new URL('/teams', env.apiOrigin), {
headers: { authorization: `Bearer ${token}` },
});
expect(held.status).toBe(403);
expect(await held.text()).toContain('two_factor_required');
const res = await setup(token, { password: 'seat-own-password' });
expect(res.status).toBe(200);
const { secret } = (await res.json()) as { secret: string };
const enabled = await configure(token, 'enable', {
code: liveTotp(seatName, secret),
});
expect(enabled.status).toBe(200);
expect((await otpState(userId)).enabled).toBe(true);
const released = await fetch(new URL('/teams', env.apiOrigin), {
headers: { authorization: `Bearer ${token}` },
});
expect(released.status).toBe(200);
});
});
@@ -30,6 +30,7 @@
import bcrypt from 'bcrypt';
import jwt from 'jsonwebtoken';
import { TOTP } from 'otpauth';
import { v4 as uuidv4, v5 as uuidv5 } from 'uuid';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import type { EventClient } from '../../clients/event/EventClient.js';
@@ -43,6 +44,7 @@ import {
type TokenSource,
} from '../../core/http/types.js';
import { PuterServer } from '../../server.js';
import { createSecret as otpCreateSecret } from '../../services/auth/OTPUtil.js';
import { kvSharePermission } from '../../services/events/kvShares.js';
import { FULL_API_ACCESS } from '../../services/permission/consts.js';
import { setupTestServer } from '../../testUtil.js';
@@ -716,9 +718,8 @@ describe('AuthController.handleSignup', () => {
}),
makeRes(),
);
const occupant = await server.stores.user.getByUsername(
occupantUsername,
);
const occupant =
await server.stores.user.getByUsername(occupantUsername);
const root = (await server.stores.fsEntry.getRootEntryForUser(
occupant!.id,
))!;
@@ -791,6 +792,22 @@ describe('AuthController.handleSignup', () => {
).rejects.toMatchObject({ statusCode: 400 });
});
it.each(['abuse', 'postmaster', 'fbl'])(
'rejects the role mailbox name "%s"',
async (username) => {
await expect(
controller.handleSignup(
makeReq({
username,
email: `a_${uniq()}@test.local`,
password: 'correct-horse-battery',
}),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
},
);
it('rejects an invalid email format', async () => {
await expect(
controller.handleSignup(
@@ -1264,6 +1281,47 @@ describe('AuthController.handleLogin', () => {
expect(isCompleteLoginResponse(res.body)).toBe(true);
});
// A deleted row still answering from cache reaches the session INSERT.
it('answers 401 rather than 500 when the account row is gone but still cached', async () => {
const goneName = `lg_${Math.random().toString(36).slice(2, 10)}`;
await controller.handleSignup(
makeReq({
username: goneName,
email: `${goneName}@test.local`,
password,
}),
makeRes(),
);
const gone = await server.stores.user.getByUsername(goneName);
const realWrite = server.clients.db.write.bind(server.clients.db);
const writeSpy = vi
.spyOn(server.clients.db, 'write')
.mockImplementation(async (sql: string, params?: unknown[]) => {
if (/INSERT INTO `sessions`/.test(sql)) {
throw Object.assign(new Error('FOREIGN KEY constraint'), {
code: 'SQLITE_CONSTRAINT_FOREIGNKEY',
});
}
return realWrite(sql, params);
});
try {
await expect(
controller.handleLogin(
makeReq({ username: goneName, password }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 401 });
} finally {
writeSpy.mockRestore();
}
expect(
await server.clients.redis.get(`users:username:${goneName}`),
).toBeNull();
});
it('refuses an email the account has moved off', async () => {
const moverName = `lm_${Math.random().toString(36).slice(2, 10)}`;
const oldEmail = `${moverName}[email protected]`;
@@ -2161,6 +2219,49 @@ describe('AuthController grant flows', () => {
expect(res.body).toEqual({});
});
it('grant-user-app: 400 on an `extra`/`meta` larger than the cap', async () => {
const appName = `tb-${uuidv4()}`;
const app = await server.stores.app.create(
{
name: appName,
title: 'TestBoundApp',
index_url: `https://${appName}.example.test/index.html`,
},
{ ownerUserId: issuer.id },
);
const permission = 'service:tb-app:ii:read';
const oversized = { blob: 'z'.repeat(5000) };
for (const body of [
{ app_uid: app.uid, permission, extra: oversized },
{ app_uid: app.uid, permission, meta: oversized },
]) {
await expect(
inCtx(issuerActor, () =>
controller.handleGrantUserApp(
makeReq(body, { actor: issuerActor }),
makeRes(),
),
),
).rejects.toMatchObject({ statusCode: 400 });
}
const res = makeRes();
await inCtx(issuerActor, () =>
controller.handleGrantUserApp(
makeReq(
{
app_uid: app.uid,
permission,
extra: { blob: 'z'.repeat(1000) },
},
{ actor: issuerActor },
),
res,
),
);
expect(res.body).toEqual({});
});
it('grant-user-app: refuses a key-value delegation over a whole namespace', async () => {
// The prompt for it would read "let this app show your app data to
// whoever it picks", which describes no bounded capability — so the
@@ -3246,6 +3347,139 @@ describe('AuthController.handleGetUserAppToken + handleCheckApp', () => {
// Proves the row came from the bootstrap path, not an earlier test.
expect(bootstrapped?.description).toMatch(/^App created from origin /);
});
it('does not hand an origin the app that was repointed away from it', async () => {
// Whoever holds the origin next must not inherit the app, or the
// users who already authorized it.
const origin = `https://repointed-${uuidv4()}.example`;
const first = makeRes();
await inCtx(actor, () =>
controller.handleGetUserAppToken(
makeReq({ origin }, { actor }),
first,
),
);
const movedUid = (first.body as { app_uid: string }).app_uid;
const moved = await server.stores.app.getByUid(movedUid);
await server.stores.app.update(moved!.id, {
index_url: `https://new-home-${uuidv4()}.example`,
});
const check = makeRes();
await inCtx(actor, () =>
controller.handleCheckApp(makeReq({ origin }, { actor }), check),
);
const checked = check.body as {
app_uid: string;
authenticated: boolean;
token?: string;
};
expect(checked.app_uid).not.toBe(movedUid);
expect(checked.authenticated).toBe(false);
expect(checked.token).toBeUndefined();
const res = makeRes();
await inCtx(actor, () =>
controller.handleGetUserAppToken(
makeReq({ origin }, { actor }),
res,
),
);
const body = res.body as { token: string; app_uid: string };
expect(body.app_uid).toBe(checked.app_uid);
const decoded = server.services.token.verify('auth', body.token) as {
app_uid: string;
};
expect(decoded.app_uid).toBe(body.app_uid);
const fresh = await server.stores.app.getByUid(body.app_uid);
expect(fresh?.index_url).toBe(origin);
});
// sqlite doesn't enforce FKs, so inject the driver error at the db client.
// The grant row keys both the app and the user; which one vanished decides
// the answer, and the drivers don't all name the constraint.
describe('when a parent row is gone but still cached', () => {
const fkError = () =>
Object.assign(new Error('FOREIGN KEY constraint'), {
code: 'SQLITE_CONSTRAINT_FOREIGNKEY',
});
const makeApp = async () =>
(
server.stores.app.create as unknown as (
fields: Record<string, unknown>,
opts: { ownerUserId: number },
) => Promise<{ uid: string; id: number }>
)(
{
name: `fk-${uuidv4()}`,
title: 'Deleted under a warm cache',
index_url: 'https://example.test/fk.html',
},
{ ownerUserId: user.id },
);
/** Fails the grant insert without disturbing the rest of the handler. */
const failingGrant = async (fn: () => Promise<unknown>) => {
const realWrite = server.clients.db.write.bind(server.clients.db);
const spy = vi
.spyOn(server.clients.db, 'write')
.mockImplementation(async (sql: string, params?: unknown[]) => {
if (/user_to_app_permissions/.test(sql)) throw fkError();
return realWrite(sql, params);
});
try {
return await fn();
} finally {
spy.mockRestore();
}
};
it('answers 404 for a deleted app, and retires the stale cache entry', async () => {
const doomed = await makeApp();
// Straight to SQL; AppStore.delete would clear the stale entry.
await server.clients.db.write('DELETE FROM `apps` WHERE `id` = ?', [
doomed.id,
]);
await expect(
failingGrant(() =>
inCtx(actor, () =>
controller.handleGetUserAppToken(
makeReq({ app_uid: doomed.uid }, { actor }),
makeRes(),
),
),
),
).rejects.toMatchObject({ statusCode: 404 });
expect(
await server.clients.redis.get(`apps:uid:${doomed.uid}`),
).toBeNull();
});
it('answers 401 for a deleted account, leaving the live app cacheable', async () => {
const live = await makeApp();
await server.stores.app.getByUid(live.uid);
await expect(
failingGrant(() =>
inCtx(actor, () =>
controller.handleGetUserAppToken(
makeReq({ app_uid: live.uid }, { actor }),
makeRes(),
),
),
),
).rejects.toMatchObject({ statusCode: 401 });
// Tombstoning the app here would strand a perfectly live row.
expect(await server.stores.app.getByUid(live.uid)).toBeTruthy();
expect(
await server.clients.redis.get(`apps:uid:${live.uid}:deleted`),
).toBeNull();
});
});
});
// ── Access tokens: create + revoke ─────────────────────────────────
@@ -3342,6 +3576,75 @@ describe('AuthController.handleCreateAccessToken + handleRevokeAccessToken', ()
).rejects.toMatchObject({ statusCode: 400 });
});
it('rejects a permission wider than the column, leaving no session behind', async () => {
// Nothing rewrites these, so an over-wide one would reach the INSERT
// only after the session row was created and the JWT signed.
const before = (await server.clients.db.read(
"SELECT COUNT(*) AS n FROM sessions WHERE kind = 'access_token'",
[],
)) as Array<{ n: number }>;
await expect(
controller.handleCreateAccessToken(
makeReq(
{ permissions: [`service:${'a'.repeat(300)}:ii:read`] },
{ actor },
),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
const after = (await server.clients.db.read(
"SELECT COUNT(*) AS n FROM sessions WHERE kind = 'access_token'",
[],
)) as Array<{ n: number }>;
expect(after[0].n).toBe(before[0].n);
});
it('rejects more permissions than one request may carry', async () => {
// Each entry costs a permission check and an INSERT.
const permissions = Array.from(
{ length: 17 },
(_, i) => `service:cap-${i}:ii:read`,
);
await expect(
controller.handleCreateAccessToken(
makeReq({ permissions }, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('still mints at the cap', async () => {
const permissions = Array.from(
{ length: 16 },
(_, i) => `service:atcap-${i}:ii:read`,
);
const res = makeRes();
await controller.handleCreateAccessToken(
makeReq({ permissions }, { actor }),
res,
);
expect((res.body as { token: string }).token).toEqual(
expect.any(String),
);
});
it.each([
['an oversized permission string', ['x'.repeat(4097)]],
[
'an oversized `extra`',
[['service:x:ii:read', { a: 'y'.repeat(5000) }]],
],
['a non-object `extra`', [['service:x:ii:read', 'nope']]],
['an empty permission string', ['']],
])('rejects %s with 400', async (_label, permissions) => {
await expect(
controller.handleCreateAccessToken(
makeReq({ permissions: permissions as never[] }, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('mints a verifiable access-token JWT for valid permissions', async () => {
const res = makeRes();
await controller.handleCreateAccessToken(
@@ -5673,6 +5976,35 @@ describe('AuthController user-protected mutations (validation paths)', () => {
}
});
it('change-username: only completed renames spend the 2-per-30-days budget', async () => {
const { user: other } = await makeUserAndActor();
const { actor } = await makeUserAndActor();
for (let i = 0; i < 3; i++) {
await expect(
controller.handleChangeUsername(
makeReq({ new_username: other.username }, { actor }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
}
for (let i = 0; i < 2; i++) {
await controller.handleChangeUsername(
makeReq({ new_username: `r_${uniq()}` }, { actor }),
makeRes(),
);
}
await expect(
controller.handleChangeUsername(
makeReq({ new_username: `r_${uniq()}` }, { actor }),
makeRes(),
),
).rejects.toMatchObject({
statusCode: 429,
legacyCode: 'too_many_requests',
});
});
it('change-username: 400 when the home path is already occupied, before the rename', async () => {
const { user, actor } = await makeUserAndActor();
// A name no account holds, whose home path a stray row does. This is
@@ -5729,6 +6061,48 @@ describe('AuthController user-protected mutations (validation paths)', () => {
expect(after!.username).toBe(mover.username);
});
it('change-username: a name just vacated stays closed to other accounts, not to its last holder', async () => {
const { user: leaver, actor: leaverActor } = await makeUserAndActor();
const vacated = leaver.username;
// Warm the entry cache under the old path, as any read would.
const docs = (await server.stores.fsEntry.getEntryByPath(
`/${vacated}/Documents`,
))!;
await controller.handleChangeUsername(
makeReq({ new_username: `r_${uniq()}` }, { actor: leaverActor }),
makeRes(),
);
// The cache still maps the old path to the leaver's folder, so a new
// holder of the name would pass ACL on it.
const cached = await server.stores.fsEntry.getEntryByPath(
`/${vacated}/Documents`,
);
expect(cached?.uuid).toBe(docs.uuid);
const { user: claimer, actor: claimerActor } = await makeUserAndActor();
await expect(
controller.handleChangeUsername(
makeReq({ new_username: vacated }, { actor: claimerActor }),
makeRes(),
),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'username_already_in_use',
});
const after = await server.stores.user.getById(claimer.id, {
force: true,
});
expect(after!.username).toBe(claimer.username);
const res = makeRes();
await controller.handleChangeUsername(
makeReq({ new_username: vacated }, { actor: leaverActor }),
res,
);
expect(res.body).toEqual({ username: vacated });
});
it('change-email: 400 on missing/invalid email and on a confirmed-account collision', async () => {
const { actor } = await makeUserAndActor();
await expect(
@@ -6606,6 +6980,34 @@ describe('AuthController dev-app permission flows', () => {
).rejects.toMatchObject({ statusCode: 400 });
});
it.each([
['a non-object `extra`', { extra: 'nope' }],
['an array `extra`', { extra: [1, 2] }],
['an oversized `extra`', { extra: { blob: 'z'.repeat(5000) } }],
['an oversized `meta`', { meta: { blob: 'z'.repeat(5000) } }],
['an oversized `permission`', { permission: 'x'.repeat(4097) }],
// Survives the route cap but not the column it lands in.
[
'a `permission` wider than the column',
{ permission: `service:${'a'.repeat(300)}:ii:read` },
],
])('grant-dev-app: 400 on %s', async (_label, patch) => {
const { actor } = await makeUserAndActor();
await expect(
controller.handleGrantDevApp(
makeReq(
{
app_uid: `app-${uuidv4()}`,
permission: 'fs:read',
...patch,
},
{ actor },
),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('revoke-dev-app: 400 on missing app_uid/origin/permission', async () => {
const { actor } = await makeUserAndActor();
await expect(
@@ -6805,6 +7207,29 @@ describe('AuthController.handleAppUidFromOrigin', () => {
// ── 2FA configure / disable ────────────────────────────────────────
/** The code an authenticator app would show right now for `secret`. */
const liveTotp = (username: string, secret: string) =>
new TOTP({
issuer: 'puter.com',
label: username,
algorithm: 'SHA1',
digits: 6,
secret,
}).generate();
/** Calls the handler directly; the route's gate is tested over HTTP. */
const setup2fa = async (actor: Actor) => {
const res = makeRes();
await controller.handleSetup2fa(makeReq({}, { actor }), res);
return res.body as { url: string; secret: string; codes: string[] };
};
const enable2fa = (actor: Actor, body: Record<string, unknown> = {}) =>
controller.handleConfigure2fa(
makeReq(body, { actor, params: { action: 'enable' } }),
makeRes(),
);
describe('AuthController 2FA flows', () => {
it('configure-2fa: 400 on an unknown :action', async () => {
const { actor } = await makeUserAndActor();
@@ -6816,18 +7241,95 @@ describe('AuthController 2FA flows', () => {
).rejects.toMatchObject({ statusCode: 400 });
});
it('configure-2fa setup: returns {url, secret, codes[10]} and stores the secret', async () => {
it('configure-2fa: has no setup action, so a bare session mints no secret', async () => {
const { user, actor } = await makeUserAndActor();
const res = makeRes();
await controller.handleConfigure2fa(
makeReq({}, { actor, params: { action: 'setup' } }),
res,
);
const body = res.body as {
url: string;
secret: string;
codes: string[];
};
await expect(
controller.handleConfigure2fa(
makeReq({}, { actor, params: { action: 'setup' } }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400 });
const after = await server.stores.user.getById(user.id, {
force: true,
});
expect(after!.otp_secret ?? null).toBeNull();
});
it('lets a team seat enable 2FA without a confirmed email', async () => {
// A seat has no address by design (PUT-1792) while its team can
// require 2FA, so this check would bar it forever.
const owner = await makeUserAndActor();
const team = await server.services.team.createTeam(owner.user.id, {
name: 'Acme',
});
const seatName = `s_${uniq()}`;
await server.services.team.provisionAccount(team.uid, owner.user.id, {
username: seatName,
});
const seat = await server.stores.user.getByUsername(seatName);
const actor = {
user: {
id: seat!.id,
uuid: seat!.uuid,
username: seat!.username,
email: seat!.email ?? null,
email_confirmed: !!seat!.email_confirmed,
},
} as Actor;
expect(Boolean(seat!.email_confirmed)).toBe(false);
const { secret } = await setup2fa(actor);
await enable2fa(actor, { code: liveTotp(seat!.username, secret) });
const after = await server.stores.user.getById(seat!.id, {
force: true,
});
expect(Boolean(after!.otp_enabled)).toBe(true);
});
it('refuses to disable 2FA while the team requires it', async () => {
const owner = await makeUserAndActor({ otp_enabled: 1 });
const team = await server.services.team.createTeam(owner.user.id, {
name: 'Acme',
});
const seatName = `s_${uniq()}`;
await server.services.team.provisionAccount(team.uid, owner.user.id, {
username: seatName,
});
const seat = await server.stores.user.getByUsername(seatName);
const actor = {
user: {
id: seat!.id,
uuid: seat!.uuid,
username: seat!.username,
email: seat!.email ?? null,
},
} as Actor;
const { secret } = await setup2fa(actor);
await enable2fa(actor, { code: liveTotp(seat!.username, secret) });
await server.services.team.updateTeam(team.uid, owner.user.id, {
require2fa: true,
});
await expect(
controller.handleDisable2fa(makeReq({}, { actor }), makeRes()),
).rejects.toMatchObject({ statusCode: 409 });
const after = await server.stores.user.getById(seat!.id, {
force: true,
});
expect(Boolean(after!.otp_enabled)).toBe(true);
});
it('still refuses an ordinary account with an unconfirmed email', async () => {
const { user, actor } = await makeUserAndActor({ email_confirmed: 0 });
const { secret } = await setup2fa(actor);
await expect(
enable2fa(actor, { code: liveTotp(user.username, secret) }),
).rejects.toMatchObject({ statusCode: 403 });
});
it('setup-2fa: returns {url, secret, codes[10]} and stores the secret', async () => {
const { user, actor } = await makeUserAndActor();
const body = await setup2fa(actor);
expect(body.codes).toHaveLength(10);
expect(typeof body.secret).toBe('string');
const after = await server.stores.user.getById(user.id, {
@@ -6839,19 +7341,10 @@ describe('AuthController 2FA flows', () => {
).toHaveLength(10);
});
it('a recovery code minted by configure-2fa setup satisfies /login/recovery-code', async () => {
it('a recovery code minted by setup-2fa satisfies /login/recovery-code', async () => {
const { user, actor } = await makeUserAndActor({ email_confirmed: 1 });
const setupRes = makeRes();
await controller.handleConfigure2fa(
makeReq({}, { actor, params: { action: 'setup' } }),
setupRes,
);
const { codes } = setupRes.body as { codes: string[] };
await controller.handleConfigure2fa(
makeReq({}, { actor, params: { action: 'enable' } }),
makeRes(),
);
const { secret, codes } = await setup2fa(actor);
await enable2fa(actor, { code: liveTotp(user.username, secret) });
const otpJwt = server.services.token.sign(
'otp',
@@ -6866,14 +7359,11 @@ describe('AuthController 2FA flows', () => {
expect(isCompleteLoginResponse(res.body)).toBe(true);
});
it('configure-2fa setup: 409 when 2FA is already enabled', async () => {
it('setup-2fa: 409 when 2FA is already enabled', async () => {
const { actor } = await makeUserAndActor({ otp_enabled: 1 });
await expect(
controller.handleConfigure2fa(
makeReq({}, { actor, params: { action: 'setup' } }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 409 });
await expect(setup2fa(actor)).rejects.toMatchObject({
statusCode: 409,
});
});
it('configure-2fa test: 400 when code is missing', async () => {
@@ -6886,6 +7376,37 @@ describe('AuthController 2FA flows', () => {
).rejects.toMatchObject({ statusCode: 400 });
});
it('configure-2fa enable: refuses a missing or wrong code and leaves 2FA off', async () => {
const { user, actor } = await makeUserAndActor({ email_confirmed: 1 });
const { secret } = await setup2fa(actor);
await expect(enable2fa(actor)).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'bad_request',
});
// A live code for a secret the server never issued, as from a
// session enrolling an authenticator of its own.
const foreign = otpCreateSecret(user.username).secret;
await expect(
enable2fa(actor, { code: liveTotp(user.username, foreign) }),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'code_mismatch',
});
// Not coerced: a number is not a code.
await expect(
enable2fa(actor, {
code: Number(liveTotp(user.username, secret)),
}),
).rejects.toMatchObject({ statusCode: 400 });
const after = await server.stores.user.getById(user.id, {
force: true,
});
expect(after!.otp_enabled).toBeFalsy();
expect(after!.otp_secret).toBe(secret);
});
it('configure-2fa enable: 403 if email is unconfirmed; 409 if already enabled or no secret', async () => {
// Email unconfirmed → 403.
const { actor: aUnconfirmed } = await makeUserAndActor();
@@ -7961,10 +8482,7 @@ describe('AuthController 2FA additional branches', () => {
it('configure-2fa test: returns ok:false on a mismatched code', async () => {
// Setup so otp_secret is populated.
const { user, actor } = await makeUserAndActor();
await controller.handleConfigure2fa(
makeReq({}, { actor, params: { action: 'setup' } }),
makeRes(),
);
await setup2fa(actor);
const refreshed = await server.stores.user.getById(user.id, {
force: true,
});
@@ -7979,7 +8497,7 @@ describe('AuthController 2FA additional branches', () => {
expect(res.body).toEqual({ ok: false });
});
it('configure-2fa enable: succeeds when email is confirmed and a secret exists', async () => {
it('configure-2fa enable: succeeds with a live code for the stored secret', async () => {
const { user, actor } = await makeUserAndActor({ email_confirmed: 1 });
// Bootstrap a secret directly so we don't depend on the setup
// handler's side effects.
@@ -7991,7 +8509,10 @@ describe('AuthController 2FA additional branches', () => {
const res = makeRes();
await controller.handleConfigure2fa(
makeReq({}, { actor, params: { action: 'enable' } }),
makeReq(
{ code: liveTotp(user.username, 'TESTSECRETBASE32') },
{ actor, params: { action: 'enable' } },
),
res,
);
expect(res.body).toEqual({});
+175 -48
View File
@@ -34,7 +34,11 @@ import {
} from '../../core/http/middleware/gates.js';
import type { Actor } from '../../core/actor.js';
import { isPlainUserActor, makeActor } from '../../core/actor.js';
import { checkRateLimit } from '../../core/http/middleware/rateLimit.js';
import {
checkRateLimit,
consumeRouteRateLimit,
peekRouteRateLimit,
} from '../../core/http/middleware/rateLimit.js';
import {
signStepUpToken,
STEP_UP_COOKIE_NAME,
@@ -116,6 +120,8 @@ const FINGERPRINT_MAX_LENGTH = 128;
// One consent prompt covers a handful of scopes at most. The cap keeps a
// crafted request from turning a single grant call into a bulk write.
const MAX_PERMISSIONS_PER_REQUEST = 16;
// Rides every row a request writes, and the per-(user, app) cache after that.
const GRANT_EXTRA_MAX_BYTES = 4096;
const DISPATCH_ID_MAX_LENGTH = 128;
// One name for the flag, so the write and the read cannot drift apart.
const APP_AUTHENTICATED_FLAG = 'flag:app-is-authenticated';
@@ -215,6 +221,26 @@ const SESSION_LIMIT = {
window: 60_000,
key: 'user',
} as const;
/**
* Completed renames. Charged by the handler after the rename lands, so a wrong
* password or a taken name doesn't spend it.
*/
const CHANGE_USERNAME_LIMIT = {
scope: 'change-username-done',
limit: 2,
window: 30 * 24 * 60 * 60_000,
key: 'user',
} as const;
/** Rename attempts, success or not. Bounds password guessing through the gate. */
const CHANGE_USERNAME_ATTEMPT_LIMIT = {
scope: 'change-username-attempt',
limit: 10,
window: 60 * 60_000,
key: 'user',
} as const;
// How long a failed-SMS-send record stays readable by its error_id — long
// enough to cover the typical support round-trip.
const SMS_SEND_ERROR_TTL_SECONDS = 7 * 24 * 60 * 60;
@@ -233,6 +259,15 @@ export const RESERVED_USERNAMES = new Set([
'contact',
'mail',
'email',
// Role mailboxes: a Puter address names its account, so these must never
// be ownable.
'abuse',
'postmaster',
'hostmaster',
'fbl',
'security',
'noreply',
'no-reply',
'null',
'undefined',
'test',
@@ -246,7 +281,7 @@ export const RESERVED_USERNAMES = new Set([
* Auth controller — login/logout, permission grants/revokes, session
* management, OTP, and permission checks.
*
* Routes are declared via decorators (@Get/@Post on each handler). The five
* Routes are declared via decorators (@Get/@Post on each handler). The
* `/user-protected/*` and `/user-protected/delete-own-user` routes also need a
* per-instance `createUserProtectedGate(...)` middleware built from
* `this.config / this.stores / this.services`, which can't live in a static
@@ -2625,7 +2660,7 @@ export class AuthController extends PuterController {
// -- User-protected mutations ------------------------------------
//
// The five `/user-protected/*` and `/user-protected/delete-own-user`
// The `/user-protected/*` and `/user-protected/delete-own-user`
// routes are wired in the `registerRoutes` override below because
// their `middleware: createUserProtectedGate(...)` argument depends
// on `this.config / this.stores / this.services` and so can't live
@@ -2743,9 +2778,18 @@ export class AuthController extends PuterController {
});
}
if (!(await peekRouteRateLimit(req, CHANGE_USERNAME_LIMIT))) {
throw new HttpError(
429,
'You can only change your username twice every 30 days.',
{ legacyCode: 'too_many_requests' },
);
}
await this.stores.user.update(req.actor!.user.id!, {
username: new_username,
});
await consumeRouteRateLimit(req, CHANGE_USERNAME_LIMIT);
// Rename the user's FS home from `/<old>` to `/<new>` and
// cascade the prefix to all descendants. Without this, any
@@ -3297,6 +3341,16 @@ export class AuthController extends PuterController {
legacyCode: 'bad_request',
});
}
if (
Buffer.byteLength(JSON.stringify(value), 'utf8') >
GRANT_EXTRA_MAX_BYTES
) {
throw new HttpError(
400,
`\`${key}\` may not exceed ${GRANT_EXTRA_MAX_BYTES} bytes`,
{ legacyCode: 'bad_request' },
);
}
}
}
@@ -3864,6 +3918,13 @@ export class AuthController extends PuterController {
async handleGrantDevApp(req: Request, res: Response): Promise<void> {
let { app_uid } = req.body ?? {};
const { origin, permission, extra, meta } = req.body ?? {};
this.#validateAppPermissionParams({
app_uid,
origin,
permission,
extra,
meta,
});
if (origin && !app_uid) {
// Registered apps only, for the same reason the user-app handlers
// insist on it: a synthesised `app-<uuidv5(origin)>` is resolved
@@ -4204,6 +4265,11 @@ export class AuthController extends PuterController {
legacyCode: 'bad_request',
});
}
if (permissions.length > MAX_PERMISSIONS_PER_REQUEST) {
throw new HttpError(400, 'Too many `permissions`', {
legacyCode: 'bad_request',
});
}
// Optional user-facing name for the manage-sessions UI. Trim and clamp
// to the same 64-char limit the rename endpoint enforces.
@@ -4237,13 +4303,25 @@ export class AuthController extends PuterController {
// Normalize specs: string → [string], [string] → [string, {}], [string, extra] → as-is
const normalized = permissions.map((spec) => {
if (typeof spec === 'string') return [spec];
if (Array.isArray(spec)) return spec;
throw new HttpError(
400,
'Each permission must be a string or [string, extra?]',
{ legacyCode: 'bad_request' },
);
const entry =
typeof spec === 'string'
? [spec]
: Array.isArray(spec)
? spec
: null;
if (!entry || typeof entry[0] !== 'string' || !entry[0]) {
throw new HttpError(
400,
'Each permission must be a string or [string, extra?]',
{ legacyCode: 'bad_request' },
);
}
// Same caps the grant routes apply, before a session or row exists.
this.#validateAppPermissionParams({
permission: entry[0],
extra: entry[1],
});
return entry;
});
const token = await this.services.auth.createAccessToken(
@@ -4303,11 +4381,51 @@ export class AuthController extends PuterController {
res.json({ ok: true });
}
// -- 2FA: setup (user-protected, wired in registerRoutes below) ---
async handleSetup2fa(req: Request, res: Response): Promise<void> {
const user = await this.stores.user.getById(req.actor!.user.id!, {
force: true,
});
if (!user)
throw new HttpError(404, 'User not found', {
legacyCode: 'not_found',
});
if (user.otp_enabled) {
throw new HttpError(409, '2FA is already enabled.', {
legacyCode: 'conflict',
});
}
const result = otpCreateSecret(user.username);
// Generate 10 recovery codes
const codes: string[] = [];
for (let i = 0; i < 10; i++) {
codes.push(createRecoveryCode());
}
const hashedCodes = codes.map((c) => hashRecoveryCode(c));
await this.clients.db.write(
'UPDATE `user` SET `otp_secret` = ?, `otp_recovery_codes` = ? WHERE `uuid` = ?',
[result.secret, hashedCodes.join(','), user.uuid],
);
await this.stores.user.invalidateById(user.id);
res.json({
url: result.url,
secret: result.secret,
codes,
});
}
// -- 2FA: configure ----------------------------------------------
@Post('/auth/configure-2fa/:action', {
subdomain: 'api',
requireUserActor: true,
// A member owing their team's 2FA reaches nothing else until this.
allowUnconfirmed: true,
rateLimit: TWO_FACTOR_LIMIT,
})
async handleConfigure2fa(req: Request, res: Response): Promise<void> {
@@ -4320,36 +4438,6 @@ export class AuthController extends PuterController {
legacyCode: 'not_found',
});
if (action === 'setup') {
if (user.otp_enabled) {
throw new HttpError(409, '2FA is already enabled.', {
legacyCode: 'conflict',
});
}
const result = otpCreateSecret(user.username);
// Generate 10 recovery codes
const codes: string[] = [];
for (let i = 0; i < 10; i++) {
codes.push(createRecoveryCode());
}
const hashedCodes = codes.map((c) => hashRecoveryCode(c));
await this.clients.db.write(
'UPDATE `user` SET `otp_secret` = ?, `otp_recovery_codes` = ? WHERE `uuid` = ?',
[result.secret, hashedCodes.join(','), user.uuid],
);
await this.stores.user.invalidateById(user.id);
res.json({
url: result.url,
secret: result.secret,
codes,
});
return;
}
if (action === 'test') {
const { code } = req.body ?? {};
if (!code)
@@ -4362,7 +4450,9 @@ export class AuthController extends PuterController {
}
if (action === 'enable') {
if (!user.email_confirmed) {
// A seat has no email by design, so this would bar it forever.
const seat = await this.stores.team.getOrgSeat(user.id);
if (!user.email_confirmed && !seat) {
throw new HttpError(
403,
'Email must be confirmed before enabling 2FA.',
@@ -4381,6 +4471,18 @@ export class AuthController extends PuterController {
{ legacyCode: 'conflict' },
);
}
// Proves the caller holds the secret setup issued; a session alone
// can't turn 2FA on.
const { code } = req.body ?? {};
if (typeof code !== 'string' || !code)
throw new HttpError(400, 'Missing `code`', {
legacyCode: 'bad_request',
});
if (!verifyOtp(user.username, user.otp_secret, code)) {
throw new HttpError(400, 'Incorrect code.', {
legacyCode: 'code_mismatch',
});
}
await this.clients.db.write(
'UPDATE `user` SET `otp_enabled` = ? WHERE `uuid` = ?',
@@ -4418,6 +4520,15 @@ export class AuthController extends PuterController {
legacyCode: 'not_found',
});
const seat = await this.stores.team.getOrgSeat(user.id);
if (seat && Number(seat.require_2fa) === 1) {
throw new HttpError(
409,
'Your team requires two-factor authentication, so it cannot be turned off.',
{ legacyCode: 'conflict' },
);
}
await this.clients.db.write(
'UPDATE `user` SET `otp_enabled` = ?, `otp_recovery_codes` = NULL, `otp_secret` = NULL WHERE `uuid` = ?',
[this.clients.db.booleanValue(false), user.uuid],
@@ -4671,7 +4782,7 @@ export class AuthController extends PuterController {
//
// The `@Controller('')` decorator would normally install a default
// `registerRoutes` walker that iterates `prototype[__puterRoutes]`.
// We override it here so we can ALSO wire the five
// We override it here so we can ALSO wire the
// `/user-protected/*` (and `/user-protected/delete-own-user`) routes
// whose `middleware: createUserProtectedGate(...)` argument is
// built from instance state — not expressible inside a static
@@ -4745,12 +4856,7 @@ export class AuthController extends PuterController {
{
requireUserActor: true,
requireVerified: true,
rateLimit: {
scope: 'change-username',
limit: 2,
window: 30 * 24 * 60 * 60_000,
key: 'user',
},
rateLimit: CHANGE_USERNAME_ATTEMPT_LIMIT,
middleware: [
createUserProtectedGate(
userProtectedDeps as never,
@@ -4798,6 +4904,27 @@ export class AuthController extends PuterController {
(req, res) => this.handleDisable2fa(req, res),
);
router.post(
'/user-protected/setup-2fa',
{
requireUserActor: true,
// A member owing their team's 2FA reaches nothing else until this.
allowUnconfirmed: true,
rateLimit: {
scope: 'setup-2fa',
limit: 10,
window: 60 * 60_000,
key: 'user',
},
middleware: [
createUserProtectedGate(
userProtectedDeps as never,
) as unknown as RequestHandler,
],
},
(req, res) => this.handleSetup2fa(req, res),
);
router.post(
'/user-protected/delete-own-user',
{
@@ -21,6 +21,7 @@ import { metrics } from '@opentelemetry/api';
import type { Request, Response } from 'express';
import { actorUid } from '../../core/actor.js';
import { Context } from '../../core/context.js';
import { abortOnDisconnect } from '../../core/http/abortOnDisconnect.js';
import { Controller } from '../../core/http/decorators.js';
import { HttpError, isHttpError } from '../../core/http/HttpError.js';
import { assertNotUserSession } from '../../core/http/middleware/gates.js';
@@ -325,7 +326,8 @@ export class DriverController extends PuterController {
if (req.actor) {
const permService = this.services.permission as unknown as
PermissionService | undefined;
| PermissionService
| undefined;
if (permService) {
// Build via PermissionUtil.join so any `:` in a driver or
// interface name is escaped — raw interpolation would let a
@@ -459,12 +461,7 @@ export class DriverController extends PuterController {
// A caller that hangs up mid-call gets nothing back, so long-running
// drivers watch this to stop working (and metering) as soon as it does.
// `close` after `finish` is the normal end of a response, not an abort.
const abort = new AbortController();
res.once('close', () => {
if (!res.writableFinished) abort.abort();
});
Context.set('abortSignal', abort.signal);
Context.set('abortSignal', abortOnDisconnect(res));
// Per-method lifecycle events, scoped to `driver.<iface>.<method>`.
// Subscribers can listen on `driver.*`, `driver.<iface>.*`, or the
+22 -9
View File
@@ -21,7 +21,7 @@ import type { RouteRateLimit } from '../../core/http/types';
import {
DEFAULT_FREE_SUBSCRIPTION,
DEFAULT_TEMP_SUBSCRIPTION,
FREE_SUBSCRIPTION_IDS,
subscriptionOverride,
} from '../../services/metering/consts.js';
// -- Shared event limits ---------------------------------------------
@@ -45,7 +45,7 @@ const userWindow = (
/**
* A plan-varying cap in route-gate shape: `limit` is what a subscribed account
* gets, `bySubscription` carves out the free tiers, unlisted plans get the
* gets, `bySubscription` carves out the free tiers, unlisted paid plans get the
* base.
*/
export interface TieredLimit {
@@ -67,11 +67,9 @@ export const limitFor = (
subscriptionId: string | null,
): number => {
if (subscriptionId === null) return tier.limit;
const own = tier.bySubscription[subscriptionId];
if (typeof own === 'number') return own;
return FREE_SUBSCRIPTION_IDS.has(subscriptionId)
? (tier.bySubscription[DEFAULT_FREE_SUBSCRIPTION] ?? tier.limit)
: tier.limit;
return (
subscriptionOverride(tier.bySubscription, subscriptionId) ?? tier.limit
);
};
/**
@@ -128,14 +126,20 @@ export const EVENTS_DURABLE_SUBSCRIPTIONS_MAX =
export const SUSPENDED_ROW_TTL_DAYS = 30;
/**
* Subscribe + unsubscribe calls per minute, per user.
* Subscribe calls per minute, per user.
*
* Both resolve a path and take a write, so a loop over them is a write loop.
* Resolves a path and takes a write, so a loop over subscribes is a write loop.
* Sized like the sharing verbs, which are the closest existing analogue: a
* client sets its subscriptions up once and then leaves them alone.
*/
export const EVENTS_SUBSCRIBE_LIMIT = userWindow('events:subscribe', 60);
/**
* Unsubscribe calls per minute, per user — its own budget, never the subscribe
* one.
*/
export const EVENTS_UNSUBSCRIBE_LIMIT = userWindow('events:unsubscribe', 600);
/**
* Subscription listings per minute, per user. Reads an index and returns one
* page, so it is budgeted well above the verbs that write.
@@ -321,6 +325,15 @@ export const EVENTS_WORKER_INVOCATION_LIMIT = userWindow(
60,
);
/**
* Handler runs one chain of events may take, by the holder's plan.
*
* A write made with a handler's token is one run deeper than the event that
* invoked it. An event at this depth still reaches sockets but runs no handler,
* which is what ends handlers whose writes trigger each other.
*/
export const EVENTS_HANDLER_DEPTH = tiered(12, 4, 4);
/**
* Filter evaluations one event may spend. Lives with the matcher because the
* primitive that enforces it does; re-exported here so every published number
@@ -353,6 +353,70 @@ describe('GET /fs/readdir over HTTP', () => {
expect(stat.shares).toEqual([]);
});
});
describe('the root listing', () => {
const mint = async (permissions: string[]) => {
const response = await fetch(
new URL('/auth/create-access-token', env.apiOrigin),
{
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${env.users.user.token}`,
},
body: JSON.stringify({ permissions }),
},
);
expect(response.status).toBe(200);
return ((await response.json()) as { token: string }).token;
};
const listRoot = async (token: string) => {
const response = await fetch(
readdirUrl({ path: '/', auth_token: token }),
);
expect(response.status).toBe(200);
return response.text();
};
const rootPaths = async (token: string) => {
const entries = JSON.parse(await listRoot(token)) as Array<{
path: string;
}>;
return entries.map((entry) => entry.path);
};
const entryAt = async (path: string) =>
(await env.server.stores.fsEntry.getEntryByPath(path))!;
it('lists the home for a session and a full-access token', async () => {
const { username, token, apiToken } = env.users.user;
for (const credential of [token, apiToken]) {
expect(await rootPaths(credential)).toEqual([`/${username}`]);
}
});
it('leaves the home out for a token scoped below it', async () => {
const { username } = env.users.user;
const home = await entryAt(`/${username}`);
const documents = await entryAt(`/${username}/Documents`);
const scoped = await mint([`fs:${documents.uuid}:read`]);
const raw = await listRoot(scoped);
expect(JSON.parse(raw)).toEqual([]);
expect(raw).not.toContain(home.uuid);
expect(raw).not.toContain(username);
});
it('lists the home for a token that can list it', async () => {
const { username } = env.users.user;
const home = await entryAt(`/${username}`);
const scoped = await mint([`fs:${home.uuid}:list`]);
expect(await rootPaths(scoped)).toEqual([`/${username}`]);
});
});
});
describe('POST /fs/completeWrite over HTTP', () => {
@@ -71,7 +71,7 @@ const makeUser = async (): Promise<{ actor: Actor; userId: number }> => {
const refreshed = (await server.stores.user.getById(created.id))!;
return {
userId: refreshed.id,
actor: {
actor: makeActor({
user: {
id: refreshed.id,
uuid: refreshed.uuid,
@@ -79,7 +79,7 @@ const makeUser = async (): Promise<{ actor: Actor; userId: number }> => {
email: refreshed.email ?? null,
email_confirmed: true,
} as Actor['user'],
},
}),
};
};
@@ -843,6 +843,90 @@ describe('FSController.statEntry', () => {
);
expect((captured.body as { name: string }).name).toBe('share-budget');
});
describe('parentUid of a home child', () => {
const tokenActorFor = async (userId: number, permission: string) => {
const user = (await server.stores.user.getById(userId))!;
const token = await server.services.auth.createAccessToken(
makeActor({ user }),
[[permission]],
);
return (await server.services.auth.authenticateFromToken(token))!;
};
it('nulls it for a token scoped below the home', async () => {
const { userId } = await makeUser();
const user = (await server.stores.user.getById(userId))!;
const username = user.username!;
const documents = (await server.stores.fsEntry.getEntryByPath(
`/${username}/Documents`,
))!;
const scoped = await tokenActorFor(
userId,
`fs:${documents.uuid}:read`,
);
const { res, captured } = makeRes();
await withActor(scoped, () =>
controller.statEntry(
makeReq({
body: { path: `/${username}/Documents` },
actor: scoped,
}),
res,
),
);
expect((captured.body as { parentUid: unknown }).parentUid).toBe(
null,
);
});
it('keeps it for a token that can list the home', async () => {
const { userId } = await makeUser();
const user = (await server.stores.user.getById(userId))!;
const username = user.username!;
const home = (await server.stores.fsEntry.getEntryByPath(
`/${username}`,
))!;
const scoped = await tokenActorFor(userId, `fs:${home.uuid}:list`);
const { res, captured } = makeRes();
await withActor(scoped, () =>
controller.statEntry(
makeReq({
body: { path: `/${username}/Documents` },
actor: scoped,
}),
res,
),
);
expect((captured.body as { parentUid: unknown }).parentUid).toBe(
home.uuid,
);
});
it('keeps it for a session actor', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const home = (await server.stores.fsEntry.getEntryByPath(
`/${username}`,
))!;
const { res, captured } = makeRes();
await withActor(actor, () =>
controller.statEntry(
makeReq({
body: { path: `/${username}/Documents` },
actor,
}),
res,
),
);
expect((captured.body as { parentUid: unknown }).parentUid).toBe(
home.uuid,
);
});
});
});
// ── /readdir (readdirEntries) ───────────────────────────────────────
+72 -8
View File
@@ -25,6 +25,7 @@ import type { Actor } from '../../core/actor.js';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { Controller, Get, Post } from '../../core/http/decorators.js';
import { clientParentUid } from '../../services/fs/rootListing.js';
import {
expandTildePath,
isOwnersTrash,
@@ -52,7 +53,9 @@ import { consumeRouteRateLimit } from '../../core/http/middleware/rateLimit.js';
import { PuterController } from '../types.js';
import { STORAGE_OP_COSTS } from '../../services/metering/costs.js';
import {
FS_BATCH_WRITE_MAX_ITEMS,
FS_MULTIPART_LIMIT,
FS_MULTIPART_MAX_PARTS,
FS_MUTATE_LIMIT,
FS_READ_CONCURRENT,
FS_READ_LIMIT,
@@ -222,6 +225,7 @@ export class FSController extends PuterController {
const userId = this.#getActorUserId(req);
const storageAllowanceMax = this.#getStorageAllowanceMaxOverride(req);
const appUidLookupCache = new Map<string, Promise<number | null>>();
this.#assertBatchItemCount(req.body);
const requests = Array.isArray(req.body)
? await Promise.all(
req.body.map(async (requestBody) => {
@@ -367,6 +371,7 @@ export class FSController extends PuterController {
res: Response<ClientCompleteWriteResponse[]>,
) {
const userId = this.#getActorUserId(req);
this.#assertBatchItemCount(req.body);
const requests = Array.isArray(req.body)
? req.body.map((requestBody) => {
return this.#withGuiMetadata(
@@ -443,6 +448,17 @@ export class FSController extends PuterController {
res: Response<ClientSignMultipartPartsResponse>,
) {
const userId = this.#getActorUserId(req);
const partNumbers = req.body?.partNumbers;
if (
Array.isArray(partNumbers) &&
partNumbers.length > FS_MULTIPART_MAX_PARTS
) {
throw new HttpError(
400,
`Too many partNumbers in one request (max ${FS_MULTIPART_MAX_PARTS})`,
{ legacyCode: 'bad_request' },
);
}
await this.#assertUploadSessionWriteAccess(req, userId, [
req.body?.uploadId,
]);
@@ -863,6 +879,7 @@ export class FSController extends PuterController {
return;
}
this.#assertBatchItemCount(req.body);
const requests = Array.isArray(req.body)
? await Promise.all(
req.body.map(async (requestBody) => {
@@ -1015,7 +1032,7 @@ export class FSController extends PuterController {
legacyCode: 'too_many_requests',
});
}
const [subtreeSize, suggestedApps, shareFlags, shares] =
const [subtreeSize, suggestedApps, shareFlags, shares, parentUid] =
await Promise.all([
entry.isDir && wantsSize
? this.services.fs.getSubtreeSize(userId, entry.path)
@@ -1030,11 +1047,21 @@ export class FSController extends PuterController {
entry.uuid,
)
: undefined,
clientParentUid(
actor,
entry,
this.services.acl,
this.stores.permission,
),
]);
entry.suggestedApps = suggestedApps;
res.json({
...this.#toClientEntry(entry, shareFlags.get(entry.uuid) ?? null),
...this.#toClientEntry(
entry,
shareFlags.get(entry.uuid) ?? null,
parentUid,
),
...(subtreeSize !== undefined ? { size: subtreeSize } : {}),
...(shares !== undefined ? { shares } : {}),
});
@@ -1047,8 +1074,16 @@ export class FSController extends PuterController {
* callers who only hold `see`/`list` on the entry — a share recipient, or
* (with public folders enabled) any authenticated user. The legacy read
* path already curates its output; this does the same for the v2 routes.
*
* `parentUid` is `entry.parentUid` unless `parentUidOverride` is passed
* (even as `null`) — callers reading a scoped access token's home pass the
* result of `clientParentUid` to hide it there.
*/
#toClientEntry(entry: FSEntry, isShared?: boolean | null): ClientFSEntry {
#toClientEntry(
entry: FSEntry,
isShared?: boolean | null,
parentUidOverride?: string | null,
): ClientFSEntry {
// Allowlist, not a denylist: a denylist silently ships every column
// added to `fsentries` later. Omits the numeric primary keys (`id`,
// `parentId`, `associatedAppId`), the storage columns, the owning
@@ -1060,7 +1095,10 @@ export class FSController extends PuterController {
return {
uuid: entry.uuid,
uid: entry.uid ?? entry.uuid,
parentUid: entry.parentUid ?? null,
parentUid:
parentUidOverride !== undefined
? parentUidOverride
: (entry.parentUid ?? null),
path: maskEntryPath(entry),
name: entry.name,
isDir: entry.isDir,
@@ -1181,6 +1219,8 @@ export class FSController extends PuterController {
const rootChildren = await listRootEntries(
actor,
this.stores.fsEntry,
this.services.acl,
this.stores.permission,
);
const rootSuggestions =
await this.services.suggestedApps.getSuggestedAppsForEntries(
@@ -1322,6 +1362,12 @@ export class FSController extends PuterController {
...this.#toClientEntry(
entry,
shareFlags.get(entry.uuid) ?? null,
await clientParentUid(
actor,
entry,
this.services.acl,
this.stores.permission,
),
),
// Fields the client cannot derive on its own.
type: fsEntryMimeType(entry),
@@ -2382,6 +2428,16 @@ export class FSController extends PuterController {
return guiMetadata;
}
#assertBatchItemCount(items: unknown): void {
if (Array.isArray(items) && items.length > FS_BATCH_WRITE_MAX_ITEMS) {
throw new HttpError(
400,
`Too many items in one request (max ${FS_BATCH_WRITE_MAX_ITEMS})`,
{ legacyCode: 'bad_request' },
);
}
}
/**
* The write handlers build on `req.body` being an object. A request whose
* body never parsed leaves it undefined, and the first field read after
@@ -2596,8 +2652,9 @@ export class FSController extends PuterController {
};
}
/** `forOwner` — these events go to the owner, not to the acting user. */
async #toGuiFsEntry(entry: FSEntry): Promise<Record<string, unknown>> {
return toLegacyEntry(this.clients.event, entry);
return toLegacyEntry(this.clients.event, entry, { forOwner: true });
}
async #emitGuiWriteEvent(
@@ -2703,10 +2760,13 @@ export class FSController extends PuterController {
return fsEntry;
}
const thumbnailPayload = { url: requestedThumbnail };
const thumbnailPayload = {
url: requestedThumbnail,
uuid: fsEntry.uuid,
};
// emitAndWait — the thumbnails extension may rewrite `url` from a
// data URL to an `s3://` pointer; plain `emit` races with the DB
// update below.
// data URL to an `s3://` pointer bound to `uuid`, or drop a pointer
// not minted for it; plain `emit` races with the DB update below.
await this.clients.event.emitAndWait(
'thumbnail.created',
thumbnailPayload,
@@ -2795,6 +2855,9 @@ export class FSController extends PuterController {
index: item.index,
contentType: item.contentType,
...(item.size !== undefined ? { size: item.size } : {}),
// The entry the upload lands on; its thumbnail key is
// bound to it.
item_uid: responses[item.index]?.objectKey ?? '',
}) as ThumbnailUploadPrepareItem,
),
};
@@ -3018,6 +3081,7 @@ export class FSController extends PuterController {
{ legacyCode: 'bad_request' },
);
}
this.#assertBatchItemCount(manifest.items);
const manifestGuiMetadata = this.#extractGuiMetadata(
manifest,
@@ -28,6 +28,7 @@ import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { generateDefaultFsentries } from '../../util/userProvisioning.js';
import type { FSController } from './FSController.js';
import { FS_BATCH_WRITE_MAX_ITEMS, FS_MULTIPART_MAX_PARTS } from './limits.js';
import type {
ClientSignedWriteResponse,
CompleteWriteRequest,
@@ -166,8 +167,18 @@ const buildMultipartBody = (parts: MultipartPart[]): Buffer => {
return Buffer.from(chunks.join(''), 'utf8');
};
const makeMultipartReq = (parts: MultipartPart[], actor: Actor): Request => {
const stream = Readable.from([buildMultipartBody(parts)]);
const makeMultipartReq = (
parts: MultipartPart[],
actor: Actor,
chunkSize?: number,
): Request => {
const body = buildMultipartBody(parts);
const chunks: Buffer[] = [];
const step = chunkSize ?? body.length;
for (let i = 0; i < body.length; i += step) {
chunks.push(body.subarray(i, i + step));
}
const stream = Readable.from(chunks);
return Object.assign(stream, {
body: undefined,
query: {},
@@ -228,6 +239,46 @@ describe('FSController.write', () => {
expect(stored?.size).toBe(5);
});
it('emits an upload.progress socket event reaching loaded === total', async () => {
const { actor, userId, username } = await makeUser();
const target = `/${username}/Documents/write-progress.txt`;
const content = 'progress-bytes';
const send = vi.spyOn(server.services.socket, 'send');
const { res } = makeRes();
try {
await withActor(actor, () =>
controller.write(
makeReq<WriteRequest>({
body: {
fileMetadata: {
path: target,
size: content.length,
contentType: 'text/plain',
},
fileContent: content,
} as WriteRequest,
actor,
}),
res,
),
);
expect(send).toHaveBeenCalledWith(
{ room: userId },
'upload.progress',
expect.objectContaining({
total: content.length,
loaded: content.length,
loaded_diff: content.length,
item_path: target,
}),
);
} finally {
send.mockRestore();
}
});
it('reports wasOverwrite and updates the row when overwriting', async () => {
const { actor, username } = await makeUser();
const target = `/${username}/Documents/write-overwrite.txt`;
@@ -611,6 +662,60 @@ describe('batch write handlers with a malformed element', () => {
});
});
describe('batch write handlers at the item cap', () => {
// Null elements fail the per-item body check, so which 400 comes back
// shows whether the cap ran before any item was read.
const handlers: Array<
[string, (c: typeof controller, r: Request, res: Response) => unknown]
> = [
[
'startBatchWrites',
(c, r, res) => c.startBatchWrites(r as never, res as never),
],
['batchWrites', (c, r, res) => c.batchWrites(r as never, res as never)],
[
'completeBatchWrites',
(c, r, res) => c.completeBatchWrites(r as never, res as never),
],
];
const callWith = async (
call: (typeof handlers)[number][1],
length: number,
) => {
const { actor } = await makeUser();
const req = makeReq({ actor });
(req as { body?: unknown }).body = Array(length).fill(null);
const { res } = makeRes();
return withActor(actor, async () => call(controller, req, res));
};
it.each(handlers)(
'%s rejects one item over the cap up front',
async (_label, call) => {
await expect(
callWith(call, FS_BATCH_WRITE_MAX_ITEMS + 1),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'bad_request',
message: `Too many items in one request (max ${FS_BATCH_WRITE_MAX_ITEMS})`,
});
},
);
it.each(handlers)(
'%s lets exactly the cap through to the per-item checks',
async (_label, call) => {
await expect(
callWith(call, FS_BATCH_WRITE_MAX_ITEMS),
).rejects.toMatchObject({
statusCode: 400,
message: 'A request body is required',
});
},
);
});
describe('FSController.write storage allowance', () => {
let limitedServer: PuterServer;
let limitedController: FSController;
@@ -807,15 +912,18 @@ describe('FSController.startWrite', () => {
it('attaches signed thumbnail upload targets published by a listener', async () => {
const { actor, username } = await makeUser();
const itemUids: string[] = [];
const listener = (_key: string, data: unknown) => {
const payload = data as {
items: Array<{
index: number;
item_uid: string;
uploadUrl?: string;
thumbnailUrl?: string;
}>;
};
for (const item of payload.items) {
itemUids.push(item.item_uid);
item.uploadUrl = `https://thumbs.test/put/${item.index}`;
item.thumbnailUrl = `https://thumbs.test/get/${item.index}`;
}
@@ -844,6 +952,13 @@ describe('FSController.startWrite', () => {
const body = captured.body as ClientSignedWriteResponse;
expect(body.thumbnailUploadUrl).toBe('https://thumbs.test/put/0');
expect(body.thumbnailUrl).toBe('https://thumbs.test/get/0');
// Listeners are told the entry the upload will complete into, so
// the thumbnail can be bound to it.
const session =
await server.stores.fsEntry.getPendingEntryBySessionId(
body.sessionId,
);
expect(itemUids).toEqual([session?.objectKey]);
} finally {
server.clients.event.off(
'thumbnail.upload.prepare',
@@ -1094,6 +1209,38 @@ describe('FSController.completeWrite', () => {
expect(stored?.thumbnail).toBe('https://thumbs.test/x.png');
});
it('tells thumbnail listeners which entry the thumbnail is for', async () => {
const { actor, username } = await makeUser();
const target = `/${username}/Documents/complete-bound-thumb.txt`;
const started = await startSignedWrite(actor, target, 2);
await fetch(started.url!, { method: 'PUT', body: 'yz' });
const seen: unknown[] = [];
const listener = (_key: string, data: unknown) => {
seen.push(data);
};
server.clients.event.on('thumbnail.created', listener as never);
try {
const { res } = makeRes();
await withActor(actor, () =>
controller.completeWrite(
makeReq<CompleteWriteRequest>({
body: {
uploadId: started.sessionId,
thumbnailData: 'https://thumbs.test/x.png',
},
actor,
}),
res,
),
);
} finally {
server.clients.event.off('thumbnail.created', listener as never);
}
const stored = await server.stores.fsEntry.getEntryByPath(target);
expect(seen).toEqual([expect.objectContaining({ uuid: stored?.uuid })]);
});
it('404s an unknown upload id', async () => {
const { actor } = await makeUser();
const { res } = makeRes();
@@ -1233,6 +1380,29 @@ describe('FSController.signMultipartParts', () => {
),
).rejects.toMatchObject({ statusCode: 404 });
});
// The session is unknown, so reaching the lookup would answer 404.
it.each([
[FS_MULTIPART_MAX_PARTS + 1, 400],
[FS_MULTIPART_MAX_PARTS, 404],
])('answers %i part numbers with %i', async (length, statusCode) => {
const { actor } = await makeUser();
const { res } = makeRes();
await expect(
withActor(actor, () =>
controller.signMultipartParts(
makeReq<SignMultipartPartsRequest>({
body: {
uploadId: uuidv4(),
partNumbers: Array(length).fill(1),
},
actor,
}),
res,
),
),
).rejects.toMatchObject({ statusCode });
});
});
// -- /fs/batchWrite (JSON mode) ---------------------------------------
@@ -1469,6 +1639,60 @@ describe('FSController.batchWrites (multipart)', () => {
expect(stored?.size).toBe(5);
});
it('coalesces streamed upload progress when the declared size is 0', async () => {
const { actor, userId, username } = await makeUser();
const path = `/${username}/Documents/mp-progress.bin`;
const content = 'x'.repeat(64 * 1024);
const chunkSize = 4 * 1024;
const send = vi.spyOn(server.services.socket, 'send');
const { res } = makeRes();
// Scoped to this user: earlier tests' trailing flushes can land here.
const progressSends = () =>
send.mock.calls.filter(
([spec, key]) =>
key === 'upload.progress' &&
(spec as { room?: unknown }).room === userId,
);
try {
await withActor(actor, () =>
controller.batchWrites(
makeMultipartReq(
[
{
kind: 'field',
name: 'manifest',
value: manifestFor([path]),
},
{
kind: 'file',
name: 'file-0',
filename: 'progress.bin',
content,
},
],
actor,
chunkSize,
),
res,
),
);
const sumDiffs = () =>
progressSends().reduce(
(sum, [, , data]) =>
sum + (data as { loaded_diff: number }).loaded_diff,
0,
);
await vi.waitFor(() => expect(sumDiffs()).toBe(content.length));
expect(progressSends().length).toBeLessThan(
content.length / chunkSize,
);
} finally {
send.mockRestore();
}
});
it('maps positional `file` parts onto manifest order', async () => {
const { actor, username } = await makeUser();
const paths = [`/${username}/Documents/mp-pos.txt`];
@@ -1612,6 +1836,35 @@ describe('FSController.batchWrites (multipart)', () => {
});
});
it('rejects a manifest over the item cap', async () => {
const { actor } = await makeUser();
const { res } = makeRes();
await expect(
withActor(actor, () =>
controller.batchWrites(
makeMultipartReq(
[
{
kind: 'field',
name: 'manifest',
value: JSON.stringify({
items: Array(
FS_BATCH_WRITE_MAX_ITEMS + 1,
).fill(null),
}),
},
],
actor,
),
res,
),
),
).rejects.toMatchObject({
statusCode: 400,
message: `Too many items in one request (max ${FS_BATCH_WRITE_MAX_ITEMS})`,
});
});
it('rejects a manifest item without fileMetadata', async () => {
const { actor } = await makeUser();
const { res } = makeRes();
@@ -18,6 +18,7 @@
*/
import type { Request, RequestHandler, Response } from 'express';
import { createHash } from 'node:crypto';
import { Readable, Writable } from 'node:stream';
import { v4 as uuidv4 } from 'uuid';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
@@ -534,6 +535,36 @@ describe('LegacyFSController.updateFsentryThumbnail', () => {
expect(row?.thumbnail).toBe(thumbnail);
});
it('tells thumbnail listeners which entry the thumbnail is for', async () => {
const { actor, username } = await makeUser();
const path = `/${username}/Documents/bound-thumb.txt`;
const { uuid } = await writeFileEntry(actor, path, 'body');
const seen: unknown[] = [];
const listener = (_key: string, data: unknown) => {
seen.push(data);
};
server.clients.event.on('thumbnail.created', listener as never);
try {
const { res } = makeRes();
await withActor(actor, () =>
controller.updateFsentryThumbnail(
makeReq({
body: {
uid: uuid,
thumbnail: 'data:image/png;base64,aGVsbG8=',
},
actor,
}),
res,
),
);
} finally {
server.clients.event.off('thumbnail.created', listener as never);
}
expect(seen).toEqual([expect.objectContaining({ uuid })]);
});
it("refuses to retag another user's entry", async () => {
const owner = await makeUser();
const stranger = await makeUser();
@@ -735,6 +766,26 @@ describe('LegacyFSController.file (signed streaming)', () => {
);
});
// Metering resolves the billed account's plan from this actor, and some
// plans are keyed on the account's email.
it('bills an anonymous read to the owner, email included', async () => {
const { actor, userId, username } = await makeUser();
const { query } = await signedQuery(
actor,
`/${username}/Documents/signed-billed.txt`,
'billed',
);
const req = makeReq({ query });
const { res, finished } = makeStreamRes();
await controller.file(req, res);
await finished;
expect(req.egressActor?.user).toMatchObject({
id: userId,
uuid: actor.user.uuid,
email: `${username}@test.local`,
});
});
it('rejects a signature minted before the owner was suspended', async () => {
const { actor, userId, username } = await makeUser();
const { query } = await signedQuery(
@@ -755,6 +806,341 @@ describe('LegacyFSController.file (signed streaming)', () => {
});
});
// -- signed URLs across an ownership change ----------------------------
describe('LegacyFSController signed URLs across an ownership change', () => {
type Signed = { uid: string; read_url: string; write_url?: string };
const queryOf = (url: string) => {
const parsed = new URL(url);
return {
uid: parsed.searchParams.get('uid')!,
expires: parsed.searchParams.get('expires')!,
signature: parsed.searchParams.get('signature')!,
};
};
// No actor: a link holder needs no account.
const listSigned = async (url: string) => {
const { res, captured } = makeRes();
await controller.file(makeReq({ query: queryOf(url) }), res);
return captured.body as Array<Signed & { fsentry_name: string }>;
};
const readSigned = async (url: string) => {
const { res, captured, finished } = makeStreamRes();
await controller.file(makeReq({ query: queryOf(url) }), res);
await finished;
return captured.text();
};
const signAs = async (actor: Actor, path: string): Promise<Signed> => {
const { res, captured } = makeRes();
await withActor(actor, () =>
controller.sign(
makeReq({
body: { items: [{ path, action: 'write' }] },
actor,
}),
res,
),
);
return (captured.body as { signatures: Signed[] }).signatures[0];
};
const moveAs = async (
actor: Actor,
body: { source: string; destination: string },
) =>
withActor(actor, () =>
controller.move(makeReq({ body, actor }), makeRes().res),
);
it('refuses the URL and the child URLs it minted once a write share hands the entry to another user', async () => {
const alice = await makeUser();
const bob = await makeUser();
const dir = `/${alice.username}/Documents/signed-dir`;
await writeFileEntry(alice.actor, `${dir}/a.txt`, 'alice-bytes');
const signed = await signAs(alice.actor, dir);
const [child] = await listSigned(signed.read_url);
expect(child.fsentry_name).toBe('a.txt');
expect(await readSigned(child.read_url)).toBe('alice-bytes');
// Bob lets Alice write into his folder, and moving hers there makes
// it Bob's. Then he revokes the share and adds a file of his own.
const inbox = await server.services.fs.mkdir(bob.userId, {
path: `/${bob.username}/Documents/inbox`,
});
const grant = `fs:${inbox.uuid}:write`;
await server.services.permission.grantUserUserPermission(
bob.actor,
alice.username,
grant,
{},
);
await moveAs(alice.actor, { source: dir, destination: inbox.path });
await server.services.permission.revokeUserUserPermission(
bob.actor,
alice.username,
grant,
);
const movedDir = `${inbox.path}/signed-dir`;
await writeFileEntry(bob.actor, `${movedDir}/bob.txt`, 'bob-bytes');
expect(
(await server.stores.fsEntry.getEntryByPath(movedDir))!.userId,
).toBe(bob.userId);
await expect(listSigned(signed.read_url)).rejects.toMatchObject({
statusCode: 403,
});
// Refused on the signature, even for a caller allowed to write there.
await expect(
withActor(bob.actor, () =>
controller.writeFile(
makeReq({
query: {
...queryOf(signed.write_url!),
operation: 'mkdir',
},
body: { name: 'via-old-url' },
actor: bob.actor,
}),
makeRes().res,
),
),
).rejects.toMatchObject({ statusCode: 403 });
// Moved descendants keep a cached owner until the entry cache expires.
await server.stores.fsEntry.invalidateEntryCacheByUuid(child.uid);
await expect(readSigned(child.read_url)).rejects.toMatchObject({
statusCode: 403,
});
});
it('keeps the URL working across moves and renames by the same owner', async () => {
const { actor, username } = await makeUser();
const dir = `/${username}/Documents/kept-dir`;
await writeFileEntry(actor, `${dir}/a.txt`, 'kept-bytes');
const signed = await signAs(actor, dir);
const [child] = await listSigned(signed.read_url);
await moveAs(actor, {
source: dir,
destination: `/${username}/Pictures`,
});
await withActor(actor, () =>
controller.rename(
makeReq({
body: { uid: signed.uid, new_name: 'renamed-dir' },
actor,
}),
makeRes().res,
),
);
const [listed] = await listSigned(signed.read_url);
expect(listed.fsentry_name).toBe('a.txt');
expect(await readSigned(listed.read_url)).toBe('kept-bytes');
expect(await readSigned(child.read_url)).toBe('kept-bytes');
});
it('still serves a URL signed before signatures were bound to the owner', async () => {
const { actor, username } = await makeUser();
const path = `/${username}/Documents/legacy.txt`;
const { uuid } = await writeFileEntry(actor, path, 'legacy-bytes');
const expires = Math.ceil(Date.now() / 1000) + 9_999_999_999_999;
const signature = createHash('sha256')
.update(`${uuid}/read/${signingCfg().secret}/${expires}`)
.digest('hex');
const url = `${signingCfg().apiBaseUrl}/file?uid=${uuid}&expires=${expires}&signature=${signature}`;
expect(await readSigned(url)).toBe('legacy-bytes');
});
it('refuses that same pre-binding URL once legacy_file_signatures is off', async () => {
const { actor, username } = await makeUser();
const path = `/${username}/Documents/legacy-off.txt`;
const { uuid } = await writeFileEntry(actor, path, 'legacy-bytes');
const expires = Math.ceil(Date.now() / 1000) + 9_999_999_999_999;
const signature = createHash('sha256')
.update(`${uuid}/read/${signingCfg().secret}/${expires}`)
.digest('hex');
const url = `${signingCfg().apiBaseUrl}/file?uid=${uuid}&expires=${expires}&signature=${signature}`;
const config = (
controller as unknown as { config: { legacy_file_signatures?: boolean } }
).config;
config.legacy_file_signatures = false;
try {
await expect(readSigned(url)).rejects.toMatchObject({
statusCode: 403,
});
} finally {
config.legacy_file_signatures = undefined;
}
});
it('still serves an owner-bound signature while legacy_file_signatures is off', async () => {
const { actor, username } = await makeUser();
const path = `/${username}/Documents/owner-bound-while-off.txt`;
await writeFileEntry(actor, path, 'owner-bytes');
const signed = await signAs(actor, path);
const config = (
controller as unknown as { config: { legacy_file_signatures?: boolean } }
).config;
config.legacy_file_signatures = false;
try {
expect(await readSigned(signed.read_url)).toBe('owner-bytes');
} finally {
config.legacy_file_signatures = undefined;
}
});
});
// -- /file: non-UUID uid skips the entry-store lookup -------------------
describe('LegacyFSController.file (non-UUID uid)', () => {
it('rejects a non-UUID uid without touching the entry store', async () => {
const spy = vi.spyOn(server.stores.fsEntry, 'getEntryByUuid');
const expires = Math.ceil(Date.now() / 1000) + 9_999_999_999_999;
const { res } = makeRes();
await expect(
controller.file(
makeReq({
query: {
uid: 'not-a-uuid',
expires: String(expires),
signature: 'deadbeef',
},
}),
res,
),
).rejects.toMatchObject({ statusCode: 403 });
expect(spy).not.toHaveBeenCalled();
spy.mockRestore();
});
it('still looks up a UUID-shaped uid that does not exist', async () => {
const spy = vi.spyOn(server.stores.fsEntry, 'getEntryByUuid');
const expires = Math.ceil(Date.now() / 1000) + 9_999_999_999_999;
const { res } = makeRes();
await expect(
controller.file(
makeReq({
query: {
uid: uuidv4(),
expires: String(expires),
signature: 'deadbeef',
},
}),
res,
),
).rejects.toMatchObject({ statusCode: 403 });
expect(spy).toHaveBeenCalledTimes(1);
spy.mockRestore();
});
it('looks up a uuid whose version and variant nibbles are nonstandard', async () => {
const spy = vi.spyOn(server.stores.fsEntry, 'getEntryByUuid');
const expires = Math.ceil(Date.now() / 1000) + 9_999_999_999_999;
const { res } = makeRes();
await expect(
controller.file(
makeReq({
query: {
uid: '0123abcd-0000-0000-0000-0123456789ab',
expires: String(expires),
signature: 'deadbeef',
},
}),
res,
),
).rejects.toMatchObject({ statusCode: 403 });
expect(spy).toHaveBeenCalledTimes(1);
spy.mockRestore();
});
});
// -- /file via a signature minted by someone other than the owner -------
//
// `/sign` binds the signature to the entry's owner, not to whoever calls
// `/sign` — a grantee reading a shared file, or an app signing inside its
// own AppData, both mint signatures over an entry they don't own. Covers
// that those still read through `/file`.
describe('LegacyFSController /file via a non-owner-minted signature', () => {
const queryOf = (url: string) => {
const parsed = new URL(url);
return {
uid: parsed.searchParams.get('uid')!,
expires: parsed.searchParams.get('expires')!,
signature: parsed.searchParams.get('signature')!,
};
};
it('reads through /file when a grantee with read access signs the URL', async () => {
const owner = await makeUser();
const grantee = await makeUser();
const path = `/${owner.username}/Documents/grantee-read.txt`;
const { uuid } = await writeFileEntry(owner.actor, path, 'grantee-bytes');
await server.services.permission.grantUserUserPermission(
owner.actor,
grantee.username,
`fs:${uuid}:read`,
{},
);
const { res: signRes, captured: signCaptured } = makeRes();
await withActor(grantee.actor, () =>
controller.sign(
makeReq({
body: { items: [{ uid: uuid, action: 'read' }] },
actor: grantee.actor,
}),
signRes,
),
);
const signed = (
signCaptured.body as { signatures: Array<{ read_url: string }> }
).signatures[0];
const { res, captured, finished } = makeStreamRes();
await controller.file(makeReq({ query: queryOf(signed.read_url) }), res);
await finished;
expect(captured.statusCode).toBe(200);
expect(captured.text()).toBe('grantee-bytes');
});
it('reads through /file when an app actor signs inside its AppData root', async () => {
const { actor, username } = await makeUser();
const appUid = `routes-sign-app-${uuidv4()}`;
const appActor = makeActor({ ...actor, app: { uid: appUid } });
const path = `/${username}/AppData/${appUid}/app-read.txt`;
await writeFileEntry(actor, path, 'app-bytes');
const { res: signRes, captured: signCaptured } = makeRes();
await withActor(appActor, () =>
controller.sign(
makeReq({
body: { items: [{ path, action: 'read' }] },
actor: appActor,
}),
signRes,
),
);
const signed = (
signCaptured.body as { signatures: Array<{ read_url: string }> }
).signatures[0];
const { res, captured, finished } = makeStreamRes();
await controller.file(makeReq({ query: queryOf(signed.read_url) }), res);
await finished;
expect(captured.statusCode).toBe(200);
expect(captured.text()).toBe('app-bytes');
});
});
// -- /writeFile operation dispatch -------------------------------------
describe('LegacyFSController.writeFile operations', () => {
File diff suppressed because it is too large. Load diff
+396 -150
View File
@@ -32,19 +32,31 @@ import { HttpError } from '../../core/http/HttpError.js';
import { RouteOptions } from '../../core/http/index.js';
import {
assertNotSuspended,
assertTeam2fa,
assertVerifiedAccount,
} from '../../core/http/middleware/gates.js';
import type { PuterRouter } from '../../core/http/PuterRouter.js';
import type { ACLService } from '../../services/acl/ACLService.js';
import { assertActorHasCredits } from '../../services/metering/enforcement.js';
import type { SignedFile } from '../../util/fileSigning.js';
import type { FSEntry } from '../../stores/fs/FSEntry.js';
import type {
SignAction,
SignedFile,
SigningConfig,
} from '../../util/fileSigning.js';
import {
NON_OWNER_SIGNATURE_TTL_SECONDS,
parseSignedQuery,
verifySignature,
} from '../../util/fileSigning.js';
import { APP_ICON_SIZES, getAppIconCdnUrl } from '../../util/appIcon.js';
import { expandTildePath } from '../../services/fs/resolveNode.js';
import {
joinChildPath,
normalizeAbsolutePath,
splitParentAndName,
} from '../../services/fs/resolveNode.js';
import { maskEntryPath } from '../../services/fs/sharePathMask.js';
import { clientParentUid } from '../../services/fs/rootListing.js';
import {
buildHostedBackingDenial,
hostedIndexUrlBackingIsUnavailable,
@@ -67,6 +79,7 @@ import {
FS_SEARCH_CONCURRENT,
FS_SEARCH_LIMIT,
FS_SIGN_LIMIT,
FS_SIGN_MAX_ITEMS,
FS_SIGNED_CONCURRENT,
FS_SIGNED_READ_LIMIT,
FS_SIGNED_WRITE_LIMIT,
@@ -77,6 +90,7 @@ import {
assertAccess,
assertCanCreate,
assertCanMoveInto,
expandClientPath,
getBoolean,
getString,
loadLegacyAssociatedApps,
@@ -90,15 +104,55 @@ type RouterCache = Map<string, RequestHandler | null>;
const additionalRoutePaths: Record<string, string> = {};
// Shape only: version and variant nibbles aren't checked, so a row whose uuid
// wasn't minted by an RFC-conforming generator still resolves.
const UUID_SHAPE =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
// Legacy `/batch` multipart upload caps. Each file is buffered fully into
// memory before any quota / storage check runs, so without these limits an
// authenticated caller could grow the process heap proportional to whatever
// they sent. Streaming uploads go through `/writeFile`; this path is for
// pre-v2 clients only.
const BATCH_MAX_FILE_SIZE = 100 * 1024 * 1024; // 100 MiB per file
const BATCH_MAX_TOTAL_SIZE = 256 * 1024 * 1024; // 256 MiB buffered per request
const BATCH_MAX_FILES = 64;
const BATCH_MAX_PARTS = 256;
const BATCH_MAX_FIELD_SIZE = 1 * 1024 * 1024; // 1 MiB per operation/fileinfo JSON
// Multipart carries at most one op per part, so JSON mode gets the same bound.
const BATCH_MAX_OPS = BATCH_MAX_PARTS;
const BATCH_MUTATING_OPS = new Set(['mkdir', 'shortcut', 'move', 'delete']);
// An op's `as` names the path it produced; later ops in the same batch refer
// to it as `$name` or `$name/rest`, since a deduped mkdir can land elsewhere.
function resolveBatchRef(raw: string, refs: Map<string, string>): string {
if (!raw.startsWith('$')) return raw;
const slashIdx = raw.indexOf('/');
const name = slashIdx === -1 ? raw.slice(1) : raw.slice(1, slashIdx);
const rest = slashIdx === -1 ? '' : raw.slice(slashIdx);
if (!name) {
throw new HttpError(400, 'Invalid batch reference', {
legacyCode: 'bad_request',
});
}
const base = refs.get(name);
if (base === undefined) {
throw new HttpError(400, `Unknown batch reference: $${name}`, {
legacyCode: 'bad_request',
});
}
// Rejects non-normalized results, so `$ref/../x` can't escape.
return normalizeAbsolutePath(base + rest);
}
const malformedMultipart = (err: unknown): HttpError =>
err instanceof HttpError
? err
: new HttpError(
400,
`Malformed multipart body${err instanceof Error ? `: ${err.message}` : ''}`,
{ legacyCode: 'bad_request' },
);
async function loadAdditionalRouter(
key: string,
@@ -461,11 +515,18 @@ export class LegacyFSController extends PuterController {
'see',
);
const [suggestedApps, appsById, shareFlags] = await Promise.all([
this.services.suggestedApps.getSuggestedApps(entry),
loadLegacyAssociatedApps(this.stores.app, [entry]),
this.services.share.shareFlags(actor, [entry]),
]);
const [suggestedApps, appsById, shareFlags, parentUid] =
await Promise.all([
this.services.suggestedApps.getSuggestedApps(entry),
loadLegacyAssociatedApps(this.stores.app, [entry]),
this.services.share.shareFlags(actor, [entry]),
clientParentUid(
actor,
entry,
this.services.acl,
this.stores.permission,
),
]);
entry.suggestedApps = suggestedApps;
const shaped = await toLegacyEntry(this.clients.event, entry, {
@@ -477,6 +538,7 @@ export class LegacyFSController extends PuterController {
},
appsById,
isShared: shareFlags.get(entry.uuid) ?? null,
parentUid,
});
// Optional hydrations:
@@ -525,6 +587,8 @@ export class LegacyFSController extends PuterController {
const rootChildren = await listRootEntries(
actor,
this.stores.fsEntry,
this.services.acl,
this.stores.permission,
);
const rootSuggestions =
await this.services.suggestedApps.getSuggestedAppsForEntries(
@@ -661,14 +725,21 @@ export class LegacyFSController extends PuterController {
// When `parent` is a path string, use it directly without requiring
// the entry to exist — `services.fs.mkdir` honors `create_missing_parents`
// and will materialize any missing intermediate directories.
let targetPath = rawPath;
const username = actor.user?.username;
let targetPath = rawPath.startsWith('/')
? await expandClientPath(this.stores.fsEntry, rawPath, username)
: rawPath;
if (body.parent !== undefined && !rawPath.startsWith('/')) {
let parentPath: string;
if (
typeof body.parent === 'string' &&
(body.parent.startsWith('/') || body.parent.startsWith('~'))
) {
parentPath = expandTildePath(body.parent, actor.user?.username);
parentPath = await expandClientPath(
this.stores.fsEntry,
body.parent,
username,
);
} else {
const parent = await resolveV1Selector(
this.stores.fsEntry,
@@ -1002,9 +1073,13 @@ export class LegacyFSController extends PuterController {
legacyCode: 'bad_request',
});
const parentPath = pathPosix.dirname(
rawPath.startsWith('/') ? rawPath : `/${rawPath}`,
const expanded = await expandClientPath(
this.stores.fsEntry,
rawPath,
actor.user?.username,
);
const targetPath = expanded.startsWith('/') ? expanded : `/${expanded}`;
const parentPath = pathPosix.dirname(targetPath);
if (parentPath === '/') {
throw new HttpError(400, 'Cannot touch in root', {
legacyCode: 'bad_request',
@@ -1019,7 +1094,7 @@ export class LegacyFSController extends PuterController {
);
await this.services.fs.touch(userId, {
path: rawPath,
path: targetPath,
setAccessed: getBoolean(body, 'set_accessed_to_now') ?? false,
setModified: getBoolean(body, 'set_modified_to_now') ?? false,
setCreated: getBoolean(body, 'set_created_to_now') ?? false,
@@ -1128,7 +1203,7 @@ export class LegacyFSController extends PuterController {
// emitAndWait is required: the thumbnails extension rewrites
// `event.url` from a data URL to an `s3://` pointer, and the DB
// write below needs to see that rewrite.
const event = { url: thumbnail };
const event = { url: thumbnail, uuid: entry.uuid };
await this.clients.event.emitAndWait('thumbnail.created', event, {});
await this.clients.db.write(
@@ -1264,6 +1339,7 @@ export class LegacyFSController extends PuterController {
// that guard every other authenticated FS route have to run here.
assertNotSuspended(actor!.user);
assertVerifiedAccount(actor!.user);
await assertTeam2fa(actor!.user, this.stores.team);
req.actor = assertResolvedActor(actor!);
Context.set('actor', actor);
@@ -1298,6 +1374,12 @@ export class LegacyFSController extends PuterController {
throw new HttpError(400, '`items` is required', {
legacyCode: 'bad_request',
});
if (items.length > FS_SIGN_MAX_ITEMS)
throw new HttpError(
400,
`Too many items in one request (max ${FS_SIGN_MAX_ITEMS})`,
{ legacyCode: 'bad_request' },
);
const actingApp = actor.effectiveApp;
const signingCfg = signingConfigFromAppConfig(this.config);
@@ -1433,25 +1515,12 @@ export class LegacyFSController extends PuterController {
writeFile = async (req: Request, res: Response): Promise<void> => {
const query = asRecord(req.query);
const signingCfg = signingConfigFromAppConfig(this.config);
verifySignature(
{
uid: query.uid as string,
expires: query.expires as string,
signature: query.signature as string,
},
const targetEntry = await this.#resolveSignedEntry(
query,
'write',
signingCfg,
);
const uid = typeof query.uid === 'string' ? query.uid : '';
const targetEntry = await resolveV1Selector(this.stores.fsEntry, {
uid,
});
if (!targetEntry)
throw new HttpError(404, 'Item not found', {
legacyCode: 'not_found',
});
// Owner suspension check.
const owner = await this.stores.user.getById(targetEntry.userId);
if (!owner)
@@ -1675,18 +1744,7 @@ export class LegacyFSController extends PuterController {
file = async (req: Request, res: Response): Promise<void> => {
const query = asRecord(req.query);
const signingCfg = signingConfigFromAppConfig(this.config);
verifySignature(
{
uid: query.uid as string,
expires: query.expires as string,
signature: query.signature as string,
},
'read',
signingCfg,
);
const uid = typeof query.uid === 'string' ? query.uid : '';
const entry = await resolveV1Selector(this.stores.fsEntry, { uid });
const entry = await this.#resolveSignedEntry(query, 'read', signingCfg);
// Owner-suspension guard — matches v1's /file. A signed URL stays
// valid forever by default, so a signature minted before a suspension
@@ -1710,6 +1768,7 @@ export class LegacyFSController extends PuterController {
uuid: owner.uuid,
id: owner.id,
username: owner.username,
email: owner.email ?? null,
suspended: !!(owner as { suspended?: unknown }).suspended,
},
};
@@ -2066,9 +2125,12 @@ export class LegacyFSController extends PuterController {
// silently ignores. Run the entry through `toLegacyEntry` first so
// the event payload matches what /stat et al. return, then overlay
// per-op extras (e.g. `old_path` for moves).
// `forOwner` — the audience is the owner, who can't read the actor's mask.
try {
const response = {
...(await toLegacyEntry(this.clients.event, entry)),
...(await toLegacyEntry(this.clients.event, entry, {
forOwner: true,
})),
...extra,
from_new_service: true,
};
@@ -2198,6 +2260,7 @@ export class LegacyFSController extends PuterController {
const results: unknown[] = [];
let hasError = false;
let sequentialFileIdx = 0;
const pathRefs = new Map<string, string>();
for (const spec of operationSpecs) {
try {
@@ -2205,6 +2268,16 @@ export class LegacyFSController extends PuterController {
const op = typeof record.op === 'string' ? record.op : '';
let shaped: unknown;
// Non-write ops draw the same budget as the standalone routes.
if (
BATCH_MUTATING_OPS.has(op) &&
!(await consumeRouteRateLimit(req, FS_MUTATE_LIMIT))
) {
throw new HttpError(429, 'Too many requests.', {
legacyCode: 'too_many_requests',
});
}
if (op === 'write') {
// Pair with a file part — prefer `item_upload_id`
// index (what puter-js sets), fall back to the op's
@@ -2238,15 +2311,22 @@ export class LegacyFSController extends PuterController {
legacyCode: 'bad_request',
});
}
const parentPath = getString(record, 'path') ?? '';
const expandedParent = expandTildePath(
const parentPath = resolveBatchRef(
getString(record, 'path') ?? '',
pathRefs,
);
const expandedParent = await expandClientPath(
this.stores.fsEntry,
parentPath,
username,
);
const targetPath =
expandedParent && expandedParent !== '/'
? `${expandedParent.replace(/\/+$/, '')}/${name}`
: `/${name}`;
// joinChildPath rejects a `name` carrying a slash or a
// `.`/`..` segment, so it can't reach outside the parent
// the ACL check below actually authorized.
const targetPath = joinChildPath(
expandedParent || '/',
name,
);
// Mirrors the per-op /write|/mkdir routes: assert write
// on the parent dir, but when the parent resolves to `/`
// fall back to the target path so the ACL check rides
@@ -2297,25 +2377,12 @@ export class LegacyFSController extends PuterController {
response.fsEntry,
);
} else if (op === 'mkdir') {
const parentPath = getString(record, 'path') ?? '';
const name = getString(record, 'name');
if (!name) {
throw new HttpError(400, 'mkdir op missing `name`', {
legacyCode: 'bad_request',
});
}
const expandedParent = expandTildePath(
parentPath,
username,
);
const targetPath =
expandedParent && expandedParent !== '/'
? `${expandedParent.replace(/\/+$/, '')}/${name}`
: `/${name}`;
const writeAclPath =
expandedParent && expandedParent !== '/'
? expandedParent.replace(/\/+$/, '')
: targetPath;
const { targetPath, writeAclPath } =
await this.#resolveBatchMkdirTarget(
record,
username,
pathRefs,
);
await assertAccess(
this.services.acl,
this.services.fs,
@@ -2333,12 +2400,13 @@ export class LegacyFSController extends PuterController {
'create_missing_parents',
) ?? false,
});
this.#registerBatchRef(record, pathRefs, entry.path);
await this.#emitGuiEvent('outer.gui.item.added', entry);
shaped = await toLegacyEntry(this.clients.event, entry);
} else if (op === 'shortcut') {
const parentPath = getString(record, 'path') ?? '';
const name = getString(record, 'name');
const shortcutToUid =
const shortcutToRaw =
getString(record, 'shortcut_to_uid') ??
getString(record, 'shortcut_to');
if (!name) {
@@ -2346,19 +2414,21 @@ export class LegacyFSController extends PuterController {
legacyCode: 'shortcut_target_not_found',
});
}
if (!shortcutToUid) {
if (!shortcutToRaw) {
throw new HttpError(
400,
'shortcut op missing `shortcut_to_uid`',
{ legacyCode: 'shortcut_target_not_found' },
);
}
// A path or a uid; the desktop sends a path.
const target = await resolveV1Selector(
this.stores.fsEntry,
{ uid: shortcutToUid },
resolveBatchRef(shortcutToRaw, pathRefs),
);
const expandedParent = expandTildePath(
parentPath,
const expandedParent = await expandClientPath(
this.stores.fsEntry,
resolveBatchRef(parentPath, pathRefs),
username,
);
const parent = await resolveV1Selector(
@@ -2388,13 +2458,21 @@ export class LegacyFSController extends PuterController {
await this.#emitGuiEvent('outer.gui.item.added', link);
shaped = await toLegacyEntry(this.clients.event, link);
} else if (op === 'move') {
const sourceRef =
typeof record.source === 'string'
? resolveBatchRef(record.source, pathRefs)
: record.source;
const destinationRef =
typeof record.destination === 'string'
? resolveBatchRef(record.destination, pathRefs)
: record.destination;
const source = await resolveV1Selector(
this.stores.fsEntry,
record.source,
sourceRef,
);
const destinationParent = await resolveV1Selector(
this.stores.fsEntry,
record.destination,
destinationRef,
);
await assertAccess(
this.services.acl,
@@ -2417,14 +2495,20 @@ export class LegacyFSController extends PuterController {
overwrite: getBoolean(record, 'overwrite') ?? false,
dedupeName: getBoolean(record, 'dedupe_name') ?? false,
});
this.#registerBatchRef(record, pathRefs, moved.path);
await this.#emitGuiEvent('outer.gui.item.moved', moved, {
old_path: source.path,
});
shaped = await toLegacyEntry(this.clients.event, moved);
} else if (op === 'delete') {
const pathField = getString(record, 'path');
const selectorRef =
pathField !== undefined
? resolveBatchRef(pathField, pathRefs)
: record;
const entry = await resolveV1Selector(
this.stores.fsEntry,
getString(record, 'path') ?? record,
selectorRef,
);
await assertAccess(
this.services.acl,
@@ -2473,64 +2557,100 @@ export class LegacyFSController extends PuterController {
}> = [];
const fileinfos: Array<Record<string, unknown>> = [];
let parseError: Error | null = null;
const bb = Busboy({
headers: req.headers,
limits: {
fileSize: BATCH_MAX_FILE_SIZE,
files: BATCH_MAX_FILES,
parts: BATCH_MAX_PARTS,
fieldSize: BATCH_MAX_FIELD_SIZE,
},
});
let totalBufferedBytes = 0;
// First error wins. Buffered files are dropped right away rather
// than held while the rest of the body drains.
const fail = (err: Error) => {
if (parseError) return;
parseError = err;
files.length = 0;
};
const tooLarge = (message: string) =>
new HttpError(413, message, {
legacyCode: 'too_large' as never,
});
const reserve = (bytes: number): boolean => {
totalBufferedBytes += bytes;
if (totalBufferedBytes <= BATCH_MAX_TOTAL_SIZE) return true;
fail(
tooLarge(
`Batch exceeds ${BATCH_MAX_TOTAL_SIZE} bytes in total`,
),
);
return false;
};
let bb: ReturnType<typeof Busboy>;
try {
bb = Busboy({
headers: req.headers,
limits: {
fileSize: BATCH_MAX_FILE_SIZE,
files: BATCH_MAX_FILES,
parts: BATCH_MAX_PARTS,
fieldSize: BATCH_MAX_FIELD_SIZE,
},
});
} catch (err) {
// Missing boundary or unparseable content-type.
reject(malformedMultipart(err));
return;
}
// Busboy emits these `*Limit` events when a configured cap is
// hit. Capture the first one as a 413 so callers get a clean
// signal instead of a silently-truncated upload.
bb.on('filesLimit', () => {
if (!parseError) {
parseError = new HttpError(
413,
bb.on('filesLimit', () =>
fail(
tooLarge(
`Too many files in batch (max ${BATCH_MAX_FILES})`,
{ legacyCode: 'too_large' as never },
);
}
});
bb.on('partsLimit', () => {
if (!parseError) {
parseError = new HttpError(
413,
),
),
);
bb.on('partsLimit', () =>
fail(
tooLarge(
`Too many parts in batch (max ${BATCH_MAX_PARTS})`,
{ legacyCode: 'too_large' as never },
);
}
});
bb.on('fieldsLimit', () => {
if (!parseError) {
parseError = new HttpError(
413,
'Too many fields in batch',
{ legacyCode: 'too_large' as never },
);
}
});
),
),
);
bb.on('fieldsLimit', () =>
fail(tooLarge('Too many fields in batch')),
);
bb.on('field', (fieldName, value) => {
bb.on('field', (fieldName, value, info) => {
if (parseError) return;
if (info?.valueTruncated) {
fail(
tooLarge(
`\`${fieldName}\` field exceeds ${BATCH_MAX_FIELD_SIZE} bytes`,
),
);
return;
}
if (!reserve(Buffer.byteLength(value))) return;
// Ignore operation_id / socket_id / misc fields — not
// needed for v2 batch semantics.
if (fieldName !== 'operation' && fieldName !== 'fileinfo') {
return;
}
let parsed: unknown;
try {
if (fieldName === 'operation') {
ops.push(JSON.parse(value));
} else if (fieldName === 'fileinfo') {
const parsed = JSON.parse(value);
fileinfos.push(
parsed && typeof parsed === 'object'
? (parsed as Record<string, unknown>)
: {},
);
}
// Ignore operation_id / socket_id / misc fields — not
// needed for v2 batch semantics.
} catch (err) {
parseError =
err instanceof Error ? err : new Error(String(err));
parsed = JSON.parse(value);
} catch {
fail(
new HttpError(
400,
`\`${fieldName}\` field is not valid JSON`,
{ legacyCode: 'bad_request' },
),
);
return;
}
if (fieldName === 'operation') {
ops.push(parsed);
} else {
fileinfos.push(
parsed && typeof parsed === 'object'
? (parsed as Record<string, unknown>)
: {},
);
}
});
@@ -2539,24 +2659,26 @@ export class LegacyFSController extends PuterController {
// For streaming uploads use the signed `/writeFile` endpoint.
bb.on('file', (_fieldName, stream, info) => {
const chunks: Buffer[] = [];
let truncated = false;
stream.on('data', (chunk: Buffer) => chunks.push(chunk));
// Busboy emits `limit` after writing the first byte past
// `fileSize`. The stream continues being drained so the
// multipart parser stays in sync, but we discard the (now
// truncated) buffer and mark the batch as failed.
stream.on('limit', () => {
truncated = true;
if (!parseError) {
parseError = new HttpError(
413,
`File in batch exceeds ${BATCH_MAX_FILE_SIZE} bytes`,
{ legacyCode: 'too_large' as never },
);
// After a failure the stream keeps draining so the multipart
// parser reaches `close`, but nothing more is held.
stream.on('data', (chunk: Buffer) => {
if (parseError || !reserve(chunk.length)) {
chunks.length = 0;
return;
}
chunks.push(chunk);
});
// Busboy emits `limit` after the first byte past `fileSize`.
stream.on('limit', () => {
chunks.length = 0;
fail(
tooLarge(
`File in batch exceeds ${BATCH_MAX_FILE_SIZE} bytes`,
),
);
});
stream.on('end', () => {
if (truncated) return;
if (parseError) return;
files.push({
content: Buffer.concat(chunks),
mimeType:
@@ -2569,16 +2691,19 @@ export class LegacyFSController extends PuterController {
: undefined,
});
});
stream.on('error', (err: Error) => {
parseError = err;
});
stream.on('error', (err: Error) => fail(err));
});
bb.on('close', () => {
if (parseError) reject(parseError);
if (parseError) reject(malformedMultipart(parseError));
else resolve({ ops, files, fileinfos });
});
bb.on('error', (err) => reject(err));
bb.on('error', (err) => reject(malformedMultipart(err)));
// A client that disconnects mid-body never lets busboy close.
req.on('error', (err) => reject(malformedMultipart(err)));
req.on('aborted', () =>
reject(malformedMultipart(new Error('Request aborted'))),
);
req.pipe(bb);
});
@@ -2586,9 +2711,100 @@ export class LegacyFSController extends PuterController {
#parseJsonBatch(req: Request): unknown[] {
const body = asRecord(req.body);
if (Array.isArray(body.operations)) return body.operations;
if (Array.isArray(body.ops)) return body.ops;
return [];
const ops = Array.isArray(body.operations)
? body.operations
: Array.isArray(body.ops)
? body.ops
: [];
if (ops.length > BATCH_MAX_OPS) {
throw new HttpError(
413,
`Too many operations in batch (max ${BATCH_MAX_OPS})`,
{ legacyCode: 'too_large' as never },
);
}
return ops;
}
/**
* Batch `mkdir` shapes: `{ parent, path }` with `path` relative to
* `parent`, `{ path, name }` with `path` as the parent, or an absolute `{
* path }`.
*/
async #resolveBatchMkdirTarget(
record: Record<string, unknown>,
username: string | undefined,
pathRefs: Map<string, string>,
): Promise<{ targetPath: string; writeAclPath: string }> {
// Write is checked on the immediate parent: a multi-segment relative
// path must not be authorized by a grant on the target itself. Under
// `/` there is no parent to check, so the target stands in.
const withAclPath = (targetPath: string) => {
const parentPath = pathPosix.dirname(targetPath);
return {
targetPath,
writeAclPath: parentPath === '/' ? targetPath : parentPath,
};
};
const name = getString(record, 'name');
if (typeof record.parent === 'string' && !name) {
const parentPath = await expandClientPath(
this.stores.fsEntry,
resolveBatchRef(record.parent, pathRefs),
username,
);
const segments = (getString(record, 'path') ?? '')
.split('/')
.filter((s) => s.length > 0);
if (segments.length === 0) {
throw new HttpError(400, 'mkdir op missing `name`', {
legacyCode: 'bad_request',
});
}
let targetPath = parentPath || '/';
for (const segment of segments) {
targetPath = joinChildPath(targetPath, segment);
}
return withAclPath(targetPath);
}
if (name) {
const parentPath = await expandClientPath(
this.stores.fsEntry,
resolveBatchRef(getString(record, 'path') ?? '', pathRefs),
username,
);
return withAclPath(joinChildPath(parentPath || '/', name));
}
const rawPath = getString(record, 'path');
if (!rawPath) {
throw new HttpError(400, 'mkdir op missing `name`', {
legacyCode: 'bad_request',
});
}
const { parentPath, name: derivedName } = splitParentAndName(
await expandClientPath(
this.stores.fsEntry,
resolveBatchRef(rawPath, pathRefs),
username,
),
);
return withAclPath(joinChildPath(parentPath, derivedName));
}
/** Record an op's `as` (a bare name, no slash) for later `$name` refs. */
#registerBatchRef(
record: Record<string, unknown>,
pathRefs: Map<string, string>,
path: string,
): void {
const as = record.as;
if (typeof as === 'string' && as.length > 0 && !as.includes('/')) {
pathRefs.set(as, path);
}
}
#serializeBatchError(err: unknown): Record<string, unknown> {
@@ -2616,6 +2832,36 @@ export class LegacyFSController extends PuterController {
}
// -- Helpers ---------------------------------------------------------
/**
* Resolve the entry a signed URL names and verify the signature against its
* current owner. Malformed or expired params fail before the lookup.
*/
async #resolveSignedEntry(
query: Record<string, unknown>,
action: SignAction,
signingCfg: SigningConfig,
): Promise<FSEntry> {
const signed = {
uid: query.uid as string,
expires: query.expires as string,
signature: query.signature as string,
};
const { uid } = parseSignedQuery(signed);
// A non-UUID uid can never match a real entry — skip the cache/DB
// round trip a forged request would otherwise pay for on every hit.
const entry = UUID_SHAPE.test(uid)
? await this.stores.fsEntry.getEntryByUuid(uid)
: null;
verifySignature(signed, action, signingCfg, entry?.userId ?? null);
// Only a signature from before owner binding verifies without an entry.
if (!entry)
throw new HttpError(404, 'Subject does not exist', {
legacyCode: 'subject_does_not_exist',
});
return entry;
}
#requireActor(req: Request) {
const actor = req.actor;
if (!actor) {
@@ -0,0 +1,173 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
// Launch adopts whatever directory already sits at `AppData/<appUid>`, and an
// origin app's uid is derived from its origin, so another app knows where a
// not-yet-opened app's root will be.
describe('AppData roots over HTTP', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const call = (path: string, token: string, body?: unknown) =>
fetch(new URL(path, env.apiOrigin), {
method: body === undefined ? 'GET' : 'POST',
headers: {
authorization: `Bearer ${token}`,
...(body === undefined
? {}
: { 'content-type': 'application/json' }),
},
...(body === undefined ? {} : { body: JSON.stringify(body) }),
});
const origin = (label: string) =>
`https://${label}-${Math.random().toString(36).slice(2, 10)}.example.com`;
const launch = async (userToken: string, appOrigin: string) => {
const res = await call('/auth/get-user-app-token', userToken, {
origin: appOrigin,
});
expect(res.status).toBe(200);
return (await res.json()) as { token: string; app_uid: string };
};
const appRoot = async (app: { token: string; app_uid: string }) => {
const res = await call('/auth/request-app-root-dir', app.token, {
app_uid: app.app_uid,
});
expect(res.status).toBe(200);
return (await res.json()) as { uid: string; path: string };
};
it('refuses an app renaming its root onto another app’s uid', async () => {
const { username, token } = env.users.user;
const a = await launch(token, origin('appdata-a'));
const bOrigin = origin('appdata-b');
const bUid = await env.server.services.auth.appUidFromOrigin(bOrigin);
const root = await appRoot(a);
// A uid-bound grant on its own root, which would follow the root.
const sign = await call('/sign', a.token, {
items: [{ uid: root.uid, action: 'write' }],
app_uid: a.app_uid,
});
expect(sign.status).toBe(200);
const rename = await call('/rename', a.token, {
uid: root.uid,
new_name: bUid,
});
expect(rename.status).toBe(403);
const unmoved = await env.server.stores.fsEntry.getEntryByUuid(
root.uid,
);
expect(unmoved?.path).toBe(`/${username}/AppData/${a.app_uid}`);
// B gets a root of its own, out of A's reach.
const b = await launch(token, bOrigin);
expect(b.app_uid).toBe(bUid);
const bRoot = await appRoot(b);
expect(bRoot.uid).not.toBe(root.uid);
const secret = `/${username}/AppData/${bUid}/secret.txt`;
const write = await call('/fs/write', b.token, {
fileMetadata: {
path: secret,
size: Buffer.byteLength('b-private'),
contentType: 'text/plain',
},
fileContent: 'b-private',
encoding: 'utf8',
});
expect(write.status).toBe(200);
const read = await call(
`/read?file=${encodeURIComponent(secret)}`,
a.token,
);
expect(read.status).toBe(404);
});
it('refuses an app moving its root into the Trash', async () => {
const { username, token } = env.users.user;
const a = await launch(token, origin('appdata-trash'));
const root = await appRoot(a);
const move = await call('/move', a.token, {
source: root.uid,
destination: `/${username}/Trash`,
});
expect(move.status).toBe(403);
const unmoved = await env.server.stores.fsEntry.getEntryByUuid(
root.uid,
);
expect(unmoved?.path).toBe(`/${username}/AppData/${a.app_uid}`);
});
it('rewrites a forged trash original_path from an app instead of trusting it', async () => {
const { username, token } = env.users.user;
const a = await launch(token, origin('appdata-origin'));
const bUid = await env.server.services.auth.appUidFromOrigin(
origin('appdata-origin-b'),
);
const root = await appRoot(a);
const made = await call('/mkdir', a.token, {
path: `${root.path}/inner`,
});
expect(made.status).toBe(200);
const inner = (await made.json()) as { uid: string; path: string };
// Forging the other app's root as the trash origin would let the
// owner's (unchecked) restore recreate the file there. Refusing the
// move outright would also break an ordinary delete by anyone who
// isn't the account, so the forged keys are corrected instead.
const trash = await call('/move', a.token, {
source: inner.uid,
destination: `/${username}/Trash`,
new_name: inner.uid,
new_metadata: {
original_name: bUid,
original_path: `/${username}/AppData/${bUid}`,
trashed_ts: 1,
},
});
expect(trash.status).toBe(200);
const trashed = await env.server.stores.fsEntry.getEntryByUuid(
inner.uid,
);
const metadata = JSON.parse(trashed!.metadata!) as Record<
string,
unknown
>;
expect(metadata.original_path).toBe(inner.path);
expect(metadata.original_name).toBe('inner');
expect(metadata.trashed_ts).not.toBe(1);
});
});
@@ -256,7 +256,21 @@ describe('signingConfigFromAppConfig', () => {
url_signature_secret: 's3cret',
api_base_url: 'https://api.test',
} as never),
).toEqual({ secret: 's3cret', apiBaseUrl: 'https://api.test' });
).toEqual({
secret: 's3cret',
apiBaseUrl: 'https://api.test',
allowLegacySignatures: true,
});
});
it('stops accepting pre-binding signatures when legacy_file_signatures is false', () => {
expect(
signingConfigFromAppConfig({
url_signature_secret: 's3cret',
api_base_url: 'https://api.test',
legacy_file_signatures: false,
} as never).allowLegacySignatures,
).toBe(false);
});
it('fails loudly when the signing secret is missing', () => {
@@ -412,6 +426,23 @@ describe('toLegacyEntry', () => {
);
expect(shaped.associated_app).toBeNull();
});
it('uses the parentUid override, including null, over the entry’s own', async () => {
const hidden = await toLegacyEntry(undefined, baseEntry(), {
parentUid: null,
});
expect(hidden.parent_id).toBeNull();
expect(hidden.parent_uid).toBeNull();
const overridden = await toLegacyEntry(undefined, baseEntry(), {
parentUid: 'other-parent',
});
expect(overridden.parent_id).toBe('other-parent');
expect(overridden.parent_uid).toBe('other-parent');
const defaulted = await toLegacyEntry(undefined, baseEntry());
expect(defaulted.parent_uid).toBe('parent-1');
});
});
describe('loadLegacyAssociatedApps short-circuit', () => {
+72 -32
View File
@@ -22,6 +22,7 @@ import { contentType as contentTypeFromMime } from 'mime-types';
import type { FSEntry } from '../../stores/fs/FSEntry.js';
import type { FSEntryStore } from '../../stores/fs/FSEntryStore.js';
import type { FSService } from '../../services/fs/FSService.js';
import type { NodeRef } from '../../services/fs/resolveNode.js';
import type { ACLService, AclMode } from '../../services/acl/ACLService.js';
import { isAppActor, type Actor } from '../../core/actor.js';
import { Context } from '../../core/context.js';
@@ -31,6 +32,7 @@ import {
resolveNode,
normalizeAbsolutePath,
isOwnersTrash,
assertChildName,
joinChildPath,
expandTildePath,
} from '../../services/fs/resolveNode.js';
@@ -115,9 +117,32 @@ export async function expandClientPath(
);
}
export async function resolveV1Selector(
// Nested `{parent, name}` selectors recurse; unbounded depth overflows the stack.
const MAX_SELECTOR_DEPTH = 4;
/**
* Resolve `ref`, reporting any miss as the same 404 an unreadable entry gets,
* so a lookup never echoes a resolved path or tells missing from hidden.
*/
async function resolveOrNotFound(
fsEntryStore: FSEntryStore,
ref: NodeRef,
): Promise<FSEntry> {
try {
const entry = await resolveNode(fsEntryStore, ref, { required: true });
if (entry) return entry;
} catch (err) {
if (!(err instanceof HttpError && err.statusCode === 404)) throw err;
}
throw new HttpError(404, 'Subject does not exist', {
legacyCode: 'subject_does_not_exist',
});
}
async function resolveV1SelectorAtDepth(
fsEntryStore: FSEntryStore,
raw: unknown,
depth: number,
): Promise<FSEntry> {
const username = Context.get('actor')?.user?.username;
@@ -130,30 +155,29 @@ export async function resolveV1Selector(
const ref = isPath
? { path: await expandClientPath(fsEntryStore, raw, username) }
: { uid: raw };
const entry = await resolveNode(fsEntryStore, ref, { required: true });
if (!entry)
throw new HttpError(404, `Entry not found: ${raw}`, {
legacyCode: 'not_found',
});
return entry;
return resolveOrNotFound(fsEntryStore, ref);
}
const record = asRecord(raw);
// {parent, name}: "child selector" — resolve parent, then child by name.
if (record.parent !== undefined && typeof record.name === 'string') {
const parent = await resolveV1Selector(fsEntryStore, record.parent);
const childPath = joinChildPath(parent.path, record.name);
const child = await resolveNode(
fsEntryStore,
{ path: childPath },
{ required: true },
);
if (!child)
throw new HttpError(404, `Entry not found: ${childPath}`, {
legacyCode: 'not_found',
if (depth >= MAX_SELECTOR_DEPTH) {
throw new HttpError(400, 'Selector is nested too deeply', {
legacyCode: 'bad_request',
});
return child;
}
// Before the parent lookup, so a bad name can't tell whether the
// parent exists.
assertChildName(record.name);
const parent = await resolveV1SelectorAtDepth(
fsEntryStore,
record.parent,
depth + 1,
);
return resolveOrNotFound(fsEntryStore, {
path: joinChildPath(parent.path, record.name),
});
}
const rawPath = typeof record.path === 'string' ? record.path : undefined;
@@ -173,12 +197,14 @@ export async function resolveV1Selector(
? record.id
: undefined,
};
const entry = await resolveNode(fsEntryStore, ref, { required: true });
if (!entry)
throw new HttpError(404, 'Entry not found', {
legacyCode: 'not_found',
});
return entry;
return resolveOrNotFound(fsEntryStore, ref);
}
export async function resolveV1Selector(
fsEntryStore: FSEntryStore,
raw: unknown,
): Promise<FSEntry> {
return resolveV1SelectorAtDepth(fsEntryStore, raw, 0);
}
// -- ACL --------------------------------------------------------------
@@ -225,7 +251,7 @@ export async function assertAccess(
// through the error code. User-actor denials keep the real 403.
if (isAppActor(actor)) {
throw new HttpError(404, `Entry not found: path=${path}`, {
throw new HttpError(404, 'Subject does not exist', {
legacyCode: 'subject_does_not_exist',
});
}
@@ -438,7 +464,9 @@ export async function signEntryThumbnail(
* URL for a signed one. Pass `fsEntryStore`/`userStore` to hydrate `is_empty`
* (directories) and `owner` — both are required fields per the legacy stat
* contract but need extra DB lookups. Pass `appsById` (built via
* `loadLegacyAssociatedApps`) to populate `associated_app`.
* `loadLegacyAssociatedApps`) to populate `associated_app`. Pass `parentUid`
* (even as `null`, via `clientParentUid`) to override `parent_id`/`parent_uid`
* — callers reading a scoped access token's home use it to hide the uuid.
*/
export async function toLegacyEntry(
eventClient: EventClient | undefined,
@@ -451,11 +479,14 @@ export async function toLegacyEntry(
appsById?: Map<number, Record<string, unknown>>;
/** Omitted from the response when undefined. */
isShared?: boolean | null;
/** Skip the mask: the mask is the actor's, and only they can read it. */
forOwner?: boolean;
parentUid?: string | null;
} = {},
): Promise<Record<string, unknown>> {
// Someone else's entry is published under its masked path; the owner's
// real one, and everything above the share, stays server-side.
const publishedPath = maskEntryPath(entry);
const publishedPath = opts.forOwner ? entry.path : maskEntryPath(entry);
const dirname = pathPosix.dirname(publishedPath);
const mimeType = fsEntryMimeType(entry);
@@ -463,12 +494,17 @@ export async function toLegacyEntry(
const appdata_app =
pathComponents[2] === 'AppData' ? pathComponents[3] : undefined;
const parentUid =
opts.parentUid !== undefined
? opts.parentUid
: (entry.parentUid ?? null);
const response: Record<string, unknown> = {
id: entry.uuid,
uid: entry.uuid,
uuid: entry.uuid,
parent_id: entry.parentUid,
parent_uid: entry.parentUid,
parent_id: parentUid,
parent_uid: parentUid,
path: publishedPath,
dirname,
dirpath: dirname,
@@ -563,7 +599,11 @@ export function signingConfigFromAppConfig(config: IConfig): SigningConfig {
{ legacyCode: 'internal_error' },
);
}
return { secret, apiBaseUrl };
return {
secret,
apiBaseUrl,
allowLegacySignatures: config.legacy_file_signatures !== false,
};
}
/**
@@ -571,7 +611,7 @@ export function signingConfigFromAppConfig(config: IConfig): SigningConfig {
*
* Pass `actorUserId` so a signature over someone else's entry — a shared file —
* expires rather than outliving the share (see
* NON_OWNER_SIGNATURE_TTL_SECONDS).
* NON_OWNER_SIGNATURE_TTL_SECONDS). The signature is bound to `entry.userId`.
*/
export function signEntry(
entry: {
@@ -582,7 +622,7 @@ export function signEntry(
accessed: number | null;
modified: number;
created: number | null;
userId?: number;
userId: number;
},
config: SigningConfig,
opts: { actorUserId?: number; ttlSeconds?: number } = {},
+4 -1
View File
@@ -28,7 +28,10 @@ import type { RouteOptions, RouteRateLimit } from '../../core/http/types';
type Spec = RouteRateLimit | NonNullable<RouteOptions['concurrent']>;
const all = Object.entries(limits) as Array<[string, Spec | RouteRateLimit[]]>;
// Rate and concurrency specs only; the per-request item caps are plain numbers.
const all = Object.entries(limits).filter(
([, value]) => typeof value === 'object',
) as Array<[string, Spec | RouteRateLimit[]]>;
const flat: Array<[string, Spec]> = all.flatMap(([name, value]) =>
Array.isArray(value)
? value.map((v, i): [string, Spec] => [`${name}[${i}]`, v])
+27 -5
View File
@@ -185,15 +185,37 @@ export const FS_HELPER_LIMIT = userWindow('fs:helper', 120, 60, 30);
*/
export const FS_POLL_LIMIT = userWindow('fs:poll', 240, 240, 120);
// -- Per-request item caps -------------------------------------------
//
// The windows above count requests; these bound what one request can fan out
// to. Each item costs its own path resolution and ACL check, and an upload item
// also a pending session and presigned URLs. Checked before any per-item work.
/**
* `startBatchWrite` / `completeBatchWrite` / `batchWrite`. Puter.js chunks
* uploads at this size.
*/
export const FS_BATCH_WRITE_MAX_ITEMS = 500;
/** Entries per `/sign`. Each may also write an app grant. */
export const FS_SIGN_MAX_ITEMS = 500;
/**
* Part numbers per `signMultipartParts`: every part a multipart upload can have
* (1-10,000).
*/
export const FS_MULTIPART_MAX_PARTS = 10_000;
// -- Legacy multipart upload -----------------------------------------
/**
* `/batch` buffers every file fully into memory before any quota or storage
* check runs, up to BATCH_MAX_FILES × BATCH_MAX_FILE_SIZE. The concurrency slot
* is doing the real work here; the window is secondary, and sized to say so —
* one upload is one call, so a per-minute ceiling in the tens is a cap on how
* many files someone may upload rather than a bound on cost. What actually
* bounds the memory this route can tie up is how many run at once.
* check runs, up to BATCH_MAX_FILES × BATCH_MAX_FILE_SIZE, capped in aggregate
* per request by BATCH_MAX_TOTAL_SIZE. The concurrency slot is doing the real
* work here; the window is secondary, and sized to say so — one upload is one
* call, so a per-minute ceiling in the tens is a cap on how many files someone
* may upload rather than a bound on cost. What actually bounds the memory this
* route can tie up is how many run at once.
*/
export const FS_BATCH_LIMIT = userWindow('fs:batch', 600, 300, 300);
export const FS_BATCH_CONCURRENT = userConcurrent('fs:batch', 5, 2, 2);
@@ -0,0 +1,197 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import {
createTestUser,
setupPuterTestEnv,
type PuterTestEnv,
} from '../../testUtil.js';
// A recipient addresses the owner's entries by a masked `/<owner>/<uuid>/<name>`
// path. Routes that create an entry have no row to resolve, so they unmask it
// themselves before the ACL check runs against the result.
describe('writes inside a folder shared with write', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 180_000);
afterAll(async () => {
await env?.shutdown();
});
const post = async (route: string, token: string, body: unknown) => {
const res = await fetch(new URL(route, env.apiOrigin), {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${token}`,
},
body: JSON.stringify(body),
});
const text = await res.text();
return {
status: res.status,
json: text ? (JSON.parse(text) as Record<string, unknown>) : null,
};
};
const makeUser = () =>
createTestUser(env.server, {
username: `sfw${Math.random().toString(36).slice(2, 9)}`,
password: 'puter-test-user-password',
});
/** `/<owner>/SharedDir` with `note.txt`, shared `write` with a recipient. */
const shareAFolder = async () => {
const owner = await makeUser();
const recipient = await makeUser();
const dir = await post('/mkdir', owner.token, {
path: `/${owner.username}/SharedDir`,
});
await post('/touch', owner.token, {
path: `/${owner.username}/SharedDir/note.txt`,
});
const note = await post('/stat', owner.token, {
path: `/${owner.username}/SharedDir/note.txt`,
});
await post('/share', owner.token, {
recipients: [recipient.username],
items: [{ uid: dir.json!.uid }],
mode: 'write',
});
return {
owner,
recipient,
realDir: `/${owner.username}/SharedDir`,
maskedDir: `/${owner.username}/${dir.json!.uid}/SharedDir`,
noteUid: note.json!.uid as string,
};
};
const ownerListing = async (s: { owner: { token: string; username: string } }) => {
const res = await post('/readdir', s.owner.token, {
path: `/${s.owner.username}/SharedDir`,
});
return (res.json as unknown as Array<{ name: string }>).map(
(entry) => entry.name,
);
};
// `{parent, path}` is the shape puter.js sends — the GUI's "New Folder".
it('mkdir accepts a masked parent', async () => {
const s = await shareAFolder();
const made = await post('/mkdir', s.recipient.token, {
parent: s.maskedDir,
path: 'New Folder',
});
expect(made.status).toBe(200);
expect(made.json).toMatchObject({
name: 'New Folder',
is_dir: true,
// Anchored to the share root the request came through, so the
// recipient can navigate into what they just made.
path: `${s.maskedDir}/New Folder`,
});
expect(await ownerListing(s)).toContain('New Folder');
}, 180_000);
it('mkdir accepts a masked absolute path', async () => {
const s = await shareAFolder();
const made = await post('/mkdir', s.recipient.token, {
path: `${s.maskedDir}/Reports`,
});
expect(made.status).toBe(200);
expect(await ownerListing(s)).toContain('Reports');
}, 180_000);
it('touch accepts a masked path', async () => {
const s = await shareAFolder();
const touched = await post('/touch', s.recipient.token, {
path: `${s.maskedDir}/from-recipient.txt`,
});
expect(touched.status).toBe(200);
expect(await ownerListing(s)).toContain('from-recipient.txt');
}, 180_000);
it('touch still expands a tilde path', async () => {
const s = await shareAFolder();
const touched = await post('/touch', s.owner.token, {
path: '~/SharedDir/from-tilde.txt',
});
expect(touched.status).toBe(200);
expect(await ownerListing(s)).toContain('from-tilde.txt');
}, 180_000);
// Addressed to the owner, so it carries the path the owner can resolve.
it('publishes the owner’s real path on events about their entries', async () => {
const s = await shareAFolder();
const seen: Array<Record<string, unknown>> = [];
const emit = env.server.clients.event.emit.bind(
env.server.clients.event,
);
vi.spyOn(env.server.clients.event, 'emit').mockImplementation(
((name: string, payload: Record<string, unknown>, meta: unknown) => {
if (name.startsWith('outer.gui.item.')) {
seen.push(payload);
}
return emit(
name as never,
payload as never,
meta as never,
);
}) as never,
);
const moved = await post('/move', s.recipient.token, {
source: s.noteUid,
destination: s.maskedDir,
new_name: 'renamed.txt',
});
expect(moved.status).toBe(200);
const ownerId = (await env.server.stores.user.getByUsername(
s.owner.username,
))!.id;
const event = seen.find(
(payload) =>
(payload.response as { uid?: string })?.uid === s.noteUid,
);
expect(event?.user_id_list).toEqual([ownerId]);
expect(event?.response).toMatchObject({
path: `${s.realDir}/renamed.txt`,
dirpath: s.realDir,
});
// The recipient's own response stays masked.
expect(
(moved.json!.moved as { path: string }).path,
).toBe(`${s.maskedDir}/renamed.txt`);
}, 180_000);
});
@@ -427,6 +427,25 @@ describe('HomepageController GET /app/:name', () => {
});
});
// -- app landing rate limit --
describe('HomepageController app landing rate limit', () => {
it('rate-limits /app/:name and /desktop/app/:name per network', () => {
for (const path of ['/app/:name', '/desktop/app/:name']) {
const route = router.routes.find(
(r) => r.method === 'get' && r.path === path,
);
if (!route) throw new Error(`No GET ${path} route`);
expect(route.options.rateLimit).toEqual({
scope: 'app-landing',
limit: 600,
window: 60_000,
key: 'ip',
});
}
});
});
// ── /show/* ─────────────────────────────────────────────────────────
describe('HomepageController GET /show/*splat', () => {
@@ -31,6 +31,17 @@ import type {
/** Longest description/og:description/twitter:description on `/app/<name>`. */
export const APP_META_DESCRIPTION_MAX = 150;
/**
* Unauthenticated, rendered on every request, so the bucket is the network
* address rather than an account.
*/
const APP_LANDING_LIMIT = {
scope: 'app-landing',
limit: 600,
window: 60_000,
key: 'ip' as const,
};
/**
* App descriptions can run long; search snippets and social cards want a
* short blurb. Collapses whitespace, then cuts at a word boundary with an
@@ -160,8 +171,16 @@ export class HomepageController extends PuterController {
});
};
router.get('/app/:name', {}, sendAppShell);
router.get('/desktop/app/:name', {}, sendAppShell);
router.get(
'/app/:name',
{ rateLimit: APP_LANDING_LIMIT },
sendAppShell,
);
router.get(
'/desktop/app/:name',
{ rateLimit: APP_LANDING_LIMIT },
sendAppShell,
);
// -- /show/* - launch explorer with the requested file path --
@@ -93,6 +93,22 @@ export class HostingController extends PuterController {
userId: req.actor.user.id,
});
// Same event the `puter-subdomains` driver fires on delete —
// listeners (CDN purge, abuse tracking, worker cleanup)
// depend on it.
try {
this.clients.event.emit(
'subdomain.delete',
{
subdomain: row.subdomain,
uid: String(row.uuid),
},
{},
);
} catch {
// Non-critical.
}
res.json({});
},
);
@@ -181,4 +181,24 @@ describe('HostingController /delete-site', () => {
expect(captured.body).toEqual({});
expect(await server.stores.subdomain.getByUuid(site.uuid)).toBeNull();
});
it('emits subdomain.delete so listeners can purge caches', async () => {
const { userId, actor } = await makeUser();
const site = await makeSite(userId);
const emitSpy = vi.spyOn(server.clients.event, 'emit');
try {
await deleteSite(
makeReq({ body: { site_uuid: site.uuid }, actor }),
makeRes().res,
);
expect(emitSpy).toHaveBeenCalledWith(
'subdomain.delete',
{ subdomain: site.subdomain, uid: String(site.uuid) },
{},
);
} finally {
emitSpy.mockRestore();
}
});
});
@@ -448,9 +448,14 @@ describe('OIDCController GET /auth/oidc/:provider/start', () => {
return String(oidc().verifyState(state!)?.redirect_uri);
};
const returnToFor = (paths: string[], path = '/') => {
const returnToFor = (
paths: string[],
path = '/',
recipientUuid?: string,
) => {
const params = new URLSearchParams();
for (const p of paths) params.append('shared', p);
if (recipientUuid) params.set('user_uuid', recipientUuid);
return `${path}?${params.toString()}`;
};
@@ -484,6 +489,16 @@ describe('OIDCController GET /auth/oidc/:provider/start', () => {
]);
});
it('carries a valid recipient account hint with the share', async () => {
const recipientUuid = uuidv4();
const uri = await redirectUriFor(
returnToFor([sharedPath('Report.pdf')], '/', recipientUuid),
);
expect(new URL(uri).searchParams.get('user_uuid')).toBe(
recipientUuid,
);
});
it('carries no more items than a share link may name', async () => {
const paths = Array.from({ length: 25 }, (_, i) =>
sharedPath(`file-${i}.txt`),
@@ -505,6 +520,8 @@ describe('OIDCController GET /auth/oidc/:provider/start', () => {
// a parameter that isn't `shared`, alone or alongside one
'/?x=1',
`${returnToFor([sharedPath('a.txt')])}&x=1`,
`${returnToFor([sharedPath('a.txt')])}&user_uuid=not-a-uuid`,
`${returnToFor([sharedPath('a.txt')])}&user_uuid=${uuidv4()}&user_uuid=${uuidv4()}`,
// the root is only a destination when it names something
'/',
// still no origin smuggling, share link or not
@@ -684,6 +701,26 @@ describe('OIDCController login callback', () => {
expect(captured.redirectUrl).toContain('auth_error=1');
});
it('redirects with auth_error when the provider is unreachable', async () => {
const state = oidc().signState({
provider: 'custom',
redirect_uri: TEST_ORIGIN + '/',
});
vi.spyOn(oidc(), 'exchangeCodeForTokens').mockRejectedValue(
new TypeError('fetch failed'),
);
const { res, captured } = makeRes();
await callRoute(
'get',
'/auth/oidc/callback/login',
makeReq({ query: { code: 'c', state } }),
res,
);
expect(captured.redirectStatus).toBe(302);
expect(captured.redirectUrl).toContain('auth_error=1');
});
it('parses code/state from the POST body (Apple form_post)', async () => {
const state = oidc().signState({
provider: 'custom',
@@ -854,7 +891,7 @@ describe('OIDCController login callback', () => {
const state = oidc().signState({
provider: 'custom',
redirect_uri: `${TEST_ORIGIN}/?shared=${encodeURIComponent(shared)}`,
redirect_uri: `${TEST_ORIGIN}/?shared=${encodeURIComponent(shared)}&user_uuid=${encodeURIComponent(created.user!.uuid)}`,
});
vi.spyOn(oidc(), 'exchangeCodeForTokens').mockResolvedValue({
access_token: 'access',
@@ -877,6 +914,7 @@ describe('OIDCController login callback', () => {
expect(url.searchParams.get('auth_error')).toBe('1');
expect(url.searchParams.get('action')).toBe('login');
expect(url.searchParams.getAll('shared')).toEqual([shared]);
expect(url.searchParams.get('user_uuid')).toBe(created.user!.uuid);
});
it('redirects back to an /app/<name> landing after sign-in', async () => {
@@ -976,7 +1014,6 @@ describe('OIDCController login callback', () => {
makeReq({ query: { code: 'c', state } }),
res,
);
expect(captured.cookies).toHaveLength(1);
expect(captured.redirectUrl).toContain('embedded_in_popup=true');
expect(captured.redirectUrl).toContain('oidc_login=true');
@@ -990,6 +1027,15 @@ describe('OIDCController login callback', () => {
const user = await server.stores.user.getByEmail(email);
expect(user?.uuid).toBeTruthy();
expect(proof?.user_uuid).toBe(user!.uuid);
// Session cookie plus the proof's binding companion.
expect(captured.cookies).toHaveLength(2);
const binding = captured.cookies.find(
(c) => c.name === 'puter_oidc_popup_return',
);
expect(binding?.value).toBeTruthy();
expect(proof?.nonce).toBe(binding!.value);
expect(proof?.action).toBe('sign-in');
});
it('uses popup-style error URL (msg_id + opener_origin) when the popup-state user is suspended', async () => {
@@ -1041,6 +1087,14 @@ describe('OIDCController login callback', () => {
expect(captured.redirectUrl).toContain(
`opener_origin=${encodeURIComponent('http://opener.test')}`,
);
// The error leg must stamp the action its own redirect names.
const url = new URL(captured.redirectUrl!);
const proof = oidc().verifyPopupReturn(
url.searchParams.get('opener_state')!,
);
expect(proof?.oidc_login).toBe(false);
expect(proof?.action).toBe(url.searchParams.get('action'));
});
it('links an OIDC identity to an existing CONFIRMED password account via email match', async () => {
@@ -1692,6 +1746,30 @@ describe('OIDCController revalidate callback', () => {
expect(String(captured.body)).toContain('Missing');
});
it('returns 400 when the userinfo request cannot reach the provider', async () => {
const state = oidc().signState({
provider: 'custom',
flow: 'revalidate',
user_uuid: uuidv4(),
});
vi.spyOn(oidc(), 'exchangeCodeForTokens').mockResolvedValue({
access_token: 'access',
} as never);
vi.spyOn(oidc(), 'getUserInfo').mockRejectedValue(
new TypeError('fetch failed'),
);
const { res, captured } = makeRes();
await callRoute(
'get',
'/auth/oidc/callback/revalidate',
makeReq({ query: { code: 'c', state } }),
res,
);
expect(captured.statusCode).toBe(400);
expect(String(captured.body)).toContain('Please try again');
});
it('returns 403 when the OIDC sub resolves to a different account than the session', async () => {
// The linked account exists, but state.user_uuid points at someone else.
const sub = `sub-${Math.random().toString(36).slice(2, 8)}`;
@@ -1859,19 +1937,34 @@ describe('OIDCController GET /auth/revalidate-done', () => {
* attested rather than read.
*/
describe('OIDCController POST /auth/oidc/verify-popup-return', () => {
const redeem = async (opener_state: unknown) => {
const NONCE = 'binding-nonce';
const mint = (payload: Record<string, unknown>) =>
server.services.oidc.signPopupReturn({
nonce: NONCE,
action: 'sign-in',
...payload,
});
const redeemWith = (openerState: unknown, nonce: string | null = NONCE) =>
makeReq({
body: { opener_state: openerState },
cookies: nonce === null ? {} : { puter_oidc_popup_return: nonce },
});
const redeem = async (openerState: unknown) => {
const { res, captured } = makeRes();
await callRoute(
'post',
'/auth/oidc/verify-popup-return',
makeReq({ body: { opener_state } }),
redeemWith(openerState),
res,
);
return captured;
};
it('hands back what a genuine proof attests', async () => {
const proof = server.services.oidc.signPopupReturn({
const proof = mint({
opener_origin: 'https://opener.test',
msg_id: '77',
oidc_login: true,
@@ -1880,6 +1973,7 @@ describe('OIDCController POST /auth/oidc/verify-popup-return', () => {
expect(captured.body).toEqual({
opener_origin: 'https://opener.test',
msg_id: '77',
action: 'sign-in',
oidc_login: true,
user_uuid: null,
});
@@ -1888,7 +1982,7 @@ describe('OIDCController POST /auth/oidc/verify-popup-return', () => {
it('hands back the account a proof is bound to', async () => {
// The popup compares this against its current user to reject a proof
// replayed from another account's login.
const proof = server.services.oidc.signPopupReturn({
const proof = mint({
opener_origin: 'https://opener.test',
msg_id: '77',
oidc_login: true,
@@ -1899,9 +1993,13 @@ describe('OIDCController POST /auth/oidc/verify-popup-return', () => {
});
it('rejects a proof signed with someone else’s key', async () => {
// The whole point: only the server can mint one of these.
const forged = jwt.sign(
{ opener_origin: 'https://console.puter.com', oidc_login: true },
{
opener_origin: 'https://console.puter.com',
oidc_login: true,
purpose: 'popup-return',
nonce: NONCE,
},
'not-the-server-secret',
{ keyid: 'v2' },
);
@@ -1909,7 +2007,7 @@ describe('OIDCController POST /auth/oidc/verify-popup-return', () => {
callRoute(
'post',
'/auth/oidc/verify-popup-return',
makeReq({ body: { opener_state: forged } }),
redeemWith(forged),
makeRes().res,
),
).rejects.toMatchObject({ statusCode: 400 });
@@ -1920,14 +2018,19 @@ describe('OIDCController POST /auth/oidc/verify-popup-return', () => {
// redeemed on the very next request, so a stale one is never genuine.
const stale = server.services.token.sign(
'oidc-state',
{ opener_origin: 'https://opener.test', oidc_login: true },
{
opener_origin: 'https://opener.test',
oidc_login: true,
purpose: 'popup-return',
nonce: NONCE,
},
{ expiresIn: -600 },
);
await expect(
callRoute(
'post',
'/auth/oidc/verify-popup-return',
makeReq({ body: { opener_state: stale } }),
redeemWith(stale),
makeRes().res,
),
).rejects.toMatchObject({ statusCode: 400 });
@@ -1939,7 +2042,7 @@ describe('OIDCController POST /auth/oidc/verify-popup-return', () => {
callRoute(
'post',
'/auth/oidc/verify-popup-return',
makeReq({ body: { opener_state: bad } }),
redeemWith(bad),
makeRes().res,
),
).rejects.toMatchObject({ statusCode: 400 });
@@ -1949,13 +2052,82 @@ describe('OIDCController POST /auth/oidc/verify-popup-return', () => {
it('reports oidc_login false when the proof does not claim a login', async () => {
// The error leg mints one of these: a real return, but nothing was
// signed in on it, so it must not suppress the account picker.
const proof = server.services.oidc.signPopupReturn({
const proof = mint({
opener_origin: 'https://opener.test',
msg_id: '77',
oidc_login: false,
});
expect((await redeem(proof)).body).toMatchObject({ oidc_login: false });
});
it('rejects an authorization state presented as a proof', async () => {
const state = server.services.oidc.signState({
provider: 'google',
redirect_uri: 'https://puter.test/action/sign-in',
embedded_in_popup: true,
msg_id: '77',
opener_origin: 'https://opener.test',
nonce: NONCE,
});
await expect(
callRoute(
'post',
'/auth/oidc/verify-popup-return',
redeemWith(state),
makeRes().res,
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('rejects a proof redeemed without its binding cookie', async () => {
const proof = mint({
opener_origin: 'https://opener.test',
msg_id: '77',
oidc_login: true,
});
for (const cookie of [null, 'a-different-nonce']) {
await expect(
callRoute(
'post',
'/auth/oidc/verify-popup-return',
redeemWith(proof, cookie),
makeRes().res,
),
).rejects.toMatchObject({ statusCode: 400 });
}
});
it('clears the binding cookie so a proof redeems once', async () => {
const proof = mint({
opener_origin: 'https://opener.test',
msg_id: '77',
oidc_login: true,
});
const { res, captured } = makeRes();
await callRoute(
'post',
'/auth/oidc/verify-popup-return',
redeemWith(proof),
res,
);
expect(captured.clearedCookies).toContainEqual(
expect.objectContaining({ name: 'puter_oidc_popup_return' }),
);
});
it('leaves the binding cookie in place when redemption fails', async () => {
// A junk POST must not consume a cookie an in-flight return needs.
const { res, captured } = makeRes();
await expect(
callRoute(
'post',
'/auth/oidc/verify-popup-return',
redeemWith('not.a.proof'),
res,
),
).rejects.toMatchObject({ statusCode: 400 });
expect(captured.clearedCookies).toEqual([]);
});
});
// ── rate-limit scopes ───────────────────────────────────────────────
+112 -43
View File
@@ -18,6 +18,7 @@
*/
import crypto from 'node:crypto';
import { validate as validateUuid } from 'uuid';
import type { Request, Response } from 'express';
import { HttpError } from '../../core/http/HttpError.js';
import type { PuterRouter } from '../../core/http/PuterRouter.js';
@@ -28,6 +29,7 @@ import { parseMaskedSharePath } from '../../services/fs/sharePathMask.js';
import {
SHARE_DEEP_LINK_ITEMS_LIMIT,
SHARE_DEEP_LINK_PARAM,
SHARE_RECIPIENT_PARAM,
} from '../../services/share/shareDeepLink.js';
const REVALIDATION_COOKIE_NAME = 'puter_revalidation';
@@ -39,6 +41,14 @@ const REVALIDATION_EXPIRY_SEC = 300;
const OIDC_NONCE_COOKIE_NAME = 'puter_oidc_nonce';
const OIDC_NONCE_EXPIRY_SEC = 600;
// Single-use, and binds a popup-return proof to the browser that earned it.
// Expiry mirrors POPUP_RETURN_EXPIRY_SEC in OIDCService.
const OIDC_POPUP_RETURN_COOKIE_NAME = 'puter_oidc_popup_return';
const OIDC_POPUP_RETURN_EXPIRY_SEC = 300;
// The popup branch below hard-codes this return target.
const POPUP_RETURN_ACTION = 'sign-in';
const OIDC_ERROR_REDIRECT_MAP: Record<string, Record<string, string>> = {
login: { account_not_found: 'signup', other: 'login' },
signup: { account_already_exists: 'login', other: 'signup' },
@@ -112,14 +122,23 @@ function isWhitelistedReturnPath(path: string): boolean {
* or the recipient returns to a bare Home with nothing to say what they were
* sent.
*
* `shared` is the only parameter that makes the trip, and only values shaped
* like the masked path the mail was built from — the value is user-visible
* text, so a hand-edited one is refused rather than reflected back into the
* browser.
* Only `shared` makes the trip, with values shaped like the masked path the
* mail was built from — the value is user-visible text, so a hand-edited one is
* refused rather than reflected back into the browser. The mail's one recipient
* hint (`user_uuid`) rides along when it is a single uuid.
*/
function sharedPathsFromReturnQuery(query: string): string[] | null {
function sharedReturnQuery(query: string): {
paths: string[];
recipientUuid?: string;
} | null {
const paths: string[] = [];
let recipientUuid: string | undefined;
for (const [key, value] of new URLSearchParams(query)) {
if (key === SHARE_RECIPIENT_PARAM) {
if (recipientUuid || !validateUuid(value)) return null;
recipientUuid = value;
continue;
}
if (key !== SHARE_DEEP_LINK_PARAM) return null;
const parsed = parseMaskedSharePath(value);
// The segment after the uuid is the shared item itself. A mask without
@@ -135,7 +154,7 @@ function sharedPathsFromReturnQuery(query: string): string[] | null {
paths.push(value);
}
}
return paths;
return { paths, recipientUuid };
}
/**
@@ -154,15 +173,19 @@ function sanitizeReturnTo(raw: string): string | null {
const shared =
separator === -1
? []
: sharedPathsFromReturnQuery(raw.slice(separator + 1));
? { paths: [] }
: sharedReturnQuery(raw.slice(separator + 1));
if (shared === null) return null;
// The root is only a destination when it names something: on its own it is
// where the flow already lands.
if (shared.length === 0) return path === '/' ? null : path;
if (shared.paths.length === 0) return path === '/' ? null : path;
const params = new URLSearchParams();
for (const value of shared) params.append(SHARE_DEEP_LINK_PARAM, value);
for (const value of shared.paths)
params.append(SHARE_DEEP_LINK_PARAM, value);
if (shared.recipientUuid) {
params.set(SHARE_RECIPIENT_PARAM, shared.recipientUuid);
}
return `${path}?${params.toString()}`;
}
@@ -173,11 +196,8 @@ function buildErrorRedirectUrl(
message: string,
stateDecoded?: Record<string, unknown>,
requestCode?: string,
// Signs the popup-return proof. Passed in because this is a module-level
// helper with no access to services; omitted by callers that have no
// state to attest (the proof is simply absent then, and the popup falls
// back to its browser-attested sources).
signPopupReturn?: (payload: Record<string, unknown>) => string,
// Omitted by callers with no state to attest.
mintPopupReturn?: (payload: Record<string, unknown>) => string,
): string {
const targetFlow =
OIDC_ERROR_REDIRECT_MAP[sourceFlow]?.[errorCondition] ?? sourceFlow;
@@ -194,12 +214,15 @@ function buildErrorRedirectUrl(
// its success reloads it, so they have to be on it to survive.
let pagePath = '/';
let sharedPaths: string[] = [];
let shareRecipientUuid: string | undefined;
if (typeof stateDecoded?.redirect_uri === 'string') {
try {
const stateUrl = new URL(stateDecoded.redirect_uri);
if (isWhitelistedReturnPath(stateUrl.pathname)) {
pagePath = stateUrl.pathname;
sharedPaths = sharedPathsFromReturnQuery(stateUrl.search) ?? [];
const shared = sharedReturnQuery(stateUrl.search);
sharedPaths = shared?.paths ?? [];
shareRecipientUuid = shared?.recipientUuid;
}
} catch {
// unparsable redirect_uri: fall back to the root page
@@ -221,13 +244,15 @@ function buildErrorRedirectUrl(
// Same reasoning as the success leg: the popup cannot tell a verified
// `opener_origin` from a typed one, so attest it. The error leg is a
// real return from the provider too — the flow failed, not the hop.
if (signPopupReturn) {
if (mintPopupReturn) {
params.set(
'opener_state',
signPopupReturn({
mintPopupReturn({
opener_origin: stateDecoded?.opener_origin ?? null,
msg_id: stateDecoded?.msg_id ?? null,
oidc_login: false,
// The popup checks this against the action it lands on.
action: targetFlow,
}),
);
}
@@ -251,6 +276,9 @@ function buildErrorRedirectUrl(
for (const path of sharedPaths) {
params.append(SHARE_DEEP_LINK_PARAM, path);
}
if (shareRecipientUuid) {
params.set(SHARE_RECIPIENT_PARAM, shareRecipientUuid);
}
return `${base}${pagePath}?${params.toString()}`;
}
@@ -293,18 +321,21 @@ export class OIDCController extends PuterController {
// A sign-in popup returning from a provider is told the opener's
// origin and that a login completed. It cannot check either: the
// values arrive as query parameters, and a URL built from a verified
// `state` looks exactly like one an attacker typed. The opener's
// `state` looks exactly like one anybody can type. The opener's
// origin decides which app a token gets minted for, so the popup
// redeems the signed proof here instead of believing the raw
// parameters.
//
// Served on the GUI origin too: the binding cookie is host-only.
//
// Unauthenticated on purpose — it reveals nothing the caller did not
// already hand over, and a forged or expired proof yields nothing.
// already hand over, and a proof that isn't this browser's yields
// nothing.
router.post(
'/auth/oidc/verify-popup-return',
{
subdomain: 'api',
subdomain: ['api', ''],
rateLimit: {
scope: 'oidc-verify-popup-return',
limit: 60,
@@ -319,14 +350,25 @@ export class OIDCController extends PuterController {
});
}
const decoded = this.services.oidc.verifyPopupReturn(proof);
if (!decoded) {
const cookieNonce =
req.cookies?.[OIDC_POPUP_RETURN_COOKIE_NAME];
if (
!decoded ||
typeof decoded.nonce !== 'string' ||
typeof cookieNonce !== 'string' ||
!constantTimeEqual(cookieNonce, decoded.nonce)
) {
throw new HttpError(400, 'Invalid `opener_state`', {
legacyCode: 'bad_request',
});
}
// Single-use; a rejected call must not burn the cookie.
res.clearCookie(OIDC_POPUP_RETURN_COOKIE_NAME, { path: '/' });
res.json({
opener_origin: decoded.opener_origin ?? null,
msg_id: decoded.msg_id ?? null,
action: decoded.action ?? null,
oidc_login: decoded.oidc_login === true,
user_uuid: decoded.user_uuid ?? null,
});
@@ -430,7 +472,7 @@ export class OIDCController extends PuterController {
: null;
if (embeddedInPopup && msgId) {
appRedirectUri = `${origin}/action/sign-in?embedded_in_popup=true&msg_id=${encodeURIComponent(msgId)}`;
appRedirectUri = `${origin}/action/${POPUP_RETURN_ACTION}?embedded_in_popup=true&msg_id=${encodeURIComponent(msgId)}`;
if (openerOrigin) {
appRedirectUri += `&opener_origin=${encodeURIComponent(openerOrigin)}`;
}
@@ -557,7 +599,7 @@ export class OIDCController extends PuterController {
resolutionErrorCode(resolved.code),
stateDecoded,
resolved.requestCode,
(p) => this.services.oidc.signPopupReturn(p),
(p) => this.#mintPopupReturn(res, p),
),
);
}
@@ -576,7 +618,7 @@ export class OIDCController extends PuterController {
'account_suspended',
stateDecoded,
undefined,
(p) => this.services.oidc.signPopupReturn(p),
(p) => this.#mintPopupReturn(res, p),
),
);
}
@@ -624,7 +666,7 @@ export class OIDCController extends PuterController {
resolutionErrorCode(resolved.code),
stateDecoded,
resolved.requestCode,
(p) => this.services.oidc.signPopupReturn(p),
(p) => this.#mintPopupReturn(res, p),
),
);
}
@@ -640,7 +682,7 @@ export class OIDCController extends PuterController {
'account_suspended',
stateDecoded,
undefined,
(p) => this.services.oidc.signPopupReturn(p),
(p) => this.#mintPopupReturn(res, p),
),
);
}
@@ -757,6 +799,23 @@ if (window.opener) {
// -- Shared helpers ----------------------------------------------
/** Sign a popup-return proof, bound to this browser and this popup. */
#mintPopupReturn(res: Response, payload: Record<string, unknown>): string {
const nonce = crypto.randomBytes(32).toString('base64url');
res.cookie(OIDC_POPUP_RETURN_COOKIE_NAME, nonce, {
// Redeemed same-origin.
...sessionCookieFlags(this.config, { crossSite: false }),
httpOnly: true,
maxAge: OIDC_POPUP_RETURN_EXPIRY_SEC * 1000,
path: '/',
});
return this.services.oidc.signPopupReturn({
action: POPUP_RETURN_ACTION,
...payload,
nonce,
});
}
/**
* Resolve an OIDC callback to a Puter user. In order:
*
@@ -904,23 +963,33 @@ if (window.opener) {
const callbackUrl = this.services.oidc.getCallbackUrl(flow);
if (!callbackUrl) return { error: 'Invalid flow.' };
const tokens = await this.services.oidc.exchangeCodeForTokens(
provider,
code,
callbackUrl,
);
if (!tokens || !tokens.access_token)
return { error: 'Token exchange failed.' };
try {
const tokens = await this.services.oidc.exchangeCodeForTokens(
provider,
code,
callbackUrl,
);
if (!tokens || !tokens.access_token)
return { error: 'Token exchange failed.' };
const userinfo = await this.services.oidc.getUserInfo(
provider,
tokens.access_token,
typeof tokens.id_token === 'string' ? tokens.id_token : undefined,
);
if (!userinfo || !userinfo.sub)
return { error: 'Could not get user info.' };
const userinfo = await this.services.oidc.getUserInfo(
provider,
tokens.access_token,
typeof tokens.id_token === 'string'
? tokens.id_token
: undefined,
);
if (!userinfo || !userinfo.sub)
return { error: 'Could not get user info.' };
return { provider, userinfo, stateDecoded };
return { provider, userinfo, stateDecoded };
} catch (e) {
// Network failures reaching the provider; the user can retry.
console.warn(`[oidc] ${provider} request failed`, e);
return {
error: 'Could not reach the sign-in provider. Please try again.',
};
}
}
async #finishLogin(
@@ -969,7 +1038,7 @@ if (window.opener) {
target = appendQueryParam(
target,
'opener_state',
this.services.oidc.signPopupReturn({
this.#mintPopupReturn(res, {
opener_origin: stateDecoded.opener_origin ?? null,
msg_id: stateDecoded.msg_id ?? null,
oidc_login: true,
@@ -458,6 +458,7 @@ describe('PeerController TURN', () => {
uuid: created.uuid,
id: created.id,
username: created.username,
email: created.email,
},
effectiveApp: null,
});
@@ -456,6 +456,7 @@ export class PeerController extends PuterController {
uuid: user.uuid,
id: user.id,
username: user.username,
email: user.email ?? null,
},
});
await this.services.metering.incrementUsage(
@@ -45,7 +45,7 @@ import {
import type { Actor } from '../../core/actor.js';
import type { ChatCompletionDriver } from '../../drivers/ai-chat/ChatCompletionDriver.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { inRequestScope, setupTestServer } from '../../testUtil.js';
import { PuterAIController } from './PuterAIController.js';
let server: PuterServer;
@@ -53,7 +53,9 @@ let controller: PuterAIController;
beforeAll(async () => {
server = await setupTestServer();
controller = server.controllers.puterAi as unknown as PuterAIController;
controller = inRequestScope(
server.controllers.puterAi as unknown as PuterAIController,
);
});
afterAll(async () => {
@@ -48,7 +48,7 @@ import type { RouteOptions } from '../../core/http/index.js';
import type { ChatCompletionDriver } from '../../drivers/ai-chat/ChatCompletionDriver.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../../drivers/util/aiLimits.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { inRequestScope, setupTestServer } from '../../testUtil.js';
import { PuterAIController } from './PuterAIController.js';
// ── Test harness ────────────────────────────────────────────────────
@@ -58,7 +58,9 @@ let controller: PuterAIController;
beforeAll(async () => {
server = await setupTestServer();
controller = server.controllers.puterAi as unknown as PuterAIController;
controller = inRequestScope(
server.controllers.puterAi as unknown as PuterAIController,
);
});
afterAll(async () => {
@@ -130,6 +132,7 @@ const makeRes = () => {
captured.ended = true;
return res;
}),
once: vi.fn(() => res),
};
return { res: res as unknown as Response, captured };
};
@@ -20,6 +20,8 @@
import type { Request, Response } from 'express';
import crypto from 'node:crypto';
import { Readable } from 'node:stream';
import { Context } from '../../core/context.js';
import { abortOnDisconnect } from '../../core/http/abortOnDisconnect.js';
import { HttpError } from '../../core/http/HttpError.js';
import { RouteOptions } from '../../core/http/index.js';
import { computeNetworkFingerprint } from '../../core/http/middleware/rateLimit.js';
@@ -342,7 +344,7 @@ export class PuterAIController extends PuterController {
...openaiCompatProvider(body),
};
const result = await this.#driver().complete(completeArgs);
const result = await this.#complete(res, completeArgs);
const effectiveModel = completeArgs.model || '';
if (stream) {
@@ -500,7 +502,7 @@ export class PuterAIController extends PuterController {
const completionId = `cmpl-${randomId()}`;
const created = Math.floor(Date.now() / 1000);
const result = await this.#driver().complete(completeArgs);
const result = await this.#complete(res, completeArgs);
const effectiveModel = completeArgs.model || '';
if (stream) {
@@ -583,7 +585,8 @@ export class PuterAIController extends PuterController {
text: extractTextContent(
(
messageResult.message as
Record<string, unknown> | undefined
| Record<string, unknown>
| undefined
)?.content,
),
index: 0,
@@ -706,7 +709,7 @@ export class PuterAIController extends PuterController {
const responseId = generateId('resp');
const createdAt = Math.floor(Date.now() / 1000);
const result = await this.#driver().complete(completeArgs);
const result = await this.#complete(res, completeArgs);
const effectiveModel = completeArgs.model || '';
if (stream) {
@@ -996,7 +999,7 @@ export class PuterAIController extends PuterController {
};
const messageId = `msg_${randomId()}`;
const result = await this.#driver().complete(completeArgs);
const result = await this.#complete(res, completeArgs);
const effectiveModel = completeArgs.model || '';
if (stream) {
@@ -1194,6 +1197,15 @@ export class PuterAIController extends PuterController {
});
return driver;
}
/** A completion that stops, and stops billing, when the caller hangs up. */
#complete(
res: Response,
args: ICompleteArguments,
): Promise<IChatCompleteResult> {
Context.set('abortSignal', abortOnDisconnect(res));
return this.#driver().complete(args);
}
}
// -- Shared helpers --------------------------------------------------
@@ -18,6 +18,12 @@
*/
import { HttpError } from '../../core/http/HttpError.js';
import {
MAX_TOTAL_ATTACHMENT_BYTES,
attachmentMetadata,
attachmentSummary,
readContactSubmission,
} from '../../util/contactAttachments.js';
import { PuterController } from '../types.js';
/**
@@ -94,6 +100,38 @@ const LSMOD_LIMIT = {
key: 'user',
};
/** Longest message the Contact Us form will carry, in characters. */
const CONTACT_MESSAGE_MAX_LENGTH = 100_000;
/** UTF-8 worst case for {@link CONTACT_MESSAGE_MAX_LENGTH} UTF-16 code units. */
const CONTACT_MESSAGE_MAX_BYTES = CONTACT_MESSAGE_MAX_LENGTH * 3;
/** Largest valid multipart body: attachments, message, part-header slack. */
const CONTACT_BODY_MAX_BYTES =
MAX_TOTAL_ATTACHMENT_BYTES + CONTACT_MESSAGE_MAX_BYTES + 64 * 1024;
const isMultipart = (req) =>
/^multipart\/form-data\b/i.test(req.headers?.['content-type'] ?? '');
/**
* Per user, plus a per-IP backstop against one machine cycling fresh accounts.
* The IP limit sits well above what a shared office would send.
*/
const CONTACT_US_LIMITS = [
{
scope: 'contact-us',
limit: 10,
window: 15 * 60_000,
key: 'user',
},
{
scope: 'contact-us-ip',
limit: 40,
window: 24 * 60 * 60_000,
key: 'ip',
},
];
export class SystemController extends PuterController {
constructor(config, clients, stores, services, drivers) {
super(config, clients, stores, services, drivers);
@@ -180,33 +218,78 @@ export class SystemController extends PuterController {
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: {
scope: 'contact-us',
limit: 10,
window: 15 * 60_000,
key: 'user',
},
rateLimit: CONTACT_US_LIMITS,
// Each in-flight request holds its attachments in memory until
// the mail is sent.
concurrent: { limit: 2, scope: 'contact-us', key: 'user' },
},
async (req, res) => {
const { message } = req.body ?? {};
let message;
let attachments = [];
if (isMultipart(req)) {
const declaredLength = Number(
req.headers['content-length'],
);
if (
Number.isFinite(declaredLength) &&
declaredLength > CONTACT_BODY_MAX_BYTES
) {
// Don't read a body already known to be too large.
res.setHeader('Connection', 'close');
throw new HttpError(413, 'Request body is too large', {
legacyCode: 'bad_request',
});
}
const submission = await readContactSubmission(req, {
maxMessageBytes: CONTACT_MESSAGE_MAX_BYTES,
maxBodyBytes: CONTACT_BODY_MAX_BYTES,
});
if (!submission.ok) {
throw new HttpError(
submission.status,
submission.reason,
{ legacyCode: 'bad_request' },
);
}
({ message, attachments } = submission);
} else {
message = req.body?.message;
if (req.body?.attachments !== undefined) {
throw new HttpError(
400,
'`attachments` must be sent as multipart/form-data',
{ legacyCode: 'bad_request' },
);
}
}
if (!message || typeof message !== 'string') {
throw new HttpError(400, '`message` is required', {
legacyCode: 'bad_request',
});
}
if (message.length > 100_000) {
if (message.length > CONTACT_MESSAGE_MAX_LENGTH) {
throw new HttpError(
400,
'`message` is too long (max 100,000 characters)',
`\`message\` is too long (max ${CONTACT_MESSAGE_MAX_LENGTH.toLocaleString('en-US')} characters)`,
{ legacyCode: 'bad_request' },
);
}
// Persist to feedback table for durability
// Persist to feedback table for durability. Attachments are
// recorded by name and size only; the mail carries the bytes.
try {
await this.clients.db.write(
'INSERT INTO `feedback` (`user_id`, `message`) VALUES (?, ?)',
[req.actor.user.id, message],
'INSERT INTO `feedback` (`user_id`, `message`, `attachments`) VALUES (?, ?, ?)',
[
req.actor.user.id,
message,
attachments.length
? JSON.stringify(
attachmentMetadata(attachments),
)
: null,
],
);
} catch (e) {
console.warn('[contactUs] feedback insert failed:', e);
@@ -221,7 +304,16 @@ export class SystemController extends PuterController {
to: supportEmail,
replyTo: req.actor.user.email,
subject: `Contact from ${req.actor.user.username}`,
text: message,
text: attachments.length
? `${message}\n\n${attachmentSummary(attachments)}`
: message,
// Never rendered inline by the mail client.
attachments: attachments.map((a) => ({
filename: a.filename,
content: a.content,
contentType: a.contentType,
contentDisposition: 'attachment',
})),
});
} catch (e) {
console.warn('[contactUs] email send failed:', e);
@@ -18,6 +18,7 @@
*/
import type { Request, RequestHandler, Response } from 'express';
import { Readable } from 'node:stream';
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import type { Actor } from '../../core/actor.js';
@@ -82,11 +83,12 @@ const makeReq = (init: {
body?: unknown;
actor?: Actor;
query?: Record<string, unknown>;
headers?: Record<string, string>;
}): Request => {
return {
body: init.body ?? {},
query: init.query ?? {},
headers: {},
headers: init.headers ?? {},
actor: init.actor,
} as unknown as Request;
};
@@ -430,11 +432,253 @@ describe('SystemController POST /contactUs', () => {
// The row landed in the real `feedback` table for the right user.
const rows = (await server.clients.db.read(
'SELECT `user_id`, `message` FROM `feedback` WHERE `user_id` = ? AND `message` = ?',
'SELECT `user_id`, `message`, `attachments` FROM `feedback` WHERE `user_id` = ? AND `message` = ?',
[userId, message],
)) as Array<{ user_id: number; message: string }>;
)) as Array<{
user_id: number;
message: string;
attachments: string | null;
}>;
expect(rows).toHaveLength(1);
expect(rows[0]?.message).toBe(message);
// No files sent — the column stays null rather than an empty array.
expect(rows[0]?.attachments ?? null).toBeNull();
});
});
describe('SystemController POST /contactUs — attachments', () => {
// A real PNG signature with a filler body; the sniffer only reads the
// first eight bytes, and nothing downstream decodes the image.
const pngBytes = (size = 64): Buffer =>
Buffer.concat([
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
Buffer.alloc(size - 8, 0x61),
]);
type Part =
| { field: string; value: string }
| { field: string; filename: string; data: Buffer };
const BOUNDARY = '----contact-us-test';
const multipart = (parts: Part[]): Buffer =>
Buffer.concat([
...parts.flatMap((part) =>
'value' in part
? [
Buffer.from(
`--${BOUNDARY}\r\nContent-Disposition: form-data; name="${part.field}"\r\n\r\n${part.value}\r\n`,
),
]
: [
Buffer.from(
`--${BOUNDARY}\r\nContent-Disposition: form-data; name="${part.field}"; filename="${part.filename}"\r\nContent-Type: application/octet-stream\r\n\r\n`,
),
part.data,
Buffer.from('\r\n'),
],
),
Buffer.from(`--${BOUNDARY}--\r\n`),
]);
const submitMultipart = async (
parts: Part[],
headers: Record<string, string> = {},
) => {
const { actor, userId } = await makeUser();
const { res, captured } = makeRes();
const body = multipart(parts);
const req = Object.assign(Readable.from([body]), {
headers: {
'content-type': `multipart/form-data; boundary=${BOUNDARY}`,
'content-length': String(body.length),
...headers,
},
actor,
query: {},
}) as unknown as Request;
const call = callRoute('post', '/contactUs', req, res);
return { call, captured, userId };
};
const readAttachments = async (userId: number, message: string) => {
const rows = (await server.clients.db.read(
'SELECT `attachments` FROM `feedback` WHERE `user_id` = ? AND `message` = ?',
[userId, message],
)) as Array<{ attachments: string | null }>;
return rows[0]?.attachments ?? null;
};
it('stores attachment metadata — names, types and sizes, no payloads', async () => {
const message = `attached ${Math.random().toString(36).slice(2)}`;
const { call, captured, userId } = await submitMultipart([
{ field: 'message', value: message },
{
field: 'attachments',
filename: 'repro-step-3.png',
data: pngBytes(128),
},
]);
await call;
expect(captured.body).toEqual({});
const stored = await readAttachments(userId, message);
expect(JSON.parse(stored!)).toEqual([
{ name: 'repro-step-3.png', type: 'image/png', size: 128 },
]);
});
it('emails the files to support with sanitized names and a manifest', async () => {
const sendRaw = vi
.spyOn(server.clients.email, 'sendRaw')
.mockResolvedValue(null);
try {
const message = `attached ${Math.random().toString(36).slice(2)}`;
const { call } = await submitMultipart([
{ field: 'message', value: message },
// Declares .html, but the bytes are a PNG: the stored and
// emailed extension must follow the bytes.
{
field: 'attachments',
filename: 'payload.html',
data: pngBytes(64),
},
]);
await call;
expect(sendRaw).toHaveBeenCalledTimes(1);
const sent = sendRaw.mock.calls[0][0] as {
text: string;
attachments: Array<{
filename: string;
content: Buffer;
contentType: string;
contentDisposition: string;
}>;
};
expect(sent.attachments).toHaveLength(1);
expect(sent.attachments[0].filename).toBe('payload.png');
expect(sent.attachments[0].contentType).toBe('image/png');
expect(sent.attachments[0].contentDisposition).toBe('attachment');
expect(sent.attachments[0].content.equals(pngBytes(64))).toBe(true);
// The body records what should have arrived, in case a gateway
// strips the files on the way.
expect(sent.text).toContain(message);
expect(sent.text).toContain('payload.png');
} finally {
sendRaw.mockRestore();
}
});
it('accepts a multipart submission with no files', async () => {
const sendRaw = vi
.spyOn(server.clients.email, 'sendRaw')
.mockResolvedValue(null);
try {
const message = `plain ${Math.random().toString(36).slice(2)}`;
const { call, captured } = await submitMultipart([
{ field: 'message', value: message },
]);
await call;
expect(captured.body).toEqual({});
const sent = sendRaw.mock.calls[0][0] as {
text: string;
attachments: unknown[];
};
expect(sent.attachments).toEqual([]);
expect(sent.text).toBe(message);
} finally {
sendRaw.mockRestore();
}
});
it('rejects base64 attachments in a JSON body', async () => {
const { actor } = await makeUser();
const { res } = makeRes();
await expect(
callRoute(
'post',
'/contactUs',
makeReq({
body: {
message: 'hi',
attachments: [{ data: pngBytes().toString('base64') }],
},
actor,
}),
res,
),
).rejects.toMatchObject({ statusCode: 400 });
});
it.each([
[
'too many files',
[
{ field: 'message', value: 'hi' },
...Array.from({ length: 6 }, (_, i) => ({
field: 'attachments',
filename: `${i}.png`,
data: pngBytes(),
})),
],
],
[
'an unsupported type',
[
{ field: 'message', value: 'hi' },
{
field: 'attachments',
filename: 'a.pdf',
data: Buffer.from('%PDF-1.7 x'),
},
],
],
[
'a script-capable SVG',
[
{ field: 'message', value: 'hi' },
{
field: 'attachments',
filename: 'a.svg',
data: Buffer.from('<svg><script/></svg>'),
},
],
],
[
'a missing message',
[{ field: 'attachments', filename: 'a.png', data: pngBytes() }],
],
] as [string, Part[]][])('rejects %s with 400', async (_label, parts) => {
const { call, captured } = await submitMultipart(parts);
await expect(call).rejects.toMatchObject({ statusCode: 400 });
expect(captured.body).toBeUndefined();
});
it('refuses a body whose declared length is over budget before reading it', async () => {
const { call, captured } = await submitMultipart(
[{ field: 'message', value: 'hi' }],
{ 'content-length': String(64 * 1024 * 1024) },
);
await expect(call).rejects.toMatchObject({ statusCode: 413 });
expect(captured.headers.Connection).toBe('close');
});
it('rejects an upload that breaks a limit mid-stream', async () => {
const { call, captured } = await submitMultipart(
[
{ field: 'message', value: 'hi' },
{
field: 'attachments',
filename: 'big.png',
data: pngBytes(10 * 1024 * 1024 + 1),
},
],
// Chunked uploads carry no length to check up front.
{ 'content-length': '' },
);
await expect(call).rejects.toMatchObject({ statusCode: 413 });
// The rest of the body is drained, so the connection can stay open.
expect(captured.headers.Connection).toBeUndefined();
});
});
@@ -0,0 +1,136 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import http from 'node:http';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
/**
* Contact Us attachments over real HTTP. Multipart bodies must skip the global
* JSON parser so the route's gates run before any of the body is read; only a
* listening server shows that.
*/
describe('POST /contactUs over HTTP', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const png = (size: number): Buffer =>
Buffer.concat([
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
Buffer.alloc(size - 8, 0x61),
]);
const BOUNDARY = 'contact-us-http-test';
const multipartHead = Buffer.from(
`--${BOUNDARY}\r\nContent-Disposition: form-data; name="message"\r\n\r\nhi\r\n` +
`--${BOUNDARY}\r\nContent-Disposition: form-data; name="attachments"; filename="a.png"\r\n` +
'Content-Type: image/png\r\n\r\n',
);
/** Send raw bytes and resolve with the response, however early it comes. */
const rawPost = (
headers: Record<string, string>,
write: (req: http.ClientRequest) => void,
) =>
new Promise<{ status: number; connection?: string }>(
(resolve, reject) => {
const url = new URL('/contactUs', env.apiOrigin);
const req = http.request(
url,
{
method: 'POST',
headers: {
Authorization: `Bearer ${env.users.user.token}`,
'Content-Type': `multipart/form-data; boundary=${BOUNDARY}`,
...headers,
},
},
(res) => {
res.resume();
resolve({
status: res.statusCode ?? 0,
connection: res.headers.connection,
});
},
);
req.on('error', reject);
write(req);
},
);
it('accepts the FormData the Contact Us window sends', async () => {
const message = `http ${Math.random().toString(36).slice(2)}`;
const body = new FormData();
body.append('message', message);
body.append(
'attachments',
new Blob([png(4096)], { type: 'image/png' }),
'shot.png',
);
const res = await fetch(new URL('/contactUs', env.apiOrigin), {
method: 'POST',
headers: { Authorization: `Bearer ${env.users.user.token}` },
body,
});
expect(res.status).toBe(200);
const rows = (await env.server.clients.db.read(
'SELECT `attachments` FROM `feedback` WHERE `message` = ?',
[message],
)) as Array<{ attachments: string | null }>;
expect(JSON.parse(rows[0]!.attachments!)).toEqual([
{ name: 'shot.png', type: 'image/png', size: 4096 },
]);
});
it('refuses an unauthenticated upload', async () => {
const body = new FormData();
body.append('message', 'hi');
const res = await fetch(new URL('/contactUs', env.apiOrigin), {
method: 'POST',
body,
});
expect(res.status).toBe(401);
});
it('answers an oversized Content-Length before the body is sent', async () => {
const res = await rawPost(
{ 'Content-Length': String(64 * 1024 * 1024) },
(req) => req.flushHeaders(),
);
expect(res).toEqual({ status: 413, connection: 'close' });
});
it('answers a chunked upload that breaks the per-file cap with 413', async () => {
const file = png(12 * 1024 * 1024);
const res = await rawPost({ 'Transfer-Encoding': 'chunked' }, (req) => {
req.write(multipartHead);
req.end(file);
});
expect(res.status).toBe(413);
});
});
@@ -732,12 +732,17 @@ describe('team endpoints over HTTP', () => {
return { username, password, userId: row.id, token };
};
/** Cookie-credentialed on the GUI origin, as the user-protected gate insists. */
/**
* Cookie-credentialed on the GUI origin, as the user-protected gate insists.
* Sent with the seat's own user agent: the session touch otherwise rewrites
* it to Node's once a second boundary passes.
*/
const changePassword = (token: string, password: string, next: string) =>
fetch(new URL('/user-protected/change-password', env.origin), {
method: 'POST',
headers: {
'content-type': 'application/json',
'user-agent': 'puter-test-seat',
authorization: `Bearer ${token}`,
cookie: `puter_auth_token=${token}`,
},
@@ -67,6 +67,7 @@ const toClientTeam = (team: TeamRow, isOwner: boolean) => ({
is_owner: isOwner,
created_at: team.created_at,
directory_enabled: Number(team.directory_enabled) === 1,
require_2fa: Number(team.require_2fa) === 1,
});
@Controller('/teams')
@@ -153,6 +154,7 @@ export class TeamController extends PuterController {
name?: string;
handle?: string | null;
directoryEnabled?: boolean;
require2fa?: boolean;
} = {};
if (body.name !== undefined)
changes.name = this.#requireString(body.name, 'name');
@@ -160,6 +162,8 @@ export class TeamController extends PuterController {
changes.handle = body.handle === null ? null : String(body.handle);
if (body.directory_enabled !== undefined)
changes.directoryEnabled = body.directory_enabled === true;
if (body.require_2fa !== undefined)
changes.require2fa = body.require_2fa === true;
// Through the service: turning the directory on writes an audit row.
const team = await this.services.team.updateTeam(
@@ -332,6 +336,24 @@ export class TeamController extends PuterController {
res.json({ temporary_password: temporaryPassword });
}
// Same budget as a password reset: both are a step toward the account.
@Post('/:uid/members/:username/2fa-reset', {
subdomain: 'api',
requireUserActor: true,
requireVerified: true,
rateLimit: TEAM_RESET_LIMIT,
})
async resetMemberTwoFactor(req: Request, res: Response): Promise<void> {
const userId = this.#requireUserId(req);
const uid = this.#param(req, 'uid');
// Authority first, or resolving `:username` is an existence oracle.
await this.services.team.requireOwner(uid, userId);
const target = await this.#requireTargetUserId(req);
await this.services.team.resetMemberTwoFactor(uid, userId, target);
res.json({ success: true });
}
@Post('/:uid/members/:username/disable', {
subdomain: 'api',
requireUserActor: true,
@@ -2,6 +2,7 @@
// but rather it performs some common sense checks to ensure that WebDAV support isn't irrevocably broken in puter
import type { Request, RequestHandler, Response } from 'express';
import { Readable, Writable } from 'node:stream';
import { GetObjectCommand } from '@aws-sdk/client-s3';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import { hash as bcryptHash } from 'bcrypt';
@@ -9,7 +10,9 @@ import { PuterRouter } from '../../core/http/PuterRouter.js';
import type { RouteDescriptor } from '../../core/http/types.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { makeActor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { FULL_API_ACCESS } from '../../services/permission/consts.js';
import { generateDefaultFsentries } from '../../util/userProvisioning.js';
import type { WebDAVController } from './WebDAVController.js';
@@ -1197,7 +1200,7 @@ describe('WebDAVController verbs', () => {
* A response double that is also a Writable, so handlers that end with
* `body.pipe(res)` (GET) work without a socket.
*/
const makeStreamRes = () => {
const makeStreamRes = ({ abortOnFirstChunk = false } = {}) => {
const chunks: Buffer[] = [];
const captured = {
statusCode: 200,
@@ -1209,6 +1212,9 @@ describe('WebDAVController verbs', () => {
const sink = new Writable({
write(chunk, _encoding, callback) {
chunks.push(Buffer.from(chunk as Buffer));
// A client that hangs up mid-download surfaces as the
// response being destroyed, not as a write error.
if (abortOnFirstChunk) sink.destroy();
callback();
},
});
@@ -1334,6 +1340,87 @@ describe('WebDAVController verbs', () => {
expect(captured.text()).toBe('2345');
});
/**
* Swap the object store's answer for one GET so the body stream is
* under the test's control. Intercepts at the S3 SDK client, the real
* boundary; every other command still hits the in-memory S3.
*/
const serveBody = (body: Readable) => {
const client = server.clients.s3.get(
server.stores.s3Object.resolveRegion(null),
);
const original = client.send.bind(client);
return vi.spyOn(client, 'send').mockImplementation(((
command: unknown,
) =>
command instanceof GetObjectCommand
? Promise.resolve({
Body: body,
ContentLength: 7,
ContentType: 'text/plain',
})
: original(command as never)) as never);
};
it('tears down the upstream stream when the client disconnects mid-download', async () => {
const { actor, username } = await makeUser();
const path = `/${username}/Documents/abort-me.txt`;
await putFile(actor, path, 'streamed');
// Never ends on its own: only a teardown can finish it.
const body = new Readable({ read() {} });
body.push('partial');
const send = serveBody(body);
const { res } = makeStreamRes({ abortOnFirstChunk: true });
try {
await dispatchMiddleware(
makeReq({ method: 'GET', path, actor }),
res,
noop,
);
} finally {
send.mockRestore();
}
expect((res as unknown as Writable).destroyed).toBe(true);
expect(body.destroyed).toBe(true);
});
it('survives the upstream stream failing mid-download', async () => {
const { actor, username } = await makeUser();
const path = `/${username}/Documents/reset-me.txt`;
await putFile(actor, path, 'streamed');
// First read hands over a chunk, the next one fails the way a
// dropped storage connection does. Without an error listener on
// the body that event would take the process down.
let reads = 0;
const body = new Readable({
read() {
reads += 1;
if (reads === 1) this.push('partial');
else this.destroy(new Error('upstream reset'));
},
});
const send = serveBody(body);
const { res, captured } = makeStreamRes();
try {
await expect(
dispatchMiddleware(
makeReq({ method: 'GET', path, actor }),
res,
noop,
),
).resolves.toBeUndefined();
} finally {
send.mockRestore();
}
expect(captured.statusCode).toBe(200);
expect((res as unknown as Writable).destroyed).toBe(true);
expect(body.destroyed).toBe(true);
});
it('refuses to GET a directory', async () => {
const { actor, username } = await makeUser();
const captured = await dispatch({
@@ -1461,6 +1548,98 @@ describe('WebDAVController verbs', () => {
);
});
describe('root collection per credential', () => {
// Own address, so the per-IP auth failure bucket the rest of the
// file draws on can't refuse these.
const propfindRoot = (authorization: string) =>
dispatch({
method: 'PROPFIND',
path: '/',
headers: { authorization },
ip: '203.0.113.53',
});
const mintToken = async (userId: number, permission: string) => {
const user = (await server.stores.user.getById(userId))!;
return server.services.auth.createAccessToken(
makeActor({ user }),
[[permission]],
);
};
it('lists the home for a password login and a full-access token', async () => {
const username = `webdav-root-${Math.random()
.toString(36)
.slice(2, 10)}`;
const created = await server.stores.user.create({
username,
uuid: uuidv4(),
password: await bcryptHash('correct-horse', 4),
email: `${username}@test.local`,
free_storage: 100 * 1024 * 1024,
requires_email_confirmation: false,
});
await generateDefaultFsentries(
server.clients.db,
server.stores.user,
created,
);
const fullAccess = await mintToken(created.id, FULL_API_ACCESS);
for (const authorization of [
basicAuth(username, 'correct-horse'),
basicAuth('-token', fullAccess),
]) {
const captured = await propfindRoot(authorization);
expect(captured.statusCode).toBe(207);
expect(captured.body as string).toContain(
`<D:href>/${username}/</D:href>`,
);
}
});
it('leaves the home out for a token scoped below it', async () => {
const { userId, username } = await makeUser();
const home = (await server.stores.fsEntry.getEntryByPath(
`/${username}`,
))!;
const documents = (await server.stores.fsEntry.getEntryByPath(
`/${username}/Documents`,
))!;
const scoped = await mintToken(
userId,
`fs:${documents.uuid}:read`,
);
const captured = await propfindRoot(
basicAuth('-token', scoped),
);
expect(captured.statusCode).toBe(207);
const xml = captured.body as string;
expect(xml).toContain('<D:href>/</D:href>');
expect(xml).not.toContain(`/${username}/`);
expect(xml).not.toContain(home.uuid);
});
it('lists the home for a token that can read it', async () => {
const { userId, username } = await makeUser();
const home = (await server.stores.fsEntry.getEntryByPath(
`/${username}`,
))!;
const scoped = await mintToken(userId, `fs:${home.uuid}:read`);
const captured = await propfindRoot(
basicAuth('-token', scoped),
);
expect(captured.statusCode).toBe(207);
expect(captured.body as string).toContain(
`<D:href>/${username}/</D:href>`,
);
});
});
it('describes a single file when asked for one', async () => {
const { actor, username } = await makeUser();
const path = `/${username}/Documents/single.md`;
@@ -20,11 +20,13 @@
import { compare as bcryptCompare } from 'bcrypt';
import type { Request, Response } from 'express';
import { posix as pathPosix } from 'node:path';
import { pipeline } from 'node:stream/promises';
import { EventMap } from '../../clients/event/types.js';
import { makeActor, type Actor } from '../../core/actor.js';
import { HttpError } from '../../core/http/HttpError.js';
import {
assertNotSuspended,
assertTeam2fa,
assertVerifiedAccount,
} from '../../core/http/middleware/gates.js';
import {
@@ -46,6 +48,7 @@ import {
import type { RouteOptions } from '../../core/http/types.js';
import { verify as verifyOtp } from '../../services/auth/OTPUtil.js';
import { expandTildePath } from '../../services/fs/resolveNode.js';
import { rootShowsHome } from '../../services/fs/rootListing.js';
import {
maskEntryPath,
parseMaskedSharePath,
@@ -283,6 +286,7 @@ export class WebDAVController extends PuterController {
// filesystem over the `dav` subdomain, bypassing the gates.
assertNotSuspended(actor.user);
assertVerifiedAccount(actor.user);
await assertTeam2fa(actor.user, this.stores.team);
// The actor was absent when the auth probe ran, so anything that reads
// it off the request or the context — shared-path masking, egress
@@ -551,7 +555,12 @@ export class WebDAVController extends PuterController {
'Content-Length': String(result.contentLength ?? 0),
});
}
result.body.pipe(res);
try {
await pipeline(result.body, res);
} catch {
// Client disconnect or upstream stream error: pipeline already
// tore down both ends, so nothing is left dangling to crash on.
}
}
// -- PROPFIND ----------------------------------------------------
@@ -616,7 +625,8 @@ export class WebDAVController extends PuterController {
/**
* The root collection: the caller's own home directory, plus one collection
* per person who has shared something with them.
* per person who has shared something with them. A scoped access token sees
* the home only when it can read it.
*
* A share is addressed `/<owner>/<uuid>/<name>`, where the uuid stands in
* for the folder the owner keeps it in. Only the `<owner>` segment is a
@@ -631,7 +641,16 @@ export class WebDAVController extends PuterController {
const home = await this.stores.fsEntry.getEntryByPath(
`/${actor.user!.username}`,
);
if (home) {
if (
home &&
(await rootShowsHome(
actor,
home,
'read',
this.services.acl,
this.stores.permission,
))
) {
responses.push(
propfindEntry(maskEntryPath(home), home, home.isDir),
);
@@ -1166,7 +1185,9 @@ export class WebDAVController extends PuterController {
void Promise.resolve()
.then(async () => {
const response = {
...(await toLegacyEntry(this.clients.event, entry)),
...(await toLegacyEntry(this.clients.event, entry, {
forOwner: true,
})),
...extra,
from_new_service: true,
};
+9
View File
@@ -75,6 +75,15 @@ describe('makeActor / effectiveApp', () => {
expect(userRelatedActor(app).effectiveApp).toBeNull();
});
it('keeps the handler depth when narrowing to the underlying user', () => {
const app = makeActor({ user, app: { uid: 'app-1' } });
app.handlerDepth = 3;
expect(userRelatedActor(app).handlerDepth).toBe(3);
expect(
userRelatedActor(makeActor({ user, app: { uid: 'app-1' } })),
).not.toHaveProperty('handlerDepth');
});
it('resolves the system actor', () => {
expect(SYSTEM_ACTOR.effectiveApp).toBeNull();
});
+20 -1
View File
@@ -64,6 +64,17 @@ export interface Actor {
* carrying one. `kind` mirrors the session row (`web`, `app`, `worker`).
*/
session?: { uid: string; kind?: string | null } | null;
/**
* Events handler runs behind this actor's writes; set only from an events
* handler token or a token one minted. Provenance, not authority, so
* derived actors keep it.
*/
handlerDepth?: number;
/**
* When the token behind `handlerDepth` expires, in unix seconds. An app
* token minted from this actor expires no later.
*/
handlerExpiresAt?: number;
}
/** UUID of the baked-in system user (see 0025 seed migration). */
@@ -155,5 +166,13 @@ export const actorUid = (actor: Actor): string => {
*/
export const userRelatedActor = (actor: Actor): Actor => {
if (!actor.app && !actor.accessToken) return actor;
return { user: actor.user, effectiveApp: null };
return {
user: actor.user,
effectiveApp: null,
...(actor.handlerDepth ? { handlerDepth: actor.handlerDepth } : {}),
};
};
/** How many events handler runs led to a write by this actor. */
export const handlerDepthOf = (actor: Actor | undefined | null): number =>
actor?.handlerDepth ?? 0;
@@ -17,17 +17,17 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { describe, expect, it } from 'vitest';
import { DeepSeekPricing } from './pricing.js';
import type { Response } from 'express';
describe.skipIf(!process.env.PUTER_TEST_AI_DEEPSEEK_PRICING)(
'DeepSeek pricing page (integration)',
() => {
it('fetches and parses current official rates', async () => {
const pricing = new DeepSeekPricing();
await pricing.refresh();
expect(pricing.costs('deepseek-flash').prompt).toBeGreaterThan(0);
expect(pricing.costs('deepseek-v4-pro').cached).toBeGreaterThan(0);
}, 15_000);
},
);
/**
* A signal that fires when the caller hangs up before its response is done, so
* long-running work can stop (and stop metering). `close` after `finish` is the
* normal end of a response, not an abort.
*/
export const abortOnDisconnect = (res: Response): AbortSignal => {
const abort = new AbortController();
res.once('close', () => {
if (!res.writableFinished) abort.abort();
});
return abort.signal;
};
@@ -33,6 +33,7 @@ import {
requireAuthGate,
requireCardVerifiedGate,
requirePhoneVerifiedGate,
requireTeam2fa,
requireVerifiedAccount,
requireNonAccessTokenGate,
requireUserActorGate,
@@ -762,6 +763,46 @@ describe('requireVerifiedAccount', () => {
expectHttpError(got, 403, 'card_verification_required');
});
it('returns 403 two_factor_required for a seat whose team requires 2FA', async () => {
const teams = { getOrgSeat: async () => ({ require_2fa: 1 }) };
const got = await runGateAsync(requireTeam2fa(teams), {
actor: { user: { uuid: 'u-1', id: 7, otp_enabled: 0 } },
});
expectHttpError(got, 403, 'two_factor_required');
});
it('lets the same seat through once it holds 2FA', async () => {
const teams = { getOrgSeat: async () => ({ require_2fa: 1 }) };
const got = await runGateAsync(requireTeam2fa(teams), {
actor: { user: { uuid: 'u-1', id: 7, otp_enabled: 1 } },
});
expect(got).toBeUndefined();
});
it('does not gate an account that is not a seat', async () => {
// No seat row: an ordinary account, whatever any team requires.
const teams = { getOrgSeat: async () => null };
const got = await runGateAsync(requireTeam2fa(teams), {
actor: { user: { uuid: 'u-1', id: 7, otp_enabled: 0 } },
});
expect(got).toBeUndefined();
});
it('does not gate a seat whose team does not require 2FA', async () => {
const teams = { getOrgSeat: async () => ({ require_2fa: 0 }) };
const got = await runGateAsync(requireTeam2fa(teams), {
actor: { user: { uuid: 'u-1', id: 7, otp_enabled: 0 } },
});
expect(got).toBeUndefined();
});
it('stays inert where Teams is not deployed', async () => {
const got = await runGateAsync(requireTeam2fa(undefined), {
actor: { user: { uuid: 'u-1', id: 7, otp_enabled: 0 } },
});
expect(got).toBeUndefined();
});
it('returns 403 password_change_required while the account owes a password', () => {
const got = runGate(requireVerifiedAccount(), {
actor: {
+36
View File
@@ -258,6 +258,42 @@ export const requireVerifiedGate = (strictFlag: boolean): RequestHandler => {
};
};
/** Just the seat read the 2FA gate needs, so gates stay store-agnostic. */
export interface Team2faLookup {
getOrgSeat(
userId: number,
): Promise<{ require_2fa?: number | null } | null | undefined>;
}
/**
* Reject a team-provisioned account whose team requires 2FA until it has some.
* Derived, not stamped: a flag left behind by a member leaving would lock them
* out of their own account with no team left to clear it.
*/
export const assertTeam2fa = async (
user: AccountGateUser | undefined,
teams: Team2faLookup | undefined,
): Promise<void> => {
if (!user || user.otp_enabled) return;
if (typeof user.id !== 'number' || !teams?.getOrgSeat) return;
const seat = await teams.getOrgSeat(user.id);
if (!seat || Number(seat.require_2fa) !== 1) return;
throw new HttpError(
403,
'Your team requires two-factor authentication. Set it up to continue.',
{ legacyCode: 'two_factor_required' as never },
);
};
/** {@link assertTeam2fa} as route middleware. */
export const requireTeam2fa = (
teams: Team2faLookup | undefined,
): RequestHandler => {
return (req, _res, next) => {
assertTeam2fa(req.actor?.user, teams).then(() => next(), next);
};
};
/**
* Reject accounts still pending a signup-time verification (email, phone, card,
* password change). Default-on for authenticated routes; the flows that clear
@@ -32,8 +32,10 @@ import {
buildPublicHostRedirect,
getBootstrapToken,
hostMatchesPrivateDomain,
isTopLevelNavigation,
normalizeHost,
normalizeHostRaw,
pathWithoutQueryParam,
renderLoginBootstrapHtml,
resolveOwnedAppForHostedSite,
resolvePrivateIdentity,
@@ -379,6 +381,61 @@ describe('buildPrivateHostRedirect', () => {
});
});
describe('isTopLevelNavigation', () => {
const reqOf = (method: string, dest?: string): Request =>
({
method,
headers: dest ? { 'sec-fetch-dest': dest } : {},
}) as unknown as Request;
it('is true only for a GET of a top-level page', () => {
expect(isTopLevelNavigation(reqOf('GET', 'document'))).toBe(true);
expect(isTopLevelNavigation(reqOf('GET', 'iframe'))).toBe(false);
expect(isTopLevelNavigation(reqOf('GET', 'script'))).toBe(false);
expect(isTopLevelNavigation(reqOf('POST', 'document'))).toBe(false);
expect(isTopLevelNavigation(reqOf('GET'))).toBe(false);
});
});
describe('pathWithoutQueryParam', () => {
const T = 'puter.auth.token';
it('removes the parameter and keeps the rest exactly as sent', () => {
expect(
pathWithoutQueryParam('/p/a.html?x=1&puter.auth.token=abc&y=a%20b+c', T),
).toBe('/p/a.html?x=1&y=a%20b+c');
});
it('drops the query when nothing else is left', () => {
expect(pathWithoutQueryParam('/?puter.auth.token=abc', T)).toBe('/');
});
it('removes every occurrence, including a percent-encoded key', () => {
expect(
pathWithoutQueryParam('/?puter.auth.token=a&k=v&puter%2Eauth%2Etoken=b', T),
).toBe('/?k=v');
});
it('returns null when the parameter is absent', () => {
expect(pathWithoutQueryParam('/?auth_token=abc', T)).toBeNull();
expect(pathWithoutQueryParam('/', T)).toBeNull();
expect(pathWithoutQueryParam(undefined, T)).toBeNull();
});
it('never produces a redirect off the current host', () => {
expect(pathWithoutQueryParam('//evil.example/?puter.auth.token=a', T)).toBe(
'/evil.example/',
);
expect(pathWithoutQueryParam('/\\evil.example/?puter.auth.token=a', T)).toBe(
'/evil.example/',
);
// URL parsing drops tabs, which would turn this into `//evil.example`.
expect(
pathWithoutQueryParam('/\t/evil.example/?puter.auth.token=a', T),
).toBeNull();
});
});
describe('buildPublicHostRedirect', () => {
// Mirror of buildPrivateHostRedirect — swaps the private hosting
// domain for the public one. Used when a non-private app (or no app
@@ -596,6 +596,43 @@ function normalizeRedirectPath(originalUrl: string | undefined): string {
return '/' + (originalUrl || '/').replace(/^[/\\]+/, '');
}
/** A GET for a top-level page, as opposed to a frame, subresource or fetch. */
export function isTopLevelNavigation(req: Request): boolean {
return (
req.method === 'GET' && req.headers?.['sec-fetch-dest'] === 'document'
);
}
/**
* Same-host path of this request with every `name` query parameter removed, or
* null when it has none. The parameters that stay keep their exact encoding.
*/
export function pathWithoutQueryParam(
originalUrl: string | undefined,
name: string,
): string | null {
const base = 'http://gate.invalid';
let url: URL;
try {
url = new URL(normalizeRedirectPath(originalUrl), base);
} catch {
return null;
}
if (url.origin !== base) return null;
const isTarget = (part: string) => {
const rawKey = part.split('=', 1)[0].replace(/\+/g, ' ');
try {
return decodeURIComponent(rawKey) === name;
} catch {
return rawKey === name;
}
};
const parts = url.search.slice(1).split('&');
if (!parts.some(isTarget)) return null;
const kept = parts.filter((part) => part && !isTarget(part));
return url.pathname + (kept.length ? `?${kept.join('&')}` : '');
}
/** Build the URL to redirect a private-app request to its private host. */
export function buildPrivateHostRedirect(
req: Request,
@@ -46,10 +46,11 @@ interface CapturedRes {
contentType?: string;
redirected?: { status?: number; url: string };
headers: Record<string, string>;
cookies: Record<string, string>;
}
const makeRes = () => {
const out: CapturedRes = { headers: {} };
const out: CapturedRes = { headers: {}, cookies: {} };
// The file-serving branch ends with `download.body.pipe(res)`, so
// `res` has to satisfy the WritableStream contract Node's `pipe()`
// expects — write/end/on/emit/etc. Use a real Writable so Node's
@@ -73,7 +74,7 @@ const makeRes = () => {
type: (ct: string) => Response;
send: (payload: unknown) => Response;
redirect: (...args: unknown[]) => Response;
cookie: () => Response;
cookie: (name: string, value: string) => Response;
set: (name: string, value: string) => Response;
setHeader: (name: string, value: string) => Response;
};
@@ -100,7 +101,10 @@ const makeRes = () => {
}
return res;
};
res.cookie = () => res;
res.cookie = (name: string, value: string) => {
out.cookies[name] = value;
return res;
};
res.set = (name: string, value: string) => {
out.headers[name] = value;
return res;
@@ -114,13 +118,16 @@ const makeRes = () => {
const makeReq = (init: {
hostname: string;
method?: string;
path?: string;
originalUrl?: string;
protocol?: string;
headers?: Record<string, string>;
cookies?: Record<string, string>;
query?: Record<string, string>;
}): Request =>
({
method: init.method ?? 'GET',
hostname: init.hostname,
path: init.path ?? '/',
// Redirect helpers use originalUrl (preserves query string).
@@ -129,7 +136,7 @@ const makeReq = (init: {
protocol: init.protocol ?? 'http',
headers: init.headers ?? {},
cookies: init.cookies ?? {},
query: {},
query: init.query ?? {},
// The file-serve branch wires `req.on('close', ...)` to destroy
// the stream on client disconnect — a no-op event surface is
// enough for the offline tests.
@@ -535,6 +542,34 @@ describe('createPuterSiteMiddleware — file serving', () => {
expect(piped!.equals(body)).toBe(true);
});
// Metering resolves the billed account's plan from this actor, and some
// plans are keyed on the account's email.
it('bills an anonymous visit to the owner, email included', async () => {
const owner = await makeUserWithHome();
const homePath = `/${owner.username}`;
const homeEntry = await server.stores.fsEntry.getEntryByPath(homePath);
const sub = `bill-${Math.random().toString(36).slice(2, 8)}`;
await server.stores.subdomain.create({
userId: owner.id,
subdomain: sub,
rootDirId: homeEntry!.id,
});
await writeFile(owner.id, `${homePath}/index.html`, Buffer.from('hi'));
const req = makeReq({
hostname: `${sub}.site.puter.localhost`,
path: '/index.html',
});
const { res } = makeRes();
await buildMiddleware()(req, res, vi.fn());
expect(req.egressActor?.user).toMatchObject({
uuid: owner.uuid,
id: owner.id,
email: owner.email,
});
});
it('emits site.htmlServed with the original URL target including query string', async () => {
const owner = await makeUserWithHome();
const homePath = `/${owner.username}`;
@@ -1657,3 +1692,131 @@ describe('createPuterSiteMiddleware — delegated site roots', () => {
}
});
});
// ── Private app sign-in token in a top-level URL ────────────────────
describe('createPuterSiteMiddleware — private app sign-in token', () => {
let owner: Awaited<ReturnType<typeof makeUserWithHome>>;
let visitor: Awaited<ReturnType<typeof makeUser>>;
let sub: string;
let host: string;
let appUid: string;
let token: string;
const allow = (
_key: string,
data: { appUid: string; result: { allowed: boolean } },
) => {
if (data.appUid === appUid) data.result.allowed = true;
};
beforeAll(async () => {
owner = await makeUserWithHome();
visitor = await makeUser();
const homePath = `/${owner.username}`;
const homeEntry = await server.stores.fsEntry.getEntryByPath(homePath);
sub = `priv-${Math.random().toString(36).slice(2, 8)}`;
host = `${sub}.app.puter.localhost`;
await server.stores.subdomain.create({
userId: owner.id,
subdomain: sub,
rootDirId: homeEntry!.id,
});
await writeFile(
owner.id,
`${homePath}/index.html`,
Buffer.from('<html>app</html>'),
'text/html',
);
appUid = `app-${uuidv4()}`;
await server.clients.db.write(
'INSERT INTO `apps` (`uid`, `name`, `title`, `index_url`, `owner_user_id`, `is_private`) VALUES (?, ?, ?, ?, ?, ?)',
[appUid, sub, sub, `http://${host}/`, owner.id, 1],
);
({ token } = await server.services.auth.createSessionToken(
visitor as never,
));
server.clients.event.on('app.privateAccess.check', allow as never);
});
afterAll(() => {
server.clients.event.off('app.privateAccess.check', allow as never);
});
const visit = async (init: {
dest?: string;
method?: string;
cookies?: Record<string, string>;
}) => {
const mw = buildMiddleware();
const { res, out } = makeRes();
await mw(
makeReq({
hostname: host,
method: init.method,
path: '/',
originalUrl: `/?a=1&puter.auth.token=${token}&b=x%20y`,
query: { a: '1', 'puter.auth.token': token, b: 'x y' },
headers: init.dest ? { 'sec-fetch-dest': init.dest } : {},
cookies: init.cookies,
}),
res,
vi.fn(),
);
await new Promise<void>((resolve) => setImmediate(resolve));
return out;
};
const cookieName = () => server.services.auth.getPrivateAssetCookieNameV2();
it('sets the cookie and redirects a top-level page to the same URL without the token', async () => {
const out = await visit({ dest: 'document' });
expect(out.redirected).toEqual({ status: 302, url: '/?a=1&b=x%20y' });
expect(out.cookies[cookieName()]).toBeTruthy();
expect(out.headers['Cache-Control']).toBe('no-store');
});
it('serves a framed launch with the token left in place for the SDK', async () => {
const out = await visit({ dest: 'iframe' });
expect(out.redirected).toBeUndefined();
expect(out.statusCode).toBe(200);
});
it('serves as before when the browser does not say what the request is for', async () => {
const out = await visit({});
expect(out.redirected).toBeUndefined();
expect(out.statusCode).toBe(200);
});
it('does not redirect a non-GET navigation', async () => {
const out = await visit({ dest: 'document', method: 'POST' });
expect(out.redirected).toBeUndefined();
});
it('still strips the token when the visitor already holds the cookie', async () => {
const cookie = await server.services.auth.createPrivateAssetToken({
appUid,
userUid: visitor.uuid,
subdomain: sub,
privateHost: host,
});
const out = await visit({
dest: 'document',
cookies: { [cookieName()]: cookie },
});
expect(out.redirected).toEqual({ status: 302, url: '/?a=1&b=x%20y' });
});
it('serves without redirecting when the cookie could not be minted', async () => {
// Without the cookie, the clean URL would bounce back to sign-in.
const spy = vi
.spyOn(server.services.auth, 'createPrivateAssetToken')
.mockRejectedValueOnce(new Error('mint failed'));
try {
const out = await visit({ dest: 'document' });
expect(out.redirected).toBeUndefined();
expect(out.statusCode).toBe(200);
} finally {
spy.mockRestore();
}
});
});
+21 -1
View File
@@ -32,6 +32,8 @@ import {
buildPrivateHostRedirect,
buildPublicHostRedirect,
hostMatchesPrivateDomain,
isTopLevelNavigation,
pathWithoutQueryParam,
renderLoginBootstrapHtml,
resolveOwnedAppForHostedSite,
resolvePrivateIdentity,
@@ -146,6 +148,7 @@ interface UserRow {
id: number;
uuid: string;
username: string;
email?: string | null;
suspended?: number | null;
}
@@ -369,7 +372,8 @@ export const createPuterSiteMiddleware = (
// Mint the sticky cookie only when we don't already have a
// valid one — keeps Set-Cookie off of the hot path for repeat
// visitors but still refreshes after rotation/expiry.
if (!identity.hasValidPrivateCookie) {
let hasStickyCookie = !!identity.hasValidPrivateCookie;
if (!hasStickyCookie) {
try {
const token =
await layers.services.auth.createPrivateAssetToken({
@@ -386,6 +390,7 @@ export const createPuterSiteMiddleware = (
requestHostname: host,
}),
);
hasStickyCookie = true;
} catch (e) {
console.warn(
'[puter-site] failed to mint private asset cookie',
@@ -397,6 +402,20 @@ export const createPuterSiteMiddleware = (
// Referrer-policy hardening — don't leak private-host URLs to
// third-party resources loaded from the app.
res.setHeader('Referrer-Policy', 'no-referrer');
// A tab opened straight on the app carries the sign-in token in
// its URL. Once the cookie stands in for it, reload without it so
// it doesn't stay in the address bar. Framed launches keep it:
// the SDK inside reads it from there.
const cleanPath =
hasStickyCookie && isTopLevelNavigation(req)
? pathWithoutQueryParam(req.originalUrl, 'puter.auth.token')
: null;
if (cleanPath) {
res.set('Cache-Control', 'no-store');
res.redirect(302, cleanPath);
return;
}
} else if (privateHostingDomains.has(matched)) {
// Non-private content landed on the private hosting domain —
// mirror of the private redirect above. Covers two cases:
@@ -745,6 +764,7 @@ export const createPuterSiteMiddleware = (
uuid: owner.uuid,
id: owner.id,
username: owner.username,
email: owner.email ?? null,
suspended: !!owner.suspended,
},
};
+25 -14
View File
@@ -21,10 +21,7 @@
import crypto from 'node:crypto';
import { withSpan } from '../../../util/span.js';
import { HttpError } from '../HttpError.js';
import {
DEFAULT_FREE_SUBSCRIPTION,
FREE_SUBSCRIPTION_IDS,
} from '../../../services/metering/consts.js';
import { subscriptionOverride } from '../../../services/metering/consts.js';
/**
* Sliding-window rate limiter with swappable, **co-resident** backends.
@@ -747,6 +744,29 @@ export async function consumeRouteRateLimit(req, spec) {
}
}
/**
* `consumeRouteRateLimit` without the spend: same spec, same bucket. Pair the
* two when only a successful outcome should count — peek before the work,
* consume after it succeeds. Single-window specs only. Fails open.
*/
export async function peekRouteRateLimit(req, spec) {
const {
window: windowMs,
key: strategy = 'fingerprint',
scope,
backend,
} = spec;
const backendPair = resolveBackend(backend);
const key = resolveKey(req, scope ?? req.route?.path ?? 'route', strategy);
try {
const limit = await resolveSubscriptionLimit(req, spec);
return await backendPair.peek(key, limit, windowMs);
} catch (err) {
console.error('[rate-limit] handler peek failed, failing open:', err);
return true;
}
}
/**
* Read whether `key` still has budget, without spending any. The twin to
* `checkRateLimit` for gates whose budget is consumed by an outcome rather than
@@ -846,15 +866,6 @@ export const CONCURRENT_SLOT_TTL_MS = ORPHAN_SAFETY_TTL_MS;
* actor, no metering, metering throws) falls through to the base — rate /
* concurrency limiting should never _amplify_ a request failure path.
*/
// An unlisted free plan would otherwise take `limit`, the paid cap.
function overrideFor(bySubscription, subscriptionId) {
const own = bySubscription[subscriptionId];
if (typeof own === 'number') return own;
return FREE_SUBSCRIPTION_IDS.has(subscriptionId)
? bySubscription[DEFAULT_FREE_SUBSCRIPTION]
: undefined;
}
async function resolveSubscriptionLimit(req, opts) {
const base = opts.limit;
if (!opts.bySubscription || !meteringService) return base;
@@ -862,7 +873,7 @@ async function resolveSubscriptionLimit(req, opts) {
if (!actor?.user?.uuid) return base;
try {
const sub = await meteringService.getActorSubscription(actor);
const override = overrideFor(opts.bySubscription, sub.id);
const override = subscriptionOverride(opts.bySubscription, sub.id);
return typeof override === 'number' ? override : base;
} catch {
return base;
+11 -3
View File
@@ -27,7 +27,12 @@ import type { Actor } from '../actor';
* it instead of accepting any token that authenticates.
*/
export type TokenSource =
'body' | 'header' | 'x-api-key' | 'cookie' | 'query' | 'handshake';
| 'body'
| 'header'
| 'x-api-key'
| 'cookie'
| 'query'
| 'handshake';
/** Express router methods plus the WebDAV verbs some endpoints use. */
export type RouteMethod =
@@ -69,8 +74,9 @@ export type AccountGateUser = Partial<
| 'requires_password_change'
| 'phone'
| 'card_fingerprint'
| 'otp_enabled'
>
>;
> & { id?: number };
/** One rate-limit window. See `RouteOptions.rateLimit` for semantics. */
export interface RouteRateLimit {
@@ -335,7 +341,9 @@ export type AuthRequired<O extends RouteOptions> = O extends {
? true
: O extends {
requireSubscription:
true | readonly string[] | string[];
| true
| readonly string[]
| string[];
}
? true
: O extends { requireReputation: string }
@@ -167,6 +167,25 @@ const errorFor = async (
// -- Provider registration -------------------------------------------
describe('ChatCompletionDriver provider registration', () => {
it('routes test mode to fake-chat without calling the selected upstream provider', async () => {
const upstream = vi.spyOn(AzureChatProvider.prototype, 'complete');
const fake = vi.spyOn(FakeChatProvider.prototype, 'complete');
const result = await withTestActor(() =>
fullDriver.complete({
model: 'gpt-4o',
provider: 'azure-openai',
messages: [{ role: 'user', content: 'hi' }],
test_mode: true,
}),
);
expect(upstream).not.toHaveBeenCalled();
expect(fake).toHaveBeenCalledOnce();
expect(fake.mock.calls[0]![0].model).toBe('fake');
expect('message' in result && result.message.model).toBe('fake');
});
it('registers a model surface spanning every credentialed provider', async () => {
const models = await fullDriver.models();
const providers = new Set(models.map((m) => m.provider));
@@ -254,6 +273,25 @@ describe('ChatCompletionDriver provider registration', () => {
});
});
// -- Input validation ------------------------------------------------
describe('ChatCompletionDriver malformed input', () => {
it.each([
['missing messages', { messages: undefined }],
[
'a null content item',
{ messages: [{ role: 'user', content: [null] }] },
],
['a null tool entry', { tools: [null] }],
])('rejects %s with 400 before any provider runs', async (_label, args) => {
const fake = vi.spyOn(FakeChatProvider.prototype, 'complete');
await expect(completeFake(args)).rejects.toMatchObject({
statusCode: 400,
});
expect(fake).not.toHaveBeenCalled();
});
});
// -- Failure classification ------------------------------------------
describe('ChatCompletionDriver exhausted-chain classification', () => {
@@ -585,9 +623,9 @@ describe('ChatCompletionDriver cross-provider fallback', () => {
Object.assign(new Error('azure down'), { status: 503 }),
);
const openai = vi.spyOn(OpenAiChatProvider.prototype, 'complete');
vi.spyOn(server.services.metering, 'getRemainingUsage')
.mockResolvedValueOnce(1_000_000) // pre-flight
.mockResolvedValue(0); // drained by a parallel request
vi.spyOn(server.services.metering, 'getUsageHeadroom')
.mockResolvedValueOnce({ balance: 1_000_000, held: 0 }) // pre-flight
.mockResolvedValue({ balance: 0, held: 0 }); // drained by a parallel request
await expect(completeShared()).rejects.toMatchObject({
statusCode: 402,
@@ -38,6 +38,7 @@ import {
vi,
} from 'vitest';
import { Context } from '../../core/context.js';
import type { UsageInput } from '../../services/metering/types.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
@@ -45,7 +46,7 @@ import { withTestActor } from '../integrationTestUtil.js';
import { ChatCompletionDriver } from './ChatCompletionDriver.js';
import { FakeChatProvider } from './providers/FakeChatProvider.js';
import type { IChatCompleteResult } from './types.js';
import type { AIChatStream } from './utils/Streaming.js';
import { type AIChatStream, ChatStreamAbortedError } from './utils/Streaming.js';
let server: PuterServer;
@@ -365,6 +366,8 @@ describe('ChatCompletionDriver credit gate against real metering', () => {
// Let the first request clear the gate and take its hold.
await vi.waitFor(() => expect(completeSpy).toHaveBeenCalledTimes(1));
// Nothing is spent yet, only held: a retry once the first finishes is
// the remedy, not a top-up.
await expect(
withTestActor(
() =>
@@ -375,8 +378,9 @@ describe('ChatCompletionDriver credit gate against real metering', () => {
actor,
),
).rejects.toMatchObject({
statusCode: 402,
legacyCode: 'insufficient_funds',
statusCode: 429,
legacyCode: 'too_many_requests',
code: 'credits_reserved',
});
// The second request never reached a provider.
expect(completeSpy).toHaveBeenCalledTimes(1);
@@ -434,6 +438,119 @@ describe('ChatCompletionDriver credit gate against real metering', () => {
});
});
describe('ChatCompletionDriver credit gate with an AI cost factor', () => {
const FACTOR = 1.3;
let seenKeys: string[];
const listener = (key: string, event: { factor: number }) => {
seenKeys.push(key);
event.factor = FACTOR;
};
beforeEach(() => {
seenKeys = [];
server.clients.event.on('ai.cost.factor.*', listener as never);
});
afterEach(() => {
server.clients.event.off('ai.cost.factor.*', listener as never);
});
const okResult = {
message: { role: 'assistant', content: [{ type: 'text', text: 'ok' }] },
usage: { input_tokens: 1, output_tokens: 1 },
finish_reason: 'stop',
} as never;
// Usage is recorded at cost × factor, so a cap sized at the bare cost
// lets a completion spend past what's left.
it('caps output at the factored price', async () => {
const actor = freeUser() as never;
const metering = server.services.metering;
const allowance = (await metering.getActorSubscription(actor))
.monthUsageAllowance;
await metering.incrementUsage(
actor,
'test:prior-spend',
1,
Math.floor(allowance * 0.9),
);
const remaining = await metering.getRemainingUsage(actor);
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockResolvedValueOnce(okResult);
await withTestActor(
() =>
driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'go' }],
max_tokens: 100_000,
}),
actor,
);
const passed = completeSpy.mock.calls[0]![0] as { max_tokens: number };
expect(passed.max_tokens * 2000 * FACTOR).toBeLessThanOrEqual(
remaining,
);
expect(passed.max_tokens).toBeGreaterThan(
remaining / (2000 * FACTOR) - 10,
);
});
it('reserves the factored worst case', async () => {
const actor = freeUser() as never;
const reserve = vi.spyOn(server.services.metering, 'reserveCredits');
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockResolvedValueOnce(okResult);
await withTestActor(
() =>
driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'go' }],
max_tokens: 10,
}),
actor,
);
const passed = completeSpy.mock.calls[0]![0] as { max_tokens: number };
expect(passed.max_tokens).toBe(10);
const [, held] = reserve.mock.calls[0]!;
// At least the 10 capped output tokens at 2000 × factor.
expect(held).toBeGreaterThanOrEqual(10 * 2000 * FACTOR);
});
// The factor is looked up by the key the provider records under, or a
// per-model factor would price the gate and the charge differently.
it('asks for the factor under the metering key the provider records with', async () => {
const proto = FakeChatProvider.prototype as unknown as {
meteringModelKey?: (id: string) => string;
};
proto.meteringModelKey = (id) => `fake-recorded:${id}`;
try {
vi.spyOn(
FakeChatProvider.prototype,
'complete',
).mockResolvedValueOnce(okResult);
await withTestActor(
() =>
driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'go' }],
}),
freeUser() as never,
);
expect(seenKeys).toContain(
'ai.cost.factor.ai-chat.fake-recorded:priced',
);
} finally {
delete proto.meteringModelKey;
}
});
});
describe('ChatCompletionDriver credit gate on multimodal prompts', () => {
it('prices attachments into the pre-flight affordability check', async () => {
const actor = freeUser() as never;
@@ -506,3 +623,244 @@ describe('ChatCompletionDriver credit gate on multimodal prompts', () => {
expect(completeSpy).toHaveBeenCalledTimes(1);
});
});
// `max_tokens` is a model's output limit and `context` its whole window; the
// cap is the lesser of the limit and what the window leaves after the prompt.
describe('ChatCompletionDriver output ceiling', () => {
// ~15k estimated tokens: past an 8192-token output limit.
const longPrompt = [{ role: 'user', content: 'x '.repeat(20_000) }];
const withModel = async (model: Record<string, unknown>) => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValue([
{ ...PRICED_MODEL, ...model },
] as never);
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: Number.MAX_SAFE_INTEGER, held: 0 });
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockResolvedValue({
message: { role: 'assistant', content: 'ok' },
usage: { input_tokens: 1, output_tokens: 1 },
finish_reason: 'stop',
} as never);
const d = await makeDriver();
await withTestActor(() =>
d.complete({ model: 'priced', messages: longPrompt }),
);
return (completeSpy.mock.calls[0]![0] as { max_tokens: number })
.max_tokens;
};
it('serves a prompt longer than the output limit when the window fits it', async () => {
expect(await withModel({ max_tokens: 8192, context: 200_000 })).toBe(
8192,
);
});
it('caps output at what the window leaves after the prompt', async () => {
const maxTokens = await withModel({
max_tokens: 32_000,
context: 32_000,
});
expect(maxTokens).toBeGreaterThan(0);
expect(maxTokens).toBeLessThan(32_000 - 10_000);
});
it('leaves a prompt estimated past the window for the provider to refuse', async () => {
expect(await withModel({ max_tokens: 8192, context: 10_000 })).toBe(
8192,
);
});
});
describe('ChatCompletionDriver when the balance runs out', () => {
const spentToATenth = async () => {
const actor = freeUser() as never;
const metering = server.services.metering;
const allowance = (await metering.getActorSubscription(actor))
.monthUsageAllowance;
await metering.incrementUsage(
actor,
'test:prior-spend',
1,
Math.floor(allowance * 0.9),
);
return actor;
};
/** A non-stream completion that reports `output(cap)` output tokens. */
const completeWithOutput = async (
actor: never,
output: (cap: number) => number,
) => {
vi.spyOn(FakeChatProvider.prototype, 'complete').mockImplementationOnce(
async (args) =>
({
message: { role: 'assistant', content: 'cut' },
usage: {
input_tokens: 1,
output_tokens: output(
(args as { max_tokens: number }).max_tokens,
),
},
finish_reason: 'length',
}) as never,
);
return withTestActor(async () => {
await driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'go' }],
});
return Context.get('driverMetadata') as Record<string, unknown>;
}, actor);
};
it('flags a completion that used all of a balance-bound cap', async () => {
const metadata = await completeWithOutput(
await spentToATenth(),
(cap) => cap,
);
expect(metadata.usage_limited).toBe(true);
});
it('leaves a completion that stopped short of the cap alone', async () => {
const metadata = await completeWithOutput(
await spentToATenth(),
(cap) => cap - 1,
);
expect(metadata.usage_limited).toBeUndefined();
});
it('does not flag a cap that only the account’s own holds shrank', async () => {
const actor = freeUser() as never;
const metering = server.services.metering;
const allowance = (await metering.getActorSubscription(actor))
.monthUsageAllowance;
// The whole 8192-token limit is affordable from the balance...
expect(allowance - 1000).toBeGreaterThan(8192 * 2000);
// ...but a request in flight holds most of it.
const hold = await metering.reserveCredits(
actor,
allowance - 8192 * 1000,
);
try {
const metadata = await completeWithOutput(actor, (cap) => cap);
expect(metadata.usage_limited).toBeUndefined();
} finally {
await hold.release();
}
});
it('marks the usage line of a stream that ran the balance out', async () => {
const actor = await spentToATenth();
vi.spyOn(FakeChatProvider.prototype, 'complete').mockImplementationOnce(
async (args) =>
streamOf(async ({ chatStream }) => {
chatStream
.message()
.contentBlock({ type: 'text' })
.addText('cut');
chatStream.end({
input_tokens: 1,
output_tokens: (args as { max_tokens: number })
.max_tokens,
});
}) as never,
);
const { stream } = (await withTestActor(
() =>
driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'go' }],
stream: true,
}),
actor,
)) as unknown as { stream: Readable };
let body = '';
for await (const chunk of stream as AsyncIterable<Buffer>) {
body += chunk.toString();
}
const usageLine = body
.trim()
.split('\n')
.map((l) => JSON.parse(l))
.find((l) => l.type === 'usage');
expect(usageLine.metadata).toEqual({ usage_limited: true });
});
it('refuses with 402, not the retryable 429, when nothing is held', async () => {
const actor = freeUser() as never;
const metering = server.services.metering;
const allowance = (await metering.getActorSubscription(actor))
.monthUsageAllowance;
await metering.incrementUsage(actor, 'test:prior-spend', 1, allowance);
await expect(
withTestActor(
() =>
driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'go' }],
}),
actor,
),
).rejects.toMatchObject({
statusCode: 402,
legacyCode: 'insufficient_funds',
});
});
});
describe('ChatCompletionDriver when the caller hangs up', () => {
it('stops the generation and gives the hold back', async () => {
const actor = freeUser() as never;
const metering = server.services.metering;
const abort = new AbortController();
let resume: () => void = () => {};
const paused = new Promise<void>((r) => {
resume = r;
});
let providerError: unknown;
vi.spyOn(FakeChatProvider.prototype, 'complete').mockResolvedValueOnce(
streamOf(async ({ chatStream }) => {
const block = chatStream.message().contentBlock({ type: 'text' });
block.addText('first');
await paused;
try {
// An upstream still generating: keeps writing until told.
for (let i = 0; i < 1000; i++) block.addText('more');
} catch (e) {
providerError = e;
throw e;
}
}) as never,
);
const metered = vi.spyOn(metering, 'batchIncrementUsages');
const { stream } = (await withTestActor(() => {
Context.set('abortSignal', abort.signal);
return driver.complete({
model: 'priced',
messages: [{ role: 'user', content: 'go' }],
stream: true,
});
}, actor)) as unknown as { stream: Readable };
expect(await metering.getOutstandingHolds(actor)).toBeGreaterThan(0);
abort.abort();
resume();
let body = '';
for await (const chunk of stream as AsyncIterable<Buffer>) {
body += chunk.toString();
}
await new Promise((r) => setTimeout(r, 20));
expect(providerError).toBeInstanceOf(ChatStreamAbortedError);
// Nobody is listening, so no error line is written for the abort.
expect(body).not.toContain('"type":"error"');
expect(await metering.getOutstandingHolds(actor)).toBe(0);
// What was generated before the hang-up is still charged.
expect(metered).toHaveBeenCalled();
});
});
@@ -333,7 +333,7 @@ describe('ChatCompletionDriver credit gate across the fallback chain', () => {
// real balance read of its own.
const remaining = vi.spyOn(
server.services.metering,
'getRemainingUsage',
'getUsageHeadroom',
);
const attempts = await attemptsFor('deepseek-v4-pro');
@@ -180,6 +180,46 @@ describe('ChatCompletionDriver model catalog', () => {
// ── Auth + model resolution ─────────────────────────────────────────
describe('ChatCompletionDriver.complete auth and model resolution', () => {
it('uses a free fake completion in test mode without taking a credit hold', async () => {
const creditGate = vi.spyOn(server.services.metering, 'getUsageHeadroom');
const complete = vi.spyOn(FakeChatProvider.prototype, 'complete');
const result = await withTestActor(() =>
driver.complete({
model: 'costly',
messages: [{ role: 'user', content: 'hi' }],
test_mode: true,
}),
);
expect(creditGate).not.toHaveBeenCalled();
expect(complete).toHaveBeenCalledOnce();
expect(complete.mock.calls[0]![0].model).toBe('fake');
expect('message' in result && result.message.model).toBe('fake');
expect('usage' in result && result.usage).toMatchObject({
input_tokens: 0,
output_tokens: 1,
});
});
it('streams a fake completion in test mode', async () => {
const complete = vi.spyOn(FakeChatProvider.prototype, 'complete');
const result = await withTestActor(() =>
driver.complete({
model: 'costly',
messages: [{ role: 'user', content: 'hi' }],
stream: true,
test_mode: true,
}),
);
expect(complete.mock.calls[0]![0].model).toBe('fake');
const streamResult = result as unknown as { dataType: string; stream: Readable };
expect(streamResult.dataType).toBe('stream');
const chunks = await collectStream(streamResult.stream);
expect(chunks).toContainEqual(expect.objectContaining({ type: 'text' }));
});
it('throws 401 when no actor is in context', async () => {
// Note: not wrapped in `withTestActor` — `Context.get('actor')`
// returns undefined.
@@ -463,53 +503,6 @@ describe('ChatCompletionDriver.complete events and cost emission', () => {
expect(data.service_used).toBe('fake-chat');
});
it('charges cached prompt tokens only at their cache rate', async () => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValueOnce([
{
id: 'cached-pricing',
costs_currency: 'usd-cents',
input_cost_key: 'prompt_tokens',
output_cost_key: 'completion_tokens',
promptTokensIncludeCached: true,
costs: {
tokens: 1_000_000,
prompt_tokens: 100,
completion_tokens: 200,
cached_tokens: 10,
},
max_tokens: 8192,
},
]);
const driver = await makeDriver();
const costEvents = captureEvent('ai.prompt.cost-calculated');
vi.spyOn(FakeChatProvider.prototype, 'complete').mockResolvedValueOnce({
message: {
role: 'assistant',
content: [{ type: 'text', text: 'ok' }],
},
usage: {
prompt_tokens: 100,
completion_tokens: 1,
cached_tokens: 40,
},
finish_reason: 'stop',
} as never);
await withTestActor(() =>
driver.complete({
model: 'cached-pricing',
messages: [{ role: 'user', content: 'hello' }],
}),
);
const data = costEvents[0]![1] as {
input_ucents: number;
output_ucents: number;
};
expect(data.input_ucents).toBe(60 * 100 + 40 * 10);
expect(data.output_ucents).toBe(200);
});
it('injects `usd_cents` on the usage object derived from the cost map', async () => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValueOnce([
{
@@ -682,9 +675,7 @@ describe('ChatCompletionDriver.complete validation event routing', () => {
describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
it('throws 402 `insufficient_funds` when the actor has no remaining credits', async () => {
vi.spyOn(server.services.metering, 'getRemainingUsage').mockResolvedValue(
0,
);
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 0, held: 0 });
await expect(
withTestActor(() =>
@@ -719,9 +710,7 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
// Approx input cost is tiny (very short prompt), so allowed
// output ≈ 100_000 / 2000 = 50 tokens, comfortably below the
// caller's 10_000 ceiling.
vi.spyOn(server.services.metering, 'getRemainingUsage').mockResolvedValue(
100_000,
);
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 100_000, held: 0 });
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
@@ -768,9 +757,7 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
// A ~100-token prompt is past the threshold. 1_000_000 microcents at
// 2000 * 1.5 per output token leaves at most 333 output tokens before
// the prompt is paid for; the standard rate would allow ~450 after it.
vi.spyOn(server.services.metering, 'getRemainingUsage').mockResolvedValue(
1_000_000,
);
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 1_000_000, held: 0 });
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
@@ -813,9 +800,7 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
// Pass the cheap pre-flight check but leave a balance too small to
// afford a single 2000-microcent output token.
vi.spyOn(server.services.metering, 'getRemainingUsage').mockResolvedValue(
100,
);
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 100, held: 0 });
const completeSpy = vi.spyOn(FakeChatProvider.prototype, 'complete');
@@ -858,8 +843,8 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
vi.spyOn(
server.services.metering,
'getRemainingUsage',
).mockResolvedValue(100_000);
'getUsageHeadroom',
).mockResolvedValue({ balance: 100_000, held: 0 });
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
@@ -901,9 +886,7 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
},
]);
const d = await makeDriver();
vi.spyOn(server.services.metering, 'getRemainingUsage').mockResolvedValue(
1_000_000,
);
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 1_000_000, held: 0 });
vi.spyOn(
server.services.metering,
'getActorSubscription',
@@ -922,6 +905,44 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
});
});
it('admits a plan outside the free set to subscriber-only models', async () => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValueOnce([
{
id: 'subonly-paid',
aliases: [],
costs_currency: 'usd-cents',
costs: { 'input-tokens': 100, 'output-tokens': 100 },
max_tokens: 8192,
subscriberOnly: true,
},
]);
const d = await makeDriver();
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 1_000_000, held: 0 });
vi.spyOn(
server.services.metering,
'getActorSubscription',
).mockResolvedValue({ id: 'some-paid-tier' } as never);
const completeSpy = vi
.spyOn(FakeChatProvider.prototype, 'complete')
.mockResolvedValueOnce({
message: {
role: 'assistant',
content: [{ type: 'text', text: 'ok' }],
},
usage: { input_tokens: 1, output_tokens: 1 },
finish_reason: 'stop',
} as never);
await withTestActor(() =>
d.complete({
model: 'subonly-paid',
messages: [{ role: 'user', content: 'hi' }],
}),
);
expect(completeSpy).toHaveBeenCalledOnce();
});
it('rejects subscriber-only models for the default free subscription', async () => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValueOnce([
{
@@ -935,9 +956,7 @@ describe('ChatCompletionDriver.complete credit gate and max_tokens cap', () => {
]);
const d = await makeDriver();
// Plenty of credits so the credit gate doesn't intercept first.
vi.spyOn(server.services.metering, 'getRemainingUsage').mockResolvedValue(
1_000_000,
);
vi.spyOn(server.services.metering, 'getUsageHeadroom').mockResolvedValue({ balance: 1_000_000, held: 0 });
await expect(
withTestActor(() =>
@@ -1452,9 +1471,9 @@ describe('ChatCompletionDriver.complete fallback and error envelope', () => {
new Error('boom'),
);
const remaining = vi
.spyOn(server.services.metering, 'getRemainingUsage')
.mockResolvedValueOnce(1_000_000) // pre-flight
.mockResolvedValueOnce(0); // drained mid-fallback
.spyOn(server.services.metering, 'getUsageHeadroom')
.mockResolvedValueOnce({ balance: 1_000_000, held: 0 }) // pre-flight
.mockResolvedValueOnce({ balance: 0, held: 0 }); // drained mid-fallback
// No second provider serves `fake`, so `#findFallback` returns
// null and the loop exits before reaching the credit re-check.
@@ -23,12 +23,15 @@ import { EventMap } from '../../clients/event/types.js';
import type { Actor } from '../../core/actor.js';
import { Context } from '../../core/context.js';
import { HttpError, isHttpError } from '../../core/http/HttpError.js';
import { FREE_SUBSCRIPTION_IDS } from '../../services/metering/consts.js';
import { isFreeSubscription } from '../../services/metering/consts.js';
import type { CreditHold } from '../../services/metering/types.js';
import { NO_CREDIT_HOLD } from '../../services/metering/types.js';
import type { MeteringService } from '../../services/metering/MeteringService.js';
import type { DriverStreamResult } from '../meta.js';
import { PuterDriver } from '../types.js';
import {
type AiMeteringService,
withAiCostFactor,
} from '../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../util/aiLimits.js';
import {
isCreditExhaustion as isUpstreamCreditExhaustion,
@@ -41,7 +44,6 @@ import { AzureResponsesProvider } from './providers/azure/AzureResponsesProvider
import { BytePlusProvider } from './providers/byteplus/BytePlusProvider.js';
import { ClaudeProvider } from './providers/claude/ClaudeProvider.js';
import { DeepSeekProvider } from './providers/deepseek/DeepSeekProvider.js';
import { deepSeekPricing } from './providers/deepseek/pricing.js';
import { FakeChatProvider } from './providers/FakeChatProvider.js';
import { GeminiChatProvider } from './providers/gemini/GeminiChatProvider.js';
import { GroqAIProvider } from './providers/groq/GroqAIProvider.js';
@@ -93,6 +95,7 @@ import {
isOutputCostKey,
longContextMultipliers,
trackedInputTokens,
trackedOutputTokens,
} from './utils/pricing.js';
import {
isRouteUnhealthy,
@@ -122,6 +125,43 @@ const HOLD_RENEW_INTERVAL_MS = 5 * 60 * 1000;
const isModerationRefusal = (e: unknown): boolean =>
isHttpError(e) && e.code === 'moderation_flagged';
/**
* The refusal for a request its funds can't cover. When what's missing is only
* held by the account's own requests still in flight, waiting is the remedy,
* not a top-up: a retryable 429 rather than a 402 that prompts an upgrade.
*/
const fundsRefusal = (heldByInFlight: boolean): HttpError =>
heldByInFlight
? new HttpError(
429,
'Usage is reserved by requests still running. Retry when they finish.',
{ legacyCode: 'too_many_requests', code: 'credits_reserved' },
)
: new HttpError(402, 'No usage left for request.', {
legacyCode: 'insufficient_funds',
});
const positiveOrInfinity = (n: unknown): number =>
typeof n === 'number' && Number.isFinite(n) && n > 0
? n
: Number.POSITIVE_INFINITY;
/**
* The most output a request to `model` can get: its output limit, and no more
* than the context window leaves after the prompt. A limit a catalog doesn't
* know bounds nothing. A prompt estimated to fill the window is left for the
* provider to refuse — the estimate is approximate, and running out of credits
* would be the wrong answer either way.
*/
const outputCeiling = (
model: IChatModel,
promptTokenEstimate: number,
): number => {
const outputLimit = positiveOrInfinity(model.max_tokens);
const contextRoom = positiveOrInfinity(model.context) - promptTokenEstimate;
return contextRoom >= 1 ? Math.min(outputLimit, contextRoom) : outputLimit;
};
type ProviderAttempt = {
model: string;
provider: string;
@@ -327,8 +367,12 @@ export class ChatCompletionDriver extends PuterDriver {
#modelIdMap: Record<string, IChatModel[]> = Object.create(null);
/** Metering scoped to this driver. Lazy: services wire up after drivers. */
get #aiMetering(): MeteringService {
return this.services.metering.withAiCostFactor(this.driverName);
get #aiMetering(): AiMeteringService {
return withAiCostFactor(
this.services.metering,
this.clients.event,
this.driverName,
);
}
override onServerStart() {
@@ -412,11 +456,7 @@ export class ChatCompletionDriver extends PuterDriver {
});
}
if (args.messages) {
args.messages = normalizeMediaParts(
normalize_messages(args.messages),
);
}
args.messages = normalizeMediaParts(normalize_messages(args.messages));
if (args.tools) {
normalize_tools_object(args.tools);
}
@@ -494,8 +534,20 @@ export class ChatCompletionDriver extends PuterDriver {
}
}
// Skipped for blocked requests since fake-chat is free and the user
// shouldn't see a billing error in place of the abuse page.
if (args.test_mode === true && !blocked) {
const fakeModel = this.#resolveModel('fake', 'fake-chat');
if (!fakeModel) {
throw new HttpError(500, 'Test chat provider unavailable', {
legacyCode: 'internal_error',
});
}
model = fakeModel;
}
const useFakeProvider = blocked || args.test_mode === true;
// Fake responses need no credit hold, including blocked prompts and
// test requests.
//
// The gate hands back a hold on what this attempt could cost, which
// stands in for its usage until the real numbers land. It is released
@@ -503,11 +555,15 @@ export class ChatCompletionDriver extends PuterDriver {
// where "done" is the stream draining rather than this method
// returning.
let hold: CreditHold = NO_CREDIT_HOLD;
if (!blocked) {
hold = await this.#applyCreditGate(actor, model, args, {
promptTokenEstimate,
requestedMaxTokens,
});
// The current attempt's balance-bound output cap, if it had one.
let fundsCap: number | undefined;
if (!useFakeProvider) {
({ hold, fundsCap } = await this.#applyCreditGate(
actor,
model,
args,
{ promptTokenEstimate, requestedMaxTokens },
));
}
// First attempt
@@ -538,7 +594,9 @@ export class ChatCompletionDriver extends PuterDriver {
};
try {
if (!blocked) await this.#resolvePuterPaths(provider, args, actor);
if (!useFakeProvider) {
await this.#resolvePuterPaths(provider, args, actor);
}
res = await provider.complete({
...args,
model: model.id,
@@ -577,17 +635,19 @@ export class ChatCompletionDriver extends PuterDriver {
// be capped against what is actually left.
// The previous attempt released its hold when it failed, so
// this one starts from nothing held.
if (!blocked) {
hold = await this.#applyCreditGate(actor, fallback, args, {
promptTokenEstimate,
requestedMaxTokens,
});
if (!useFakeProvider) {
({ hold, fundsCap } = await this.#applyCreditGate(
actor,
fallback,
args,
{ promptTokenEstimate, requestedMaxTokens },
));
}
tried.add(routeId(fallback.provider!, fallback.id));
try {
if (!blocked) {
if (!useFakeProvider) {
await this.#resolvePuterPaths(fbProvider, args, actor);
}
res = await fbProvider.complete({
@@ -622,6 +682,13 @@ export class ChatCompletionDriver extends PuterDriver {
const username = actor.user?.username;
// A completion that used all of a balance-bound cap stopped because
// the account ran dry, so the caller is told to top up.
const ranOutOfFunds = (usage: Record<string, unknown> | undefined) =>
fundsCap !== undefined &&
!!usage &&
trackedOutputTokens(usage, model) >= fundsCap;
// Streaming result — create a PassThrough, kick off the provider's
// stream populator, and return a DriverStreamResult so the route
// handler pipes it to the HTTP response as chunked NDJSON.
@@ -631,6 +698,13 @@ export class ChatCompletionDriver extends PuterDriver {
const init = res.init_chat_stream;
const cleanup = res.finally_fn;
// A caller that hangs up stops the generation, and with it the
// hold and the billing for output nobody will read.
const abortSignal = Context.get('abortSignal');
const abortStream = () => chatStream.abort();
if (abortSignal?.aborted) abortStream();
else abortSignal?.addEventListener('abort', abortStream);
// Intercept `chatStream.end(usage)` to fire complete + cost events
// (mirrors the non-streaming branch). Clone usage so providers that
// meter after this call (e.g. Claude) don't pick up `usd_cents`.
@@ -662,7 +736,12 @@ export class ChatCompletionDriver extends PuterDriver {
intendedProvider,
});
}
return originalEnd(enrichedUsage!);
return originalEnd(
enrichedUsage!,
ranOutOfFunds(usage)
? { metadata: { usage_limited: true } }
: undefined,
);
};
// The hold lives for the whole stream, which can outlast its TTL —
@@ -678,17 +757,20 @@ export class ChatCompletionDriver extends PuterDriver {
try {
await init({ chatStream });
} catch (e) {
passthrough.write(
`${JSON.stringify({
type: 'error',
message: sanitizeUpstreamMessage(
e instanceof Error ? e.message : String(e),
),
})}\n`,
);
if (!chatStream.aborted) {
passthrough.write(
`${JSON.stringify({
type: 'error',
message: sanitizeUpstreamMessage(
e instanceof Error ? e.message : String(e),
),
})}\n`,
);
}
passthrough.end();
} finally {
clearInterval(renewHold);
abortSignal?.removeEventListener('abort', abortStream);
// Providers report usage the moment they meter it (see
// `AIChatStream.reportUsage`); a stream that never got
// there was never charged for.
@@ -757,6 +839,13 @@ export class ChatCompletionDriver extends PuterDriver {
Context.set('driverMetadata', {
service_used: model.provider,
providerUsed: model.id,
...(ranOutOfFunds(
'usage' in res
? (res.usage as Record<string, unknown>)
: undefined,
)
? { usage_limited: true }
: {}),
});
// Response-format precedence: an explicit per-call `normalize` wins in
@@ -825,9 +914,6 @@ export class ChatCompletionDriver extends PuterDriver {
model,
trackedInputTokens(usage, model),
);
const cachedTokens = Number.isFinite(usage.cached_tokens)
? Math.max(0, usage.cached_tokens)
: 0;
let inputMicroCents = 0;
let outputMicroCents = 0;
@@ -841,11 +927,6 @@ export class ChatCompletionDriver extends PuterDriver {
if (key === 'usd_cents') continue;
if (key === 'tokens') continue;
const amount =
model.promptTokensIncludeCached && key === inputKey
? Math.max(0, rawAmount - cachedTokens)
: rawAmount;
// thinking_tokens → output rate fallback
let rate = costs[key];
if (typeof rate !== 'number' || !Number.isFinite(rate)) {
@@ -868,9 +949,9 @@ export class ChatCompletionDriver extends PuterDriver {
sawAnyRate = true;
if (isOutputKey(key)) {
outputMicroCents += amount * rate * multipliers.output;
outputMicroCents += rawAmount * rate * multipliers.output;
} else {
inputMicroCents += amount * rate * multipliers.input;
inputMicroCents += rawAmount * rate * multipliers.input;
}
}
@@ -913,7 +994,9 @@ export class ChatCompletionDriver extends PuterDriver {
*
* Returns a hold on what the attempt can cost at worst, so requests this
* account is running in parallel see the spend before it is recorded. The
* caller releases it once the attempt is done.
* caller releases it once the attempt is done. `fundsCap` is set when the
* account's balance, not the request or the model, decided the output cap —
* a completion that uses all of it has run the account dry.
*/
async #applyCreditGate(
actor: Actor,
@@ -933,34 +1016,46 @@ export class ChatCompletionDriver extends PuterDriver {
*/
requestedMaxTokens: number | undefined;
},
): Promise<CreditHold> {
): Promise<{ hold: CreditHold; fundsCap?: number }> {
const metering = this.services.metering;
const { promptTokenEstimate, requestedMaxTokens } = estimates;
const { inputKey, outputKey } = costKeys(model);
// A prompt estimated past a long-context threshold pays the raised
// rates on input and output alike.
const multipliers = longContextMultipliers(model, promptTokenEstimate);
// Recorded costs pass through the AI cost factor; the gate prices at
// the same rate or it under-reserves on every request.
const costFactor = await this.#aiMetering.costFactor(
actor,
this.#meteringModelKey(model),
);
// `|| 0` also catches NaN from a malformed cost table.
const inputTokenCost =
(Number(model.costs?.[inputKey] ?? 0) || 0) * multipliers.input;
(Number(model.costs?.[inputKey] ?? 0) || 0) *
multipliers.input *
costFactor;
const outputTokenCost =
(Number(model.costs?.[outputKey] ?? 0) || 0) * multipliers.output;
(Number(model.costs?.[outputKey] ?? 0) || 0) *
multipliers.output *
costFactor;
const approximateInputCost = promptTokenEstimate * inputTokenCost;
const minimumCredits = Number(model.minimumCredits || 1);
const needed = Math.max(approximateInputCost, minimumCredits);
// One balance read serves the whole gate: the affordability check
// here and the output cap below.
const remainingCredits = await metering.getRemainingUsage(actor);
if (remainingCredits < Math.max(approximateInputCost, minimumCredits)) {
throw new HttpError(402, 'No usage left for request.', {
legacyCode: 'insufficient_funds',
});
// One read of the balance and its holds serves the whole gate: the
// affordability check here and the output cap below, net of holds,
// and the balance alone, which tells spent from committed to requests
// that are still running.
const { balance, held } = await metering.getUsageHeadroom(actor);
const remainingCredits = Math.max(0, balance - held);
if (remainingCredits < needed) {
throw fundsRefusal(balance >= needed);
}
if (model.subscriberOnly) {
const subscription = await metering.getActorSubscription(actor);
// Every free plan, not two named ones.
if (FREE_SUBSCRIPTION_IDS.has(subscription.id)) {
if (isFreeSubscription(subscription.id)) {
throw new HttpError(
403,
`The model ${model.id} is only available to subscribers. Please subscribe to access this model.`,
@@ -969,25 +1064,17 @@ export class ChatCompletionDriver extends PuterDriver {
}
}
let fundsCap: number | undefined;
if (outputTokenCost > 0) {
const maxAllowedOutputUcents =
remainingCredits - approximateInputCost;
const maxAllowedOutputTokens =
maxAllowedOutputUcents / outputTokenCost;
// A provider may not know a model's output ceiling. Drop the term
// rather than let a missing value drive the cap: `null` coerces to
// 0, so the subtraction goes negative instead of NaN and the user
// is told they're out of credits.
const modelOutputCeiling =
Number.isFinite(model.max_tokens) && model.max_tokens > 0
? model.max_tokens - promptTokenEstimate
: Number.POSITIVE_INFINITY;
const affordableOutputTokens = (credits: number): number =>
(credits - approximateInputCost) / outputTokenCost;
// What the request would get with money no object.
const limit = Math.min(
requestedMaxTokens ?? Number.POSITIVE_INFINITY,
outputCeiling(model, promptTokenEstimate),
);
const cap = Math.floor(
Math.min(
requestedMaxTokens ?? Number.POSITIVE_INFINITY,
maxAllowedOutputTokens,
modelOutputCeiling,
),
Math.min(limit, affordableOutputTokens(remainingCredits)),
);
// `cap` is the credit-bounded ceiling on output tokens. When it
// drops below 1 the user can't afford even a single output token,
@@ -999,11 +1086,20 @@ export class ChatCompletionDriver extends PuterDriver {
// too: a NaN requested max_tokens poisons the Math.min above, and
// `NaN < 1` is false — letting an uncapped request through.
if (!(cap >= 1)) {
throw new HttpError(402, 'No usage left for request.', {
legacyCode: 'insufficient_funds',
});
// Holds only explain a shortfall in funds, not a request that
// asked for less than a token.
const fundsShort = !(
affordableOutputTokens(remainingCredits) >= 1
);
throw fundsRefusal(
fundsShort && affordableOutputTokens(balance) >= 1,
);
}
args.max_tokens = cap;
// Holds alone shrinking the cap is a busy account, not a broke one.
if (cap < limit && affordableOutputTokens(balance) < limit) {
fundsCap = cap;
}
} else {
// No output price, nothing to bound — but a previous attempt may
// have written its cap here; give this one the user's own value.
@@ -1013,13 +1109,23 @@ export class ChatCompletionDriver extends PuterDriver {
// What this attempt can cost at worst: the prompt, plus output run to
// the cap just set. Capped output is what makes the number finite —
// for a model with no output price the output term is zero and the
// prompt estimate stands alone.
// prompt estimate stands alone. Already factored, so it's held through
// the plain service rather than scaled a second time.
const worstCaseCost =
approximateInputCost + (args.max_tokens ?? 0) * outputTokenCost;
return this.services.metering.reserveCredits(
const hold = await this.services.metering.reserveCredits(
actor,
Math.max(worstCaseCost, minimumCredits),
);
return { hold, fundsCap };
}
/** The key `model`'s usage is recorded and cost-factored under. */
#meteringModelKey(model: IChatModel): string {
return (
this.#providers[model.provider!]?.meteringModelKey?.(model.id) ??
`${model.provider}:${model.id}`
);
}
/**
@@ -1077,7 +1183,7 @@ export class ChatCompletionDriver extends PuterDriver {
[`estimated_${outputKey}`]: outputTokens,
},
actor,
`${model.provider}:${model.id}`,
this.#meteringModelKey(model),
{
// Undefined when the model has no cost table: the entry is
// recorded unpriced rather than free.
@@ -1285,15 +1391,6 @@ export class ChatCompletionDriver extends PuterDriver {
const deepseekKey = readKey(providers['deepseek']);
if (deepseekKey) {
deepSeekPricing.onRefreshError = () => {
this.clients.alarm.create(
'deepseek_pricing_refresh_failed',
'DeepSeek pricing refresh failed; using last verified rates',
{},
'warning',
{ dedup: true },
);
};
this.#providers['deepseek'] = new DeepSeekProvider(
{ apiKey: deepseekKey },
metering,
@@ -135,7 +135,10 @@ export class FakeChatProvider implements IChatProvider {
inputTokens += Math.ceil(message.content.length / 4);
} else if (Array.isArray(message.content)) {
for (const content of message.content) {
if (content.type === 'text') {
if (
content.type === 'text' &&
typeof content.text === 'string'
) {
inputTokens += Math.ceil(
content.text.length / 4,
);
@@ -20,7 +20,7 @@
/**
* Integration test for the Alibaba provider.
*
* Uses `qwen-turbo` (cheapest generally-available model). Skipped
* Uses `qwen-flash` (cheapest generally-available model). Skipped
* when `PUTER_TEST_AI_ALIBABA_API_KEY` is unset.
*/
@@ -39,7 +39,7 @@ const ENV_VAR = 'PUTER_TEST_AI_ALIBABA_API_KEY';
describe.skipIf(skipUnlessEnv(ENV_VAR))(
'AlibabaProvider (integration)',
() => {
it('returns a non-empty completion from qwen-turbo', { timeout: INTEGRATION_TEST_TIMEOUT_MS }, async () => {
it('returns a non-empty completion from qwen-flash', { timeout: INTEGRATION_TEST_TIMEOUT_MS }, async () => {
const provider = new AlibabaProvider(
{ apiKey: optionalEnv(ENV_VAR)! },
makeMeteringStub(),
@@ -47,7 +47,7 @@ describe.skipIf(skipUnlessEnv(ENV_VAR))(
const result = await withTestActor(() =>
provider.complete({
model: 'qwen-turbo',
model: 'qwen-flash',
messages: [
{ role: 'user', content: 'Say hi in one word.' },
],
@@ -178,6 +178,34 @@ describe('AlibabaProvider model catalog', () => {
expect(ids).toContain('qwen-max');
expect(ids).toContain('qwen/qwen-max');
});
it.each([
['qwen3.8-flash', 15, 47, 1.6],
['qwen3.8-omni-flash', 15, 47, 1.6],
['qwen3.8-max-0902', 200, 600, 25],
['qwen3.7-max-2026-05-17', 250, 750, 0],
['deepseek-v4.1-flash', 30, 120, 0],
])('exposes %s at its list price', (id, input, output, cached) => {
const model = ALIBABA_MODELS.find((m) => m.id === id);
expect(model?.costs).toMatchObject({
prompt_tokens: input,
completion_tokens: output,
cached_tokens: cached,
});
});
it.each([
'qwen3-max',
'qwen3.6-max-preview',
'qwen-turbo',
'qwen3-vl-flash',
'qwen3-coder-plus',
'qwen-omni-turbo',
'deepseek-v4-flash',
])('drops %s, which Model Studio is retiring', async (id) => {
const { provider } = makeProvider();
expect(await provider.list()).not.toContain(id);
});
});
// ── Request shape ───────────────────────────────────────────────────
@@ -552,13 +580,13 @@ describe('AlibabaProvider.complete non-stream output', () => {
await withTestActor(() =>
provider.complete({
model: 'qwen3.6-max-preview',
model: 'qwen3.8-max',
messages: [{ role: 'user', content: 'hi' }],
}),
);
const model = ALIBABA_MODELS.find(
(m) => m.id === 'qwen3.6-max-preview',
(m) => m.id === 'qwen3.8-max',
)!;
const [usage, , prefix, overrides] = recordSpy.mock.calls[0]!;
expect(usage).toEqual({
@@ -566,7 +594,7 @@ describe('AlibabaProvider.complete non-stream output', () => {
completion_tokens: 20,
cached_tokens: 15,
});
expect(prefix).toBe('alibaba:qwen3.6-max-preview');
expect(prefix).toBe('alibaba:qwen3.8-max');
expect(overrides).toEqual({
prompt_tokens: 50 * Number(model.costs.prompt_tokens),
completion_tokens: 20 * Number(model.costs.completion_tokens),
Loaded 100 of 569 files, more files were not shown because too many files have changed in this diff. Show more