Compare commits

...
Author SHA1 Message Date
Nariman JelvehandClaude Opus 5.5 c3275b1514 fix(puter-js): apps launched from desktop.puter.com take its messages
In an app, puter.js trusted only https://puter.com as the GUI origin, so
every desktop message from desktop.puter.com failed the origin check and
was dropped. The desktop waits for the app's answer to windowWillClose,
so app windows there never closed; itemsOpened, drag-and-drop, focus,
theme and locale changes were lost the same way, and a reauth request
was posted to the wrong origin.

An app now takes its GUI origin from the launch URL's `puter.origin`,
but only when that is one of Puter's own desktops (exactly
https://puter.com or https://desktop.puter.com); anything else keeps
https://puter.com, so a page that frames an app still can't drive it.
An explicit PUTER_ORIGIN still wins, and a top-level page never reads it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-13 20:43:52 -07:00
Daniel Salazar b2eb8ed895 feat(auth): a first-party page served on its own origin signs in with a session of its own (#4635)
A deployment can list origins of its own pages in
`first_party_session_origins`. When such a page asks the GUI to sign it in,
the sign-in hands it a GUI token on a `web` session of its own, parented to
the user's sign-in, instead of an app token.

The session is minted by POST /auth/first-party-session, which only the GUI's
own page script may call, and only as the user's sign-in itself: app tokens,
access tokens and a session minted this way are refused. It never gets a
session cookie, ends with the sign-in it was handed out under (revoking either
takes the tokens minted under the page along), is not counted as a sign-in,
and stays up with its sign-in when all other sessions are revoked.

The sessions list shows it under its sign-in, titled with the page's host and
badged as a site session.
2026-11-13 17:47:57 -07:00
Daniel Salazar d1cbfb9fb1 fix(puter-js): a godmode app opened in its own tab stays signed in across a reload (#4632)
A tab the desktop opened for an app boots the SDK as a top-level page, where
a session-type token was kept in memory only. Once the desktop hands such a
tab its launch token, the SDK now keeps it in sessionStorage, as a framed app
does, and a reload of that tab reads it back before any stored app token.

A page that wasn't handed a token this way still persists no session token.
2026-11-13 15:46:17 -07:00
Neal ShahandClaude Opus 5.5 ac44de819d fix(hosting): choose the site and subdomain from Host, not X-Forwarded-Host (#4459)
With trust proxy on, express derives req.hostname and req.subdomains from
X-Forwarded-Host, which any client can send, while host validation reads the
raw Host. A request could serve one hosted site's files on the main domain,
or reach the API on a hosting host. The header is now dropped before routing.

Fixes PUT-2272

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-13 14:56:16 -05:00
Daniel Salazar 4ad53899da feat(auth): debug app sessions that act as a user in one app (#4629)
Adds a `debug_app` session kind and AuthService.createDebugAppToken, which
mints an app-under-user token on its own session row instead of the user's
app session. The row has a hard expiry that activity never slides, its calls
record no user activity, it is hidden from the user's session list, and a
password reset revokes it. SessionStore.deleteExpiredDebugAppSessions purges
expired rows. Migrations add the kind to the sessions constraint on sqlite,
mysql and postgres.
2026-11-13 11:19:08 -07:00
Daniel Salazar c253b7e05e feat(auth): signups record the request's TLS fingerprint when a fronting proxy supplies one (#4622)
`tlsFingerprintHeader` names a request header that a TLS-terminating proxy
sets to the client's TLS fingerprint. When set, the value is exposed as
`req.tlsFingerprint`, stored on the new `user.signup_tls_fingerprint`
column, passed to `puter.signup.validate`, and forwarded to the phone
verification provider when it is a JA4.
2026-11-13 10:41:57 -07:00
647995133e fix(http): responses stay under Node's close-listener limit, starting with WebDAV GET (#4621)
* fix(webdav): GET pipes the file without stacking close listeners on the response

stream.pipeline put seven close listeners on the response. With metering,
compression, the concurrency gate and request tracing on top, every WebDAV
download crossed Node's ten-listener warning. GET now uses pipeToResponse,
like the other file download routes.

* fix(drivers): a streamed result stops its source on unpipe instead of its own close listener

pipe already unpipes when the response finishes, closes or fails, so the
extra close listener only spent part of the response's listener budget. The
routes that hold a response open now have a test that keeps each within that
budget.

* refactor(http): everything that waits on a response's close shares one listener

Egress metering, the concurrency gate, the large-body budget, route
lifecycle events, driver call slots, disconnect aborts and SSE writers each
put their own close listener on the response. They now register callbacks
with onResponseClose, which keeps one listener per response and runs them in
the order added, so a streamed response holds three close listeners instead
of five or six and request tracing has room for its own.

* test(webdav): handle streamed response errors in test sink

Co-authored-by: Salazareo <[email protected]>

* test(webdav): a storage failure mid-download is covered against a real response

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: Salazareo <[email protected]>
2026-11-13 09:56:21 -07:00
Reynaldi Chernando 9d3a049988 document puter.host behavior (#4606)
* document puter.host behavior

* update
2026-11-13 19:13:59 +07:00
Juan Fernando CastroandDaniel Salazar 1c5909331c fix(fs): a path lookup of an old row returns its healed path while the replica catches up (#4612)
* fix(fs): a healed entry carries its written path, and an entry with no path answers 404

The lineage heal now returns the entry with the path it just wrote
instead of re-reading the row, so a replica that hasn't caught up can't
hand back the old NULL path. Entries that still have no path are kept
out of the cache, resolve to the usual 404 in the fs routes, and an ACL
check on a resource with no path refuses instead of throwing.

* fix(fs): heal paths along one parent chain, fill only empty paths, and return the healed row from the primary

The path lineage now picks the target row and walks up its own
parent_uid links, so each row in a healed path is the parent of the
next. The backfill only fills a NULL path and never replaces one that
is set. Both lineage routes return the row read back from the primary,
so every column is current. Path-less entries are kept out of the
cache by the cache writers themselves, and a missing or empty path is
unresolvable both in resolveOrNotFound and in ACLService.check.

* fix(fs): a path lineage resolves only to a row still at that path, and the backfill runs its updates concurrently

The lineage lookup now answers a miss when the row read back from the
primary sits at a different path than the one walked, and doesn't cache
it. The backfill runs its per-row updates through
runWithConcurrencyLimit and drops cached entries only for rows it
actually filled. ACLService.check refuses only a resource with no
string path, so an empty client path keeps its existing 400.

* fix(fs): build healed paths from the primary's ancestor chain, and invalidate the paths the heal wrote

Both lineage routes now read the ancestor chain from the primary and
write the path it spells, so a lagging replica's names never end up in
a path column. A path lookup whose replica walk no longer matches the
primary's chain is a miss and writes nothing. The backfill drops the
cached entries of the rows it filled in one call, keyed by the path it
wrote. The lazy-chain comparison now lets a missing path reach
ACLService.check on both sides.

* fix(fs): leave path-less uid and id lookups to resolveNode

resolveNode already answers a uid or id lookup whose entry has no path
as not found, and a path lookup always returns an entry with its path,
so resolveOrNotFound goes back to trusting what it resolves. The
readdir case for a broken parent chain is covered by the existing HTTP
test.

* fix(fs): the ancestor walk stops at a fixed depth and updates only rows with an empty path

The primary's ancestor walk now selects each row's path and hands only
the rows whose path is NULL to the backfill, so filled ancestors cost no
UPDATE. The walk stops after LINEAGE_MAX_DEPTH steps; a chain that has
not reached a row without a parent by then is treated as broken and the
row is left alone.

* fix(fs): a listing heals its path-less entries a bounded number at a time

* test(fs): a stored path the parent chain no longer spells heals from the primary, and a listing leaves a path-less entry uncached

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-11-12 22:24:37 -07:00
Juan Fernando CastroandDaniel Salazar 5d541cd8e2 fix(auth): a just-minted read URL or token is found while the replica catches up (#4556)
* fix(auth): a just-minted read URL or token is found while the replica catches up

* fix(auth): a token's grants aren't cached when it's revoked mid-mint

* fix(permissions): a mint succeeds when its grants cannot be read back to cache

The grants are already stored at that point, so a failed read-back drops
the cache key and the next read fills it.

* test(auth): assert the grant list cached at mint; skip the sqlite-copy replica test on postgres

* fix(auth): test that a token row is read from the primary only when both replica lookups miss

* fix(auth): one-line comments on the replica-then-primary token row lookup

* perf(auth): a mint caches the grants it stored without reading them back

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-11-12 22:22:42 -07:00
Juan Fernando CastroandDaniel Salazar fb526bfb7e perf(share): skip the ancestor read for owners with nothing shared in their own home (#4597)
* fix(share): look up a cross-owner move's origin by its former owner

The move listener now passes the event's fromUserId down to the origin
side of the fan-out, so the shares at the former location are looked up
under the owner they belong to, and re-marks the new owner as shared
before the fan-out runs instead of after it starts.

Neither changes what recipients hear today: the origin's ancestors
already carry the former owner, and the fan-out reads ancestors before
it consults the owner-shares cache. Both are needed once the fan-out
checks the entry's owner before reading ancestors. Tests cover a real
cross-owner move with a cold owner-shares cache, and a hand-over to an
owner with nothing shared.

* perf(share): skip the ancestor read when the entry's owner shares nothing

Every fs write, create, rename and move resolved the entry's ancestors
(a cache batch) before asking whether any owner on that chain had
anything shared. Ask about the entry's owner first, plus the former
owner for the origin side of a cross-owner move, and return no audience
without reading ancestors when none of them has a share. Otherwise the
lookup runs as before.

Self-contained so it can be reverted on its own. It assumes every entry
in a shared subtree carries the subtree owner's user id. Entries that
older writers created under someone else's shared folder with the
writer's own id would stop notifying that folder's recipients when the
writer has nothing shared, so this only holds once existing data has no
such mixed-owner subtrees.

* fix(share): read ancestors whenever an entry sits outside its owner's home

The early return in the share fan-out assumed every entry in a subtree
carries the subtree owner's id. Rows from older writers don't always:
an app writing into another user's AppData stamped itself as the owner,
so an entry in a shared folder can name an owner with nothing shared.

Skip the ancestor read only when the entry's owner has nothing shared
and the path sits in that owner's own home directory; anything under
another user's home takes the full lookup as before, whatever the data
holds. The owner's username comes from a per-process map (60s TTL,
10k entries) backed by the user store's cache, so a hit costs nothing
and a miss one user cache read, against the ancestor batch it replaces.

* refactor(share): drop the former-owner hint from the move fan-out

With the home-directory check, the origin side of a cross-owner move
sits under the former owner's home, not the entry's new owner's, so it
always takes the full ancestor lookup and finds the former owner's
shares through the chain. Passing fromUserId down and re-marking the new
owner before the fan-out no longer change anything, so both go back to
how they were. The test for a hand-over to an owner with nothing shared
stays; the slower duplicate that waited out the cache is removed.

* perf(share): keep the chain owner check after the ancestor read

The early return replaced the check that ran after the ancestor read,
so an entry outside its owner's home went on to the share-table lookup
even when nobody on its chain had anything shared. Ask about every owner
on the chain again once the ancestors are in hand, as before.

* fix(share): mark the new owner shared before a hand-over's fan-out

The fan-out now reads the owner-shares cache before it reads ancestors.
On a move into another user's home, a "nothing shared" answer cached
for the new owner shortly before the move made the destination side
skip the entry's re-pointed share, so its recipient was told the item
was removed rather than moved. Mark the new owner shared before the
fan-out starts whenever the move changed the owner.

The earlier "drop the former-owner hint" commit said this ordering no
longer mattered; it does for the destination side.

* perf(share): cache missing usernames and share in-flight lookups

The home check now uses isOwnHomePath with the owner's username. The
per-process username map also keeps a null answer for an owner with no
account or no username, and a burst of writes by one owner shares a
single user lookup, as the owner-shares lookups already do. A failed
lookup is not cached and reads as someone else's home, so the full
ancestor lookup runs. Tests cover an owner with no account, a home that
differs only in case and a renamed owner, each taking the full lookup.

* test(share): run the hand-over's revoke for real and assert the destination read

* perf(share): check the home path before asking whether the owner shares

An entry outside its owner's home goes straight to the full read, which asks about every owner on the chain together. The username cache is read once per call and the single-caller home helper is inlined.

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-11-12 22:22:26 -07:00
Thibaud-VdbandDaniel Salazar 1abd25eb31 ci: take release-please from its current home, scope three workflow tokens and lock the actions (#4521)
* ci: take release-please from its current home

google-github-actions/release-please-action is archived: the action
moved to googleapis/release-please-action, same code, same inputs,
same major version, and that is where the fixes now land. The
release-please workflow references the maintained repository.

* ci: declare what the three unscoped workflows may do with the token

Docs Build, Maintain Release Merge PR and Notify HeyPuter had no
permissions block, so they ran with the repository's default token
scope. Docs Build only checks out and builds, Notify HeyPuter only
calls the dispatch endpoint with its own token: both declare
contents: read. Maintain Release Merge PR pushes the
auto-update/release branch and opens the release pull request, so it
declares contents: write and pull-requests: write. The other
workflows already declared their scopes and are left as they are.

* ci: stop persisting the checkout credentials in the jobs that never push

actions/checkout keeps the job token in the clone's git config, where
every later step can read it. Twelve checkouts now set
persist-credentials: false: the backend and puter.js test jobs, the
coverage jobs, the type checks, the docs build, the AI provider tests
and the Docker image build read the clone and push nothing with git.
The two checkouts of Maintain Release Merge PR keep the token: that
job resets the release branch and pushes it through
create-pull-request.

* ci: lock the workflow actions to the commits they resolve to

GitHub's workflow dependency lockfile holds the commit each action
reference resolves to, transitive actions included, and the runner uses
those commits even if a tag moves later. .github/workflows/actions.lock
comes from gh actions-lock (v0.4.1), which also narrowed every reference
to the exact release it locked, so Dependabot keeps bumping versions and
its security alerts keep matching. Re-run the same command after adding
or bumping an action.

* ci: drop the unused token scope from Notify HeyPuter, match workflow indentation

* ci: remove the disabled release workflows

release-please and Maintain Release Merge PR have been disabled since 2024; releases are cut from calver tags. Their entries leave the lockfile with them.

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-11-13 03:59:06 +00:00
Daniel Salazar b5e4f20087 fix: KV size and credit edge cases, a metering double count, site name caching, /token-read content type (#4618)
* fix(fs): /token-read serves a file with no known extension as application/octet-stream

* fix(kv): a value too large once stored rejects with value_too_large, and a consistent read of several keys is consistent

* fix(kv): remove on an account with no credits left removes the paths and returns null

* fix(metering): a buffered increment whose base read fails is not also written through

* fix(hosting): site lookups by name share one cache entry for every spelling

* fix(metering): an increment whose base read fails is written through and taken back out of the buffer

* fix(hosting): create stores a site name trimmed and lowercased
2026-11-12 20:49:51 -07:00
Reynaldi Chernando bdb88507df throw error for get usage APIs (#4608) 2026-11-12 20:47:03 -07:00
Juan Fernando Castro 0694b76751 feat(email): sends name a stream, and streams add headers, a from address and message tags from config (#4520)
* feat(email): sends name a stream, and streams add headers, a from address and message tags from config

`email.streams` maps a stream name to optional headers, a from address and a
tags header. A send that names a configured stream gets that stream's
headers (caller headers win) and from address (when the caller gives none),
and its tags rendered as `name=value` pairs into the tags header. Unknown or
missing streams change nothing, and `streams` never reaches the transport.

Account mail is sent on `auth`, share and app feedback mail on `notify`, and
the signup confirmation on `signup` unless a `puter.signup.validate` listener
names another stream, which is remembered in the account metadata for
resends, deferred sends and save-account.

The signup response carries `confirmation_email_sent` when a confirmation
email was due at signup, and `/send-confirm-email` answers `{ sent }`.

PUT-2346

* fix(email): review fixes for stream naming

* fix(email): contact form mail names the notify stream, streams resolve from own config entries, the no-transport log shows the stream

Templated sends take only replyTo and stream; the header and tag options had no caller.

* fix(signup): a claimed row drops its earlier confirmation stream, and save-account reports a failed send under its own stage
2026-11-12 22:46:34 -05:00
Juan Fernando Castro e7ee778985 perf(notifications): notify and mark-ack skip their primary read-backs (#4547)
* perf(notifications): notify returns the row it wrote instead of reading it back

The service hands the driver the row the store built on insert, and a
group replacement builds the row it leaves from the one it looked up, so
neither path reads the row back from the primary.

* perf(notifications): mark-ack reads the row's fixed columns from a replica

The type, app uid and audience the dismissal event carries never change,
so they are read from a replica alongside the update, and from the
primary only when the replica has no row yet.

* test(notifications): cover a repeat dismissal and tidy notify docs

* fix(notifications): a repeat mark-ack returns without waiting for the replica read, and a failed replica read is logged

* refactor(notifications): NotificationRow spells out the row's columns, and a replacement passes the app uid it already has
2026-11-12 22:46:25 -05:00
Juan Fernando Castro c3ff8ca3b2 fix(permissions): an app opened on a file in a shared folder can read and save it (#4555)
* fix(permissions): an app opened on a file in a shared folder can read and save it

* perf(permissions): app scans check implied options before scanning the user

* test(permissions): cover app grants on folders and revoked shares

* fix(permissions): an app's folder grant counts only while its user holds that folder

The app branch of the ACL check counted a user-to-app grant on any folder above the file. It now counts a grant on the file itself, or on a folder above it that the user holds at the asked mode, so the grant and the user's own access meet on the same node.

* test(fs): the other-app case pins 404 beside the opened app's 200
2026-11-12 22:46:15 -05:00
Juan Fernando Castro 39d931306d fix(auth): sign-in relays a token set between waits (#4572)
* fix(auth): sign-in relays a token set between waits

* fix(auth): sign-in relay skips another app's stored token and survives a failed store

* fix(auth): sign-in relay always clears the stored copy and logs only the error message

* fix(auth): the sign-in relay keeps only signed app tokens, per app, and a subscribed wait answers without Redis

/login/set stores a token only when this server signed it for an app, under a
key that includes the app, so a later set for another app or an unsigned value
leaves a pending token in place. /login/wait takes the stored copy once and
falls back to listening when Redis cannot be read. A token that arrives by
event is delivered to every subscribed wait for that app; the stored copy is
then removed best-effort, and left in place when the wait's client has gone.
A relayed token for another app no longer ends the wait. The event payload is
back to { authtoken }.

* chore(events): event types file matches main again
2026-11-12 22:46:04 -05:00
Juan Fernando Castro 70e823c5af feat(email): notification emails ask listeners before each recipient (#4596)
* feat(email): notification emails ask listeners before each recipient

Share notification emails (account holders and invites) and the app
feedback email to the owner now emit `email.notify.validate` with
`{ to, kind, allow }` once per recipient address, next to the existing
unsubscribe and blocked-domain checks. A listener that sets `allow` to
false skips that recipient's email; the others are sent as before. The
invite check runs before its send budget is claimed and the feedback
check before the per-app cap slot, so a skipped address uses neither.
When no listener answers or the hook fails, the email is sent.

* refactor(email): type the notify kind as a template name, drop the dead catch

`emitAndWait` already catches a listener that throws, so the try/catch
around it never ran; a failed listener leaves `allow` true and the email
is sent. `kind` is now `EmailTemplateName`, so renaming a template breaks
the type check instead of the listener.

* fix(share): ask email.notify.validate when the digest is sent

A digest entry can wait out its window or a retried send, so the
listener is asked about the recipient right before the mail goes.
Invites are still asked once more before their budget is claimed.

* fix(email): any falsy allow on email.notify.validate skips the mail

Matches how email.validate reads its answer. The helper also takes the
event client as required, since every caller has one.

* feat(email): email.notify.validate carries every recipient of one kind of mail

The payload is { kind, recipients: [{ to, allow }] }. Share invites ask
once for all their addresses before any budget is claimed; the digest
send and the feedback mail ask with one entry. kind is narrowed to the
three templates that ask.
2026-11-12 22:45:56 -05:00
dependabot[bot] 1bc1953344 chore(deps): bump ws from 8.22.0 to 8.22.2 (#4567)
Bumps [ws](https://github.com/websockets/ws) from 8.22.0 to 8.22.2.
- [Release notes](https://github.com/websockets/ws/releases)
- [Commits](https://github.com/websockets/ws/compare/8.22.0...8.22.2)

---
updated-dependencies:
- dependency-name: ws
  dependency-version: 8.22.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-12 20:35:51 -07:00
dependabot[bot] 0f83bb9b56 chore(deps): bump nodemailer from 10.1.4 to 10.1.7 (#4478)
Bumps [nodemailer](https://github.com/nodemailer/nodemailer) from 10.1.4 to 10.1.7.
- [Release notes](https://github.com/nodemailer/nodemailer/releases)
- [Changelog](https://github.com/nodemailer/nodemailer/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodemailer/nodemailer/compare/v10.1.4...v10.1.7)

---
updated-dependencies:
- dependency-name: nodemailer
  dependency-version: 10.1.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-12 20:15:05 -07:00
dependabot[bot] 1ae95d97e8 build(deps-dev): bump @babel/core from 7.29.7 to 7.29.13 (#4565)
Bumps [@babel/core](https://github.com/babel/babel/tree/HEAD/packages/babel-core) from 7.29.7 to 7.29.13.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.29.13/packages/babel-core)

---
updated-dependencies:
- dependency-name: "@babel/core"
  dependency-version: 7.29.13
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-12 20:13:34 -07:00
dependabot[bot] 31f8936da2 build(deps): bump undici from 7.30.1 to 7.30.2 (#4566)
Bumps [undici](https://github.com/nodejs/undici) from 7.30.1 to 7.30.2.
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v7.30.1...v7.30.2)

---
updated-dependencies:
- dependency-name: undici
  dependency-version: 7.30.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-12 20:12:08 -07:00
dependabot[bot] aab559c9c1 chore(deps): bump zod from 4.4.3 to 4.6.7 (#4465)
Bumps [zod](https://github.com/colinhacks/zod) from 4.4.3 to 4.6.7.
- [Release notes](https://github.com/colinhacks/zod/releases)
- [Commits](https://github.com/colinhacks/zod/compare/v4.4.3...v4.6.7)

---
updated-dependencies:
- dependency-name: zod
  dependency-version: 4.6.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-12 20:08:45 -07:00
dependabot[bot] 7f1d46028a chore(deps): bump fastify from 5.12.5 to 5.12.7 (#4464)
Bumps [fastify](https://github.com/fastify/fastify) from 5.12.5 to 5.12.7.
- [Release notes](https://github.com/fastify/fastify/releases)
- [Commits](https://github.com/fastify/fastify/compare/v5.12.5...v5.12.7)

---
updated-dependencies:
- dependency-name: fastify
  dependency-version: 5.12.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-12 20:08:27 -07:00
dependabot[bot] 39d9482409 chore(deps): bump body-parser from 2.3.0 to 2.4.0 (#4463)
Bumps [body-parser](https://github.com/expressjs/body-parser) from 2.3.0 to 2.4.0.
- [Release notes](https://github.com/expressjs/body-parser/releases)
- [Changelog](https://github.com/expressjs/body-parser/blob/master/HISTORY.md)
- [Commits](https://github.com/expressjs/body-parser/compare/v2.3.0...v2.4.0)

---
updated-dependencies:
- dependency-name: body-parser
  dependency-version: 2.4.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-12 20:08:06 -07:00
Juan Fernando CastroandDaniel Salazar 4a958b3ca1 fix(gui): desktop push status counts only Puter's devices and follows subscription changes (#4550)
* fix(gui): desktop push status counts only Puter's devices and follows subscription changes

- fix(gui): desktop push status counts only Puter's own devices
- fix(gui): the desktop keeps its push subscription when the browser replaces it
- docs(gui): shorten push comments to one line
- fix(gui): removed browsers stay off after a subscription change, and websites can reach Puter push
- docs(puter-js): fit the desktop push request comment on one line
- fix(gui): background push registration checks every earlier endpoint and waits out a 429
- fix(puter-js): inside the desktop, requestPermission leaves a held push grant to the desktop
- fix(gui): a browser that stays off after being removed forgets its receipt
- fix(gui): push status holds right after enable(), and a subscription change is picked up after a run in flight
- fix(gui): an app's push grant counts whenever the app list can't be read

* fix(puter.js): a held push grant is left to the desktop only when it says it answers push

The desktop lists `push-permission` in `puter.gui_features` on the app
iframe's URL. Without it the SDK settles a held `notifications:push` from
its own check, so a desktop whose requestPermission handler has no held
check shows no dialog for a grant the app already holds.

* fix(gui): a held push grant isn't asked again beside other access, and never counts without an app uid

requestPermission drops a push grant held for Puter's devices from the
dialog and asks only for what is missing. The app list lookup answers no
for a missing app uid, whatever the list says; an app the list leaves out
still keeps its grant, since the list is capped and can trail a grant.

* test(puter.js): e2e for a push grant the app already holds inside the desktop

A held grant resolves true with no dialog, and a request for it beside
access the app is missing shows only the missing access.

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-11-12 20:06:23 -07:00
Juan Fernando Castro 4653ff6dae fix(gui): a removed browser stays off, and launches reuse the page's push status (#4549)
* fix(gui): a browser removed from push stays off until its user turns it on

The silent registration on load and app launch no longer replaces a
subscription the user removed in Puter. When the server answers
push_device_removed, the browser's subscription and its receipt are
dropped and the browser is recorded as turned off, so later loads leave
it alone. Turning push on from Settings or a push dialog still replaces
the removed subscription.

* perf(gui): launches reuse the page's last push status

The built-in push client keeps the last registered status it read in
memory, for the same user and subscription. App launches and an app's
puter.notifications.status() answer from it instead of listing devices
again. It is dropped on enable(), disable(), an unregistered answer and
the worker's subscription-changed message, and the Notifications tab
and push dialogs still read fresh, so a new page load always asks the
server.

* fix(notifications): a background enable opens no sign-in, and a page keeps its last push on for five minutes

_enableWebPush takes { background } in place of { renewRemoved }: a call
without a click leaves a removed browser removed and sends its requests
without interactive sign-in.

The desktop's push client keeps the on that enable() read, so the next
cached status lists no devices, and asks the server again once that on is
five minutes old.

* fix(gui): the kept push on belongs to the user whose devices were listed, and a failed enable() keeps none

A status read or enable() that another user's sign-in overlapped keeps no
on, and an enable() that fails drops what a read begun meanwhile kept.
2026-11-12 20:06:23 -07:00
Juan Fernando Castro a5c7b427b8 fix(puter-js): a confirmed push subscription registers for another account without a new confirmation (#4548)
* fix(puter-js): a confirmed push subscription registers for another account without a new confirmation

The server's verify and active subscribe answers can carry a receipt for
the subscription. It is kept per worker scope and sent with the next
announce of the same endpoint, so the Puter desktop registers the same
browser for the next signed-in account without another confirmation push.
The receipt is forgotten when the subscription is ended, replaced or
removed, and when the app is turned off in Puter. A server without
receipts behaves as before.

* docs(puter-js): shorten a push helper comment

* fix(notifications): keep receipts only in the Puter GUI and store the new one as soon as the server answers
2026-11-12 20:06:22 -07:00
Daniel Salazar cd3600bae9 fix(fs): overwrite a file in place when a write spells its name in another case or normalization form (#4620)
* fix(fs): overwrite a file in place when a write spells its name in another case or normalization form

* docs(fs): say a write needs overwrite off to get a new name beside a look-alike one
2026-11-12 20:02:39 -07:00
Juan Fernando Castro 2db55bd636 perf(share): read share owners from the entry owner column and backfill missing ones (#4598)
* perf(share): skip the legacy owner probe once a recorded share answers

ownerHasShares ran its recorded-owner and legacy NULL-owner queries side
by side, so the join to fsentries ran even when the first query already
found a share. Run the legacy half only when the recorded half finds
nothing; the answers stay the same.

* perf(share): drop the NULL-owner fallbacks from owner and outbound reads

ownerHasShares, listOutbound, countOutbound, the outbound apps summary
and the team-row filter behind listByHolder/countByHolder each carried a
second half for share rows written before entry_owner_user_id existed,
matched by joining fsentries on a NULL owner. The column migration
backfilled those rows from fsentries, so the joins only add work: each
costs about min(owner's files, NULL-owner rows) per call.

Every remaining read goes through entry_owner_user_id alone. Invite rows
(no fsentry, no owner) are matched as before: they were never reachable
through the joins.

Self-contained so it can be reverted on its own. It assumes existing
data has no share row that names an fsentry but has a NULL
entry_owner_user_id; if any remain, they stop showing up in these reads.

* fix(share): backfill the entry owner on share rows written without it

The share reads now match on entry_owner_user_id alone. Add a migration
for each engine that fills that column from fsentries on every row that
names an entry but still has a NULL owner, so the backfill runs before
the reads stop looking at NULL-owner rows and their answers don't depend
on what existing data holds. Rows that name no entry (invites) keep a
NULL owner, and a replay finds nothing to fill.

* fix(share): record the entry owner when a claim names its entry

applyPending can point a claimed invite at an entry, but left
entry_owner_user_id NULL, so the row was missing from the owner's reads
that now match on that column alone. Fill it from fsentries with the
same subquery the upserts use. The share service claims without an
entry today; the parameter stays for callers that pass one.

* refactor(share): read the app from the data column directly

Every caller of the app filter and the app expression passes the share
row's own data column now, so the column argument goes. Also names the
owner probe test after what it checks and puts back an unrelated line
the earlier formatting pass had reflowed.

* chore(share): drop the redundant MySQL and Postgres owner backfills

MySQL and Postgres replay every migration file on boot, and mysql_mig_47
and postgres_mig_36 already end with the same owner backfill, so they
re-run it on every boot. Only SQLite, which tracks a schema version and
runs each file once, needs its own backfill migration (0103).

* fix(share): list a team row for its members when no entry owner is recorded

* refactor(share): claiming an invite leaves its entry and owner as written

* test(db): run the share owner backfill against the full prior schema

* fix(share): ship the owner backfill as its own MySQL and Postgres migration

The backfill at the end of mysql_mig_47 and postgres_mig_36 replays on boot
only when database.migrationPaths is set. A new file gives installs that apply
migrations another way a step to run; a replay finds nothing to fill.
2026-11-12 18:06:29 -07:00
Daniel Salazar 36e2d199c8 fix(fs): refuse instead of replacing an entry whose name differs only by an emoji or accent (#4589)
* fix(fs): refuse instead of replacing an entry whose name differs only by an emoji or accent

MySQL compares fsentries names and paths under collations that also fold
accents and most emoji, so a lookup for one name could return a sibling
with another, and writes then overwrote and renamed it.

Path lookups now return only an entry whose stored path matches the
requested one after NFC normalization and lower-casing; case-only and
NFC/NFD spellings still resolve as before. Name-collision checks opt in
to the folded match and treat it as taken by a different entry: write,
mkdir, touch, rename, shortcut, move and copy dedupe when asked and
otherwise answer 409, and never replace or return that entry.

* fix(fs): leave a sibling's subtree alone when a folder its path folds onto is renamed or moved

The subtree rewrite matches descendants with `path LIKE 'old/%'`, and LIKE
compares under the path collation (ASCII case on SQLite). With a sibling
folder whose path differs only that way, renaming or moving one folder also
rewrote the paths below the other.

When another row sits at such a path, the rewrite now follows parent links
down from the moved folder instead. Without one, the rewrite is unchanged,
after one indexed lookup.

* fix(fs): resolve an entry whose stored path is spelled unlike its names, and heal that path on the first read

Older rows can store a `path` whose spelling differs from the names on their
parent chain in ways the path collation ignores. A lookup that finds such a row
by its real spelling now confirms it through the parent chain, writes the path
from the stored names, and returns it, so overwriting, touching or copying over
it by name works and the next read takes the single indexed query again.

The lineage walk now writes a row's path only when it differs byte for byte
from the one its names give, and drops only those rows' cache keys, so a walk
over rows that are already right issues no writes. Whether an entry found at a
name is a different, folded name is decided by its own name and its path after
that walk, not by a stale path column. With `foldedNames`, a row only the path
collation matches still counts as holding the name.

* fix(fs): match entry names under full case mapping

Names are compared after upper- then lower-casing, with a dotted capital I
taken as I, so a request in another case finds the stored entry for Greek
final sigma (`οδος.txt` for `ΟΔΟΣ.txt`), German sharp s (`STRASSE.txt` for
`Straße.txt`) and Turkish dotted I (`istanbul.txt` for `İstanbul.txt`).
Accent, emoji, kana and width differences still make two names different.

* fix(fs): name the existing entry when a folded name is already taken, and show why a confirmed replace failed

`item_with_same_name_exists` from write, move and copy now carries the
existing entry's own name in `entry_name` when it differs from the requested
one only by accents or emoji, so the prompt names the file the folder shows.

In the GUI, a replace confirmed from that prompt that still fails is reported
with its error message and counts as not written, instead of rejecting out of
`handle_same_name_exists` and leaving the caller waiting.

* fix(webdav): answer MKCOL with 405 and COPY or MOVE with Overwrite: F with 412 on a folded name

The existence checks before MKCOL, COPY and MOVE now see an entry whose name
differs from the target only by accents or emoji, so those requests answer as
for an existing target instead of reaching the filesystem and coming back as
409.

* docs(fs): list the errors write, move and copy return when a name is taken

`puter.fs.write()` gets an Errors section with `item_with_same_name_exists`,
`cannot_overwrite_a_directory` and `conflict`, the last for a name that differs
from an existing one only by accents or emoji. The move and copy pages say how
they reject in that case.

* fix(fs): walk the parent chain on the primary when a path lookup reads hard

A lookup that asks for a try-hard read (the checks after a create or move hits
the unique key) now also runs its parent-chain walk on the primary, and falls
back to the primary for the entry it finds. Before, the walk read a replica, so
a name the replica had not seen yet came back as missing and the mkdir or move
surfaced the duplicate-key error as a 500 instead of a 409.

Lookups without a try-hard read are unchanged. A try-hard lookup issues the
same number of queries; its walk reads the primary instead of a replica.

* fix(fs): treat a folder only the collation matches as missing instead of naming a file in it

A name-collision lookup lets only the last segment of the path match a
differently spelled name. When a folder above it matches only under the
collation (`😁dir` for `😀dir`), the lookup reports no occupant, so a write,
mkdir or touch there fails as for a missing folder: 404, or with
`createMissingParents` a 409 that names the folder. Before, it reported the
file inside the other folder and answered 409 naming that file. A folder whose
stored path is spelled that way still counts as the same folder, so lookups
built from stored paths find what they did before.

A rename onto a name the unique key already holds now answers 409 when the
lookup before it missed the occupant, instead of a 500.

* fix(fs): name the existing entry when a rename's name is folded, and leave a copy onto itself out of the copy docs

A rename refused because the folder holds a name that differs only by accents
or emoji now names that entry in its 409, as write, move and copy do.

A copy onto a folded spelling of the source's own name answers as a copy onto
itself (400 with `overwrite`), so the copy page's paragraph on folded names
now covers only items other than the source.

* fix(fs): walk the parent chain on the primary only when a hard read found another spelling of the path
2026-11-12 17:26:22 -07:00
Daniel Salazar 8616b00f7b fix(gui): save a URL from showSaveFilePicker, and always answer the app when the save dialog closes (#4587)
* fix(gui): save a URL from showSaveFilePicker and always answer the picker

The save picker downloads an http(s) URL itself, without cookies or a
referrer, up to 100 MiB, and writes it where the user chose, in apps and on
websites. A failed download shows an error. In apps, a save dialog that
closes without saving (failed download, failed write, close button) answers
fileSaveCancelled once instead of leaving the call pending.

* fix(gui): answer the save picker with the save's real outcome when it closes mid-save

Closing the save dialog while a URL is downloading stops the download,
writes nothing and answers fileSaveCancelled, in apps and on websites. In
apps, closing it once a write has started leaves the answer to the write:
fileSaved if it lands, fileSaveCancelled if it fails, never both. A save
that throws shows an error and answers fileSaveCancelled instead of leaving
the dialog stuck, and a second Enter during a save is ignored.

On an https desktop only https URLs are accepted, with a message saying so,
since the browser blocks http downloads there.

* fix(gui): retry signing a saved file once, and say so when the app can't get it

The app save picker signs the written file for the app with one retry. If
both tries fail, the user is told the file was saved but couldn't be passed
back, and the app gets fileSaveCancelled. The picker's one-answer state
(answered, closed, saving, writing) lives in helpers/savePickerAnswer.js so
the website picker can share it.

* fix(gui): answer the website save picker once, with the save's real outcome

The website save picker runs one save at a time, so a second Enter is
ignored. Closing it while a URL downloads stops the download, writes nothing
and answers fileSaveCancelled; closing it once the write has started leaves
the answer to the write, so the page hears fileSaved or fileSaveCancelled,
never both. A failed download shows an error, answers fileSaveCancelled and
closes the picker, as in apps, and so does a save that throws. A saved file
that can't be signed is reported to the user and answered as cancelled.

* fix(gui): show apostrophes in the save picker's failure messages
2026-11-12 17:26:05 -07:00
Daniel Salazar 873303a7c4 fix(events): offer a queued delivery only to connections that run its handler (#4590)
* fix(events): offer a queued delivery only to connections that run its handler

A queue subscription with an events worker was sent to every connection of
the app. A connection without the handler declined it at once, which moved it
to the worker while the tab with the handler was still running it.

puter.js now sends `events.handle` with the persistent subscriptions a
connection runs a handler for, on connect and whenever that set changes. The
server puts the connection in a room for each queue row it holds, and offers
the row's deliveries only to that room. With nobody in it anywhere in the
region, the delivery goes to the worker at once. A peer region handed such a
delivery declines it straight back when none of its connections run the
handler. Rows without a worker are still sent to the whole app room.

* fix(events): probe the handler room only when the app has a connection in the region

A queued delivery for a pair with no connection here goes to the worker on
the same local lookup and presence read as before, with no request to the
other nodes.

* fix(events): probe a forwarded batch's handler rooms together, within one timeout

A region receiving a batch now looks up every handler room its queued
deliveries need before it sends any of them: each distinct room once, up to
16 at a time, all under one shared 500 ms deadline. A room not answered by
then counts as empty, so its delivery is declined home with `no_handler` and
never sent. Deliveries still go out in the batch's order, and a batch with no
such delivery probes nothing.

* fix(events): resend a handler announce refused for the call budget

puter.js now sends `events.handle` with an ack and a 10 second timeout. When
the server refuses it with `too_many_requests`, the client sends the current
set again after 10 seconds, until one goes through or the connection is
replaced. No answer, as from a server without the verb, and any other refusal
are left alone. A connection that never had a persistent handler sends no
announce; one that had sends an empty set when the last is removed.

The rate limits page now says that announces count toward the
acknowledgement budget.

* docs(events): say when a client starts getting first try at queued events

A client counts once the server has registered its handler, shortly after it
connects or calls `onPersistent()`. Before then, an event for a subscription
with an events worker goes to another client with the handler or to the
worker.

* fix(socket): settle a cluster room count on the first node that holds a socket

A count ask now hears each node's answer as it arrives: the first node holding
a socket answers it at once, every node saying none answers it once all have
said so, and a node that never answers only decides it when no answering node
holds one. The request sent is unchanged, so every node answers it as before.

* fix(events): look up handler connections in the region's record, not every node

Each connection that announces a handler is kept in a per-subscription
sorted set in the region's Redis, scored by renewal: added after it joins
the handler room, removed before it leaves or on disconnect, renewed every
minute while it runs one, and counted only for three minutes after its last
renewal. Handing out a queued delivery, and receiving a forwarded one, now
check this node's sockets and then that set with one read, instead of
asking every node in the region to count the room.

A connection the set names but that is gone gets the delivery sent to an
empty room, and the socket lease hands it to the worker as before.

* fix(events): try the next region with the app open after one declines for want of a handler

A queued delivery's region is picked by app presence, so the first region
tried can have the app open without the handler. Its no_handler decline now
hands the delivery to the next region with the app open that hasn't had it,
and the worker runs it once none is left. A lease that lapses still goes to
the worker.

A decline now counts only when it comes from the region the current attempt
went to (this region for a local attempt). The region goes into the entry in
the same write as the attempt's lease, so a second decline from a region
already tried can't take the next region's lease. An attempt made after a
decline stamps its lease in its own field, which a server that doesn't make
such attempts never reads, so it can't take that lease back either.
2026-11-12 17:25:54 -07:00
198 changed files with 14234 additions and 1502 deletions

No files matched your search

+106
View File
@@ -0,0 +1,106 @@
# This file is machine-generated by `gh actions-lock`.
# Do not edit by hand; run `gh actions-lock` to update.
# Docs: https://gh.io/actions-lockfile
version: 'v0.0.3'
workflows:
'.github/workflows/ai-provider-integration-tests.yaml':
- 'actions/[email protected]'
- 'actions/[email protected]'
'.github/workflows/backend-tests.yaml':
- 'actions/[email protected]'
- 'actions/[email protected]'
- 'actions/[email protected]'
- 'actions/[email protected]'
- 'davelosert/[email protected]'
- 'dorny/[email protected]'
'.github/workflows/dependabot-automerge.yml':
- 'dependabot/[email protected]'
'.github/workflows/docker-image.yaml':
- 'actions/[email protected]'
- 'docker/[email protected]'
- 'docker/[email protected]'
- 'docker/[email protected]'
- 'docker/[email protected]'
- 'docker/[email protected]'
- 'peter-evans/[email protected]'
'.github/workflows/docs-build.yml':
- 'actions/[email protected]'
- 'actions/[email protected]'
'.github/workflows/notify-dev.yaml': []
'.github/workflows/notify-prod.yaml': []
'.github/workflows/puterjs-tests.yaml':
- 'actions/[email protected]'
- 'actions/[email protected]'
- 'actions/[email protected]'
- 'actions/[email protected]'
- 'davelosert/[email protected]'
- 'dorny/[email protected]'
'.github/workflows/tag-alias.yaml':
- 'docker/[email protected]'
- 'docker/[email protected]'
dependencies:
'actions/[email protected]':
ref: 'v7.0.1'
commit: 'sha1-3d3c42e5aac5ba805825da76410c181273ba90b1'
owner_id: 44036562
repo_id: 197814629
'actions/[email protected]':
ref: 'v8.0.2'
commit: 'sha1-9000827ccba6bdab643e8b6fd33ac0654aef8333'
owner_id: 44036562
repo_id: 192626254
'actions/[email protected]':
ref: 'v7.1.0'
commit: 'sha1-949feb2413d6458794dcd2491c4babbbce0c15c1'
owner_id: 44036562
repo_id: 189476904
'actions/[email protected]':
ref: 'v7.0.2'
commit: 'sha1-cf430e030ddbb5b0abf93d22962f4752f3646cd9'
owner_id: 44036562
repo_id: 192625955
'davelosert/[email protected]':
ref: 'v2.13.2'
commit: 'sha1-74c2d9210b4fc44c84d8213f61d7ec99837b7aa0'
owner_id: 4287128
repo_id: 501176456
'dependabot/[email protected]':
ref: 'v3.1.0'
commit: 'sha1-25dd0e34f4fe68f24cc83900b1fe3fe149efef98'
owner_id: 27347476
repo_id: 371068214
'docker/[email protected]':
ref: 'v7.4.0'
commit: 'sha1-c3c9e263c25d99ce0380d002d59b67737d91b0dc'
owner_id: 5429470
repo_id: 241092383
'docker/[email protected]':
ref: 'v4.8.0'
commit: 'sha1-8aa5831fb8d48d8f76594c7ddde61ae58019f550'
owner_id: 5429470
repo_id: 287743349
'docker/[email protected]':
ref: 'v6.2.0'
commit: 'sha1-dc802804100637a589fabce1cb79ff13a1411302'
owner_id: 5429470
repo_id: 306769011
'docker/[email protected]':
ref: 'v4.4.1'
commit: 'sha1-f87e5991a6d7451dcb8d9637bfbc97413f497069'
owner_id: 5429470
repo_id: 288485773
'docker/[email protected]':
ref: 'v4.4.0'
commit: 'sha1-99012661954931238ded8c8b007157a8430204e1'
owner_id: 5429470
repo_id: 288487894
'dorny/[email protected]':
ref: 'v4.0.3'
commit: 'sha1-ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d'
owner_id: 1588572
repo_id: 265601040
'peter-evans/[email protected]':
ref: 'v5.0.0'
commit: 'sha1-1b9a80c056b620d92cedb9d9b5a223409c68ddfa'
owner_id: 18365890
repo_id: 193685475
@@ -17,10 +17,12 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
- name: Set up Node.js
uses: actions/setup-node@v7
uses: actions/setup-node@v7.1.0
with:
node-version: '26'
cache: 'npm'
+18 -12
View File
@@ -23,8 +23,10 @@ jobs:
outputs:
backend: ${{ steps.filter.outputs.backend }}
steps:
- uses: actions/checkout@v7
- uses: dorny/paths-filter@v4
- uses: actions/checkout@v7.0.1
with:
persist-credentials: false
- uses: dorny/[email protected]
id: filter
with:
filters: |
@@ -52,10 +54,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
- name: Set up Node.js
uses: actions/setup-node@v7
uses: actions/setup-node@v7.1.0
with:
node-version: '26'
cache: 'npm'
@@ -80,13 +84,14 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
ref: ${{ github.event.pull_request.head.sha || github.sha }}
repository: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
- name: Set up Node.js
uses: actions/setup-node@v7
uses: actions/setup-node@v7.1.0
with:
node-version: '26'
cache: 'npm'
@@ -106,7 +111,7 @@ jobs:
- name: Upload blob report
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: backend-blob-${{ matrix.shard }}
path: .vitest-reports/
@@ -146,13 +151,14 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
ref: ${{ github.event.pull_request.head.sha || github.sha }}
repository: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
- name: Set up Node.js
uses: actions/setup-node@v7
uses: actions/setup-node@v7.1.0
with:
node-version: '26'
cache: 'npm'
@@ -161,7 +167,7 @@ jobs:
run: npm ci
- name: Download blob reports
uses: actions/download-artifact@v8
uses: actions/download-artifact@v8.0.2
with:
pattern: backend-blob-*
path: .vitest-reports
@@ -173,7 +179,7 @@ jobs:
CI: 'true'
- name: Upload coverage artifact
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: backend-coverage-${{ github.event_name == 'push' && 'main' || 'pr' }}
path: src/backend/coverage/
@@ -204,7 +210,7 @@ jobs:
- name: Report coverage on PR
if: github.event_name == 'pull_request'
uses: davelosert/vitest-coverage-report-action@v2
uses: davelosert/vitest-coverage-report-action@v2.13.2
with:
json-summary-path: src/backend/coverage/coverage-summary.json
json-final-path: src/backend/coverage/coverage-final.json
+1 -1
View File
@@ -13,7 +13,7 @@ jobs:
steps:
- name: Dependabot metadata
id: metadata
uses: dependabot/fetch-metadata@v3
uses: dependabot/fetch-metadata@v3.1.0
with:
github-token: "${{ secrets.GITHUB_TOKEN }}"
+10 -8
View File
@@ -45,30 +45,32 @@ jobs:
fi
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
- name: Log in to GitHub Package Container registry
uses: docker/login-action@v4
uses: docker/login-action@v4.8.0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Log in to Docker Hub
uses: docker/login-action@v4
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Set up QEMU
uses: docker/setup-qemu-action@v4
uses: docker/setup-qemu-action@v4.4.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
uses: docker/setup-buildx-action@v4.4.1
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v6
uses: docker/metadata-action@v6.2.0
with:
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
@@ -82,7 +84,7 @@ jobs:
type=raw,value=main,enable=${{ github.event_name == 'push' || inputs.push_main }}
- name: Build and push Docker image
uses: docker/build-push-action@v7
uses: docker/build-push-action@v7.4.0
with:
platforms: linux/amd64,linux/arm64
context: .
@@ -95,7 +97,7 @@ jobs:
cache-to: type=gha,mode=max
- name: Sync README to Docker Hub
uses: peter-evans/dockerhub-description@v5
uses: peter-evans/dockerhub-description@v5.0.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
+7 -2
View File
@@ -5,16 +5,21 @@ on:
paths:
- 'src/docs/**'
permissions:
contents: read
jobs:
build-docs:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
- name: Set up Node.js
uses: actions/setup-node@v7
uses: actions/setup-node@v7.1.0
with:
node-version: '26'
cache: 'npm'
-40
View File
@@ -1,40 +0,0 @@
name: Maintain Release Merge PR
on:
push:
branches:
- main
jobs:
update-release-pr:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- name: Setup Git
run: |
git config user.name github-actions
git config user.email [email protected]
- name: Check for existing PR
id: find-pr
uses: juliangruber/find-pull-request-action@v1
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
branch: release
- uses: actions/checkout@v7
if: steps.find-pr.outputs.number == ''
with:
ref: release
- name: Reset release branch
if: steps.find-pr.outputs.number == ''
run: |
git fetch origin main:main
git reset --hard main
- name: Create/Update PR
if: steps.find-pr.outputs.number == ''
uses: peter-evans/create-pull-request@v8
with:
token: ${{ secrets.GITHUB_TOKEN }}
commit-message: Update release branch
title: Merge Release Auto-PR
body: Merging this PR will invoke release actions
branch: auto-update/release
+2
View File
@@ -5,6 +5,8 @@ on:
branches:
- main
permissions: {}
jobs:
notify:
runs-on: ubuntu-latest
+23 -14
View File
@@ -22,8 +22,10 @@ jobs:
outputs:
puterjs: ${{ steps.filter.outputs.puterjs }}
steps:
- uses: actions/checkout@v7
- uses: dorny/paths-filter@v4
- uses: actions/checkout@v7.0.1
with:
persist-credentials: false
- uses: dorny/[email protected]
id: filter
with:
filters: |
@@ -52,10 +54,12 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
- name: Set up Node.js
uses: actions/setup-node@v7
uses: actions/setup-node@v7.1.0
with:
node-version: '26'
cache: 'npm'
@@ -73,10 +77,12 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
- name: Set up Node.js
uses: actions/setup-node@v7
uses: actions/setup-node@v7.1.0
with:
node-version: '26'
cache: 'npm'
@@ -130,14 +136,15 @@ jobs:
steps:
- name: Checkout ${{ matrix.ref }}
if: needs.changes.outputs.puterjs == 'true'
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
ref: ${{ matrix.ref }}
repository: ${{ matrix.repository }}
- name: Set up Node.js
if: needs.changes.outputs.puterjs == 'true'
uses: actions/setup-node@v7
uses: actions/setup-node@v7.1.0
with:
node-version: '26'
cache: 'npm'
@@ -161,7 +168,7 @@ jobs:
- name: Upload coverage artifact
if: needs.changes.outputs.puterjs == 'true' && !cancelled()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: puterjs-coverage-${{ matrix.artifact }}
path: src/puter-js/coverage/coverage-*.json
@@ -174,13 +181,15 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
persist-credentials: false
# Missing when the PR leg failed; `coverage (pr)` already reports that.
- name: Download PR coverage
id: pr-coverage
continue-on-error: true
uses: actions/download-artifact@v8
uses: actions/download-artifact@v8.0.2
with:
name: puterjs-coverage-pr
path: src/puter-js/coverage
@@ -191,14 +200,14 @@ jobs:
id: base-coverage
if: steps.pr-coverage.outcome == 'success'
continue-on-error: true
uses: actions/download-artifact@v8
uses: actions/download-artifact@v8.0.2
with:
name: puterjs-coverage-base
path: coverage-base
- name: Report coverage on PR
if: steps.pr-coverage.outcome == 'success' && steps.base-coverage.outcome == 'success'
uses: davelosert/vitest-coverage-report-action@v2
uses: davelosert/vitest-coverage-report-action@v2.13.2
with:
name: puter.js SDK
json-summary-path: src/puter-js/coverage/coverage-summary.json
@@ -207,7 +216,7 @@ jobs:
- name: Report coverage on PR (no base comparison)
if: steps.pr-coverage.outcome == 'success' && steps.base-coverage.outcome != 'success'
uses: davelosert/vitest-coverage-report-action@v2
uses: davelosert/vitest-coverage-report-action@v2.13.2
with:
name: puter.js SDK
json-summary-path: src/puter-js/coverage/coverage-summary.json
-18
View File
@@ -1,18 +0,0 @@
on:
push:
branches:
- main
permissions:
contents: write
pull-requests: write
name: release-please
jobs:
release-please:
runs-on: ubuntu-latest
steps:
- uses: google-github-actions/release-please-action@v4
with:
release-type: node
+2 -2
View File
@@ -45,14 +45,14 @@ jobs:
fi
- name: Log in to GitHub Package Container registry
uses: docker/login-action@v4
uses: docker/login-action@v4.8.0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
uses: docker/setup-buildx-action@v4.4.1
- name: Create alias tag
env:
+14
View File
@@ -62,6 +62,10 @@
// direct socket peer. NEVER set to `true` in prod — it trusts every hop
// and makes X-Forwarded-For forgeable.
"trust_proxy": false,
// Request header a TLS-terminating proxy in front of the server sets to
// the client's TLS fingerprint (JA3/JA4). Only set it when that proxy
// overwrites any value the client sends. Unset disables it.
// "tlsFingerprintHeader": "x-tls-fingerprint",
"no_browser_launch": false,
// ── Dev watcher (devWatcher extension) ──────────────────────────────
@@ -254,6 +258,16 @@
"user": "",
"pass": ""
}
// Each send names a stream: "auth", "signup" or "notify". A stream
// listed here adds its headers to every message on it, and its "from"
// when the sender gives none. Tags are written to "tagsHeader".
// "streams": {
// "notify": {
// "from": "\"Puter\" <[email protected]>",
// "headers": { "X-Example-Stream": "notify" },
// "tagsHeader": "X-Example-Tags"
// }
// }
},
// ── Inbound mail (separate process) ─────────────────────────────────
+6 -6
View File
@@ -16,7 +16,7 @@ Core subscribes the browser to a push service only when the server also publishe
`uid` is the notification's uid, a later push with the same `group` replaces the one showing, and `target` and `data` reach the app when the user taps it.
When the browser replaces the subscription (`pushsubscriptionchange`), `sw.js` posts `{ type: 'puter-push-subscription-changed' }` to every open window on Puter's origin. The worker has no token or session, so the page's push client registers the new subscription.
When the browser replaces the subscription (`pushsubscriptionchange`), `sw.js` keeps the new one, or subscribes again with Puter's stored key when the browser gave none and the event, the receipt or an earlier mark names the subscription that went. It writes `puter:subscription-changed:<scope>` to the `puter-push` IndexedDB as `{ at, oldEndpoint }`, the old endpoint kept from an earlier mark no page handled yet, else taken from the event or from the receipt, forgets the receipt kept for that scope, and posts `{ type: 'puter-push-subscription-changed' }` to every open window on Puter's origin. The worker has no token or session, so an open page runs `helpers/pushSync.js` on that message, after any run already in flight, and a later load does the same through the marker.
## Receipts
@@ -59,9 +59,9 @@ Apps on their own origin do the same through `push-sw.js`, which they import int
```ts
interface PushClient {
enable(): Promise<{ deviceUid: string }>;
enable(options?: { background?: boolean }): Promise<{ deviceUid: string }>;
disable(): Promise<void>;
status(): Promise<{
status(options?: { cached?: boolean }): Promise<{
permission: 'default' | 'granted' | 'denied';
registered: boolean;
deviceUid?: string;
@@ -69,12 +69,12 @@ interface PushClient {
}
```
- `enable()` may ask for notification permission, so core calls it from a click: Turn on in the Notifications tab, Allow on a permission prompt that includes `notifications:push` while this browser is off, and the "Turn on notifications" button of the `/action/enable-push` popup and the matching desktop dialog (`UIEnablePushDialog`). Core also calls it without a click, from `helpers/pushSync.js`, when `Notification.permission` is already `granted`, `status()` reports the browser as not registered, the user has not turned push off in this browser, and an app holds `notifications:push`; the browser does not prompt then. The built-in client runs `puter.notifications.enable()`, which subscribes with the `/sw.js` registration and registers the browser through the Web Push flow above (`platform: 'webpush'`). It then resolves the `uid` of this browser's `webpush` device from `listDevices()`. It rejects when permission is refused or the browser wasn't registered.
- `enable()` may ask for notification permission, so core calls it from a click: Turn on in the Notifications tab, Allow on a permission prompt that includes `notifications:push` while this browser is off, and the "Turn on notifications" button of the `/action/enable-push` popup and the matching desktop dialog (`UIEnablePushDialog`). Core also calls it without a click, from `helpers/pushSync.js`, when `Notification.permission` is already `granted`, `status()` reports the browser as not registered, the user has not turned push off in this browser, and an app may push to Puter's devices (`puterDevices` in `GET /push/apps`); the browser does not prompt then. That call passes `{ background: true }`: a browser the user removed in Puter stays removed, and the built-in client rejects with code `push_device_removed`, after which core records the browser as turned off. Before a background call the built-in client asks `POST /push/web/subscriptions/status` about every earlier endpoint it knows: the marker's `oldEndpoint` and the last endpoint the server answered `active` for (`puter.push.last-endpoint:<user uuid>` in `localStorage`). On a 429 it registers nothing and rejects with `too_many_requests`, which the sync reports as `later`, keeping the marker for the next run. When any answer is `removed`, it ends the current subscription in the browser, clears the marker and rejects with `push_device_removed`, so a subscription change or a dropped subscription never brings back a browser removed from another device. A click renews it as before. For a click the built-in client runs `puter.notifications.enable()`, which subscribes with the `/sw.js` registration and registers the browser through the Web Push flow above (`platform: 'webpush'`); in the background it runs the same registration without replacing a removed subscription or opening Puter's sign-in. It then resolves the `uid` of this browser's `webpush` device from `listDevices()` and clears the subscription-changed marker. It rejects when permission is refused or the browser wasn't registered.
- `disable()` runs `puter.notifications.disable()`, which sends `DELETE /push/web/subscriptions { endpoint }` and ends the browser's subscription. The tab calls it for Turn off, and for Remove on the device marked as this browser. Remove on any other device is `DELETE /push/devices/:uid`.
- `status()` reports the state without prompting. `permission` is `Notification.permission`. `registered` is true when this browser has a device registered for the signed-in user, and `deviceUid` is that device's `uid`.
- `status()` reports the state without prompting. `permission` is `Notification.permission`. `registered` is true when this browser has a device registered for the signed-in user and `POST /push/web/subscriptions/status` answers `active` for the current subscription's endpoint, and `deviceUid` is that device's `uid`. A server that can't answer leaves the device match alone, and so does an `unknown` for an endpoint `enable()` registered in this page in the last 5 minutes, since that check can trail a registration. While the subscription-changed marker is set, `registered` is false. With `{ cached: true }` a client may answer with the last registered state it read in this page. Core passes it for the silent registration on app launch and for an app's `puter.notifications.status()`; the Notifications tab and the push dialogs read fresh. The built-in client keeps that state in memory for the same user and subscription for up to 5 minutes, from `enable()` or once the server answered `active` for the endpoint, and answers from it without either request. Reads of the same user and endpoint share one real check while it is in flight. It drops it on `disable()`, a failed `enable()`, an unregistered answer, a set subscription-changed marker, and the worker's `puter-push-subscription-changed` message, so a new page load always asks the server. Whether the calling app may push is checked on every call, against a `GET /push/apps` answer the page reuses for 30 seconds and drops on `enable()`, `disable()`, a grant, a revoke and a user change. When it can't be read, or leaves the app out, the grant alone counts.
After `enable()` resolves, the tab calls `POST /push/devices/:uid/test` and asks the user whether a notification showed up.
The tab also reads `GET /push/apps` and `GET`/`POST /push/preferences`.
Core keeps its own record of a browser the user turned off, `puter.push.browser-off:<user uuid>` in `localStorage`, written by Turn off and by Remove on this browser's device, and cleared when `enable()` resolves. The silent registration above skips a browser with that record, so a device removed here stays removed while one removed from another device, or a new sign-in in a browser that already allows notifications, registers again.
Core keeps its own record of a browser the user turned off, `puter.push.browser-off:<user uuid>` in `localStorage`, written by Turn off, by Remove on this browser's device, and by a silent registration the server refuses with `push_device_removed`, and cleared when `enable()` resolves. The silent registration above skips a browser with that record, so a device removed here or from another device stays removed until the user turns it on from a click, while a new sign-in in a browser that already allows notifications registers again.
+14
View File
@@ -326,6 +326,20 @@ Used for password resets, email confirmation, and notifications. Without it thos
To require email confirmation before login, also set `"strict_email_verification_required": true`.
Account and notification mail is sent on a named stream: `auth` (password, address and 2FA changes), `signup` (address confirmation; a `puter.signup.validate` listener may name another: lowercase letters, digits, `:`, `_` and `-`, up to 64 characters, starting with a letter or digit) and `notify` (share, app feedback and contact form mail). Add `"streams"` inside `"email"` to configure one, for instance to let your mail provider route or report each kind of mail separately:
```json
"streams": {
"notify": {
"from": "\"Puter\" <[email protected]>",
"headers": { "X-Example-Stream": "notify" },
"tagsHeader": "X-Example-Tags"
}
}
```
`headers` are added to every message on the stream, `from` replaces the default sender, and `tagsHeader` names the header that carries a message's tags as `name=value` pairs joined by `, `. A stream that is not listed adds nothing: the message goes out with the default from address and the caller's headers. `streams` is never handed to the transport.
To read the mail yourself instead of sending it, point the transport at a local SMTP sink — [MailHog](https://github.com/mailhog/MailHog) catches everything and serves it at `http://localhost:8025`. Publish both ports; a container that only exposes them is unreachable from the host:
```sh
+86 -77
View File
@@ -897,21 +897,21 @@
}
},
"node_modules/@babel/core": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/core/-/core-7.29.7.tgz",
"integrity": "sha512-RgHBCvtjbOK2gXSNBNIkNoEc9qoVEtau3hj8gEqKQuL3HZAibKarWFEI3Lfm6EYKkLalOh8eSrj9b+ch9H/VBA==",
"version": "7.29.13",
"resolved": "https://registry.npmjs.org/@babel/core/-/core-7.29.13.tgz",
"integrity": "sha512-j1nvOQCZYSgp7HLmisnCXFTpVvFODiijQ9QHkORfGKWH6YDtskTEeCJS7eV545GZ9UBzrq0e2xsOd5QQ5PjIqw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@babel/code-frame": "^7.29.7",
"@babel/generator": "^7.29.7",
"@babel/helper-compilation-targets": "^7.29.7",
"@babel/helper-module-transforms": "^7.29.7",
"@babel/helpers": "^7.29.7",
"@babel/parser": "^7.29.7",
"@babel/generator": "^7.29.13",
"@babel/helper-compilation-targets": "^7.29.13",
"@babel/helper-module-transforms": "^7.29.13",
"@babel/helpers": "^7.29.13",
"@babel/parser": "^7.29.13",
"@babel/template": "^7.29.7",
"@babel/traverse": "^7.29.7",
"@babel/types": "^7.29.7",
"@babel/traverse": "^7.29.13",
"@babel/types": "^7.29.11",
"@jridgewell/remapping": "^2.3.5",
"convert-source-map": "^2.0.0",
"debug": "^4.1.0",
@@ -938,14 +938,14 @@
}
},
"node_modules/@babel/generator": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/generator/-/generator-7.29.7.tgz",
"integrity": "sha512-DkXD5OJQaAQIdZ1bt3UZdEnHAn9Imd3IVBdX03UFe+ony9Ojw5pzr9YVKGDY1jt+Gcn/FnGkNf8r+Vj5NOJWtQ==",
"version": "7.29.13",
"resolved": "https://registry.npmjs.org/@babel/generator/-/generator-7.29.13.tgz",
"integrity": "sha512-T5FRo+RzVHvGjWOuF1Ljc44b+NXAOazodHkGUHp4I3zCkQ4/sNVJm56h1S278LMR0bY5xBhztGziSbli8zObIw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@babel/parser": "^7.29.7",
"@babel/types": "^7.29.7",
"@babel/parser": "^7.29.13",
"@babel/types": "^7.29.11",
"@jridgewell/gen-mapping": "^0.3.12",
"@jridgewell/trace-mapping": "^0.3.28",
"jsesc": "^3.0.2"
@@ -955,9 +955,9 @@
}
},
"node_modules/@babel/helper-compilation-targets": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/helper-compilation-targets/-/helper-compilation-targets-7.29.7.tgz",
"integrity": "sha512-wem6WaBj4NaVYVdNhLPPVacES6ZJ+KBBfSkTMD3YZxbP3rm3Di85tJU5ljaUNhaOynt+Aj0xruhYuzQBt8n71g==",
"version": "7.29.13",
"resolved": "https://registry.npmjs.org/@babel/helper-compilation-targets/-/helper-compilation-targets-7.29.13.tgz",
"integrity": "sha512-8WxJ4/Vmes3ghacj8VOJy3NW8eyHiFY/4thU/LSBuBE470+igjR/j1ZTNn8fV5VdNrLx6iXUwig526UdOvfP0w==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -982,22 +982,15 @@
}
},
"node_modules/@babel/helper-compilation-targets/node_modules/semver": {
"version": "6.3.1",
"resolved": "https://registry.npmjs.org/semver/-/semver-6.3.1.tgz",
"integrity": "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA==",
"version": "6.3.2",
"resolved": "https://registry.npmjs.org/semver/-/semver-6.3.2.tgz",
"integrity": "sha512-GYrCADwpu6yV5OAvchkQ27Z5wwMxWAEMg5NNWjyHh/5DboFFqbdbvYNIvZouQpxXyeMPoTOh0vdPYKlf0ZswdQ==",
"dev": true,
"license": "ISC",
"bin": {
"semver": "bin/semver.js"
}
},
"node_modules/@babel/helper-compilation-targets/node_modules/yallist": {
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/yallist/-/yallist-3.1.1.tgz",
"integrity": "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g==",
"dev": true,
"license": "ISC"
},
"node_modules/@babel/helper-globals": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/helper-globals/-/helper-globals-7.29.7.tgz",
@@ -1023,15 +1016,15 @@
}
},
"node_modules/@babel/helper-module-transforms": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/helper-module-transforms/-/helper-module-transforms-7.29.7.tgz",
"integrity": "sha512-UPUVSyXbOh627KiCIGQSgwWzGeBKLkaJ9PJEdrngIwMSzxLR4jS4+f1f1jb7VzBbg8nFLaYotvVPFCTqdrmTAg==",
"version": "7.29.13",
"resolved": "https://registry.npmjs.org/@babel/helper-module-transforms/-/helper-module-transforms-7.29.13.tgz",
"integrity": "sha512-/U96wOz+SCoZ7Da/8kjDGltcNY280JPQcRNPK0Hk3PTbdpaiBDQImtC6JsbVwb1HGMr3usZQU2WsWVjhYNmRzw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@babel/helper-module-imports": "^7.29.7",
"@babel/helper-validator-identifier": "^7.29.7",
"@babel/traverse": "^7.29.7"
"@babel/traverse": "^7.29.13"
},
"engines": {
"node": ">=6.9.0"
@@ -1051,9 +1044,9 @@
}
},
"node_modules/@babel/helper-string-parser": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.29.7.tgz",
"integrity": "sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw==",
"version": "7.29.13",
"resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.29.13.tgz",
"integrity": "sha512-fWegr+WsEsiMVW4KUNdIIzzMTZSzNiqhy0//6+2MRi2d5NFNsL/AYwBQW8eBnRbTBVRnDVzjaJEUqKaCWjMMdQ==",
"dev": true,
"license": "MIT",
"engines": {
@@ -1081,27 +1074,27 @@
}
},
"node_modules/@babel/helpers": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/helpers/-/helpers-7.29.7.tgz",
"integrity": "sha512-1k2lAGRMfHTcwuNYcCNUmaUffmQv8KWMfh2iJUUeRlwlwH4FdNG7mfPI10NPfLHJFThE4Tyr4mv7kTNZOiPuBg==",
"version": "7.29.13",
"resolved": "https://registry.npmjs.org/@babel/helpers/-/helpers-7.29.13.tgz",
"integrity": "sha512-laoRpD/kO/iA4CDlgJbixHekg7oPvWALK0PQyK3Ob2I2njZRvVAD77RfWCIImtqWCq6KijoBoZK7bp/HlTWASQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@babel/template": "^7.29.7",
"@babel/types": "^7.29.7"
"@babel/types": "^7.29.11"
},
"engines": {
"node": ">=6.9.0"
}
},
"node_modules/@babel/parser": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.7.tgz",
"integrity": "sha512-hnORnjP/1P/zFEndoeX+n+t1RwWRJiJpM/jO7FW32Kn9r5+sJB2JWOdYo4L6k78j15eCwY3Gm/7364B1EMwtNg==",
"version": "7.29.13",
"resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.13.tgz",
"integrity": "sha512-fJM50YYjlTiBhgdkGmC1tBpv+FMAyo4annf2X6Aks+0yRlaiVkYW/3f07xAfVVkYIoZ9fFZp+mEThUFnuybhkQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@babel/types": "^7.29.7"
"@babel/types": "^7.29.11"
},
"bin": {
"parser": "bin/babel-parser.js"
@@ -1135,18 +1128,18 @@
}
},
"node_modules/@babel/traverse": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/traverse/-/traverse-7.29.7.tgz",
"integrity": "sha512-EhlfNQtZ+NK22w5BM61ciuiq1m58ed33Wr1Xan//ZRTy6hgjnwyCffRYwzsGXdASJSUJ1guZILsErh1eQcl+zw==",
"version": "7.29.13",
"resolved": "https://registry.npmjs.org/@babel/traverse/-/traverse-7.29.13.tgz",
"integrity": "sha512-0ve0xMlFA73dgY1oyJRY6RoW+/bncdcOQqg6jjYcVfXBK90OdLXjm9R6lmRetdZVrPjXkKIoMVBqYFPjcAlZ4Q==",
"dev": true,
"license": "MIT",
"dependencies": {
"@babel/code-frame": "^7.29.7",
"@babel/generator": "^7.29.7",
"@babel/generator": "^7.29.13",
"@babel/helper-globals": "^7.29.7",
"@babel/parser": "^7.29.7",
"@babel/parser": "^7.29.13",
"@babel/template": "^7.29.7",
"@babel/types": "^7.29.7",
"@babel/types": "^7.29.11",
"debug": "^4.3.1"
},
"engines": {
@@ -1154,13 +1147,13 @@
}
},
"node_modules/@babel/types": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.7.tgz",
"integrity": "sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA==",
"version": "7.29.11",
"resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.11.tgz",
"integrity": "sha512-WCsemW40Cpc6gouC1SrY6PBaZ+oLRlEVY9XBvZBI7il/HyUrb2nJ505wE1MHEFyDZNnrWDEuy4jL50WVHfTTSQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@babel/helper-string-parser": "^7.29.7",
"@babel/helper-string-parser": "^7.29.11",
"@babel/helper-validator-identifier": "^7.29.7"
},
"engines": {
@@ -7460,18 +7453,18 @@
"license": "MIT"
},
"node_modules/body-parser": {
"version": "2.3.0",
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.3.0.tgz",
"integrity": "sha512-2cGmJupaNgg+QUwVLAucDuWuoMZ6EX9iHDRswZ5lsNYEmwPaRknMPCLZz07yTzVq/83p4o/wzbDZbBrTvGGTIw==",
"version": "2.4.0",
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.4.0.tgz",
"integrity": "sha512-Ifk1KcLrxaGt1occQagQNFNNOqRt1LFuehEnHHzB8Z6cRgIxRU1TBaiIsUiCdA+gy5PL7Y1IX31DNKkRmxtXjQ==",
"license": "MIT",
"dependencies": {
"bytes": "^3.1.2",
"content-type": "^2.0.0",
"content-type": "^2.1.0",
"debug": "^4.4.3",
"http-errors": "^2.0.1",
"iconv-lite": "^0.7.2",
"iconv-lite": "^0.7.3",
"on-finished": "^2.4.1",
"qs": "^6.15.2",
"qs": "^6.16.1",
"raw-body": "^3.0.2",
"type-is": "^2.1.0"
},
@@ -7484,9 +7477,9 @@
}
},
"node_modules/body-parser/node_modules/content-type": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/content-type/-/content-type-2.0.0.tgz",
"integrity": "sha512-j/O/d7GcZCyNl7/hwZAb606rzqkyvaDctLmckbxLzHvFBzTJHuGEdodATcP3yIRoDrLHkIATJuvzbFlp/ki2cQ==",
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/content-type/-/content-type-2.1.0.tgz",
"integrity": "sha512-mj7UPXE0jaqaOsukNZRUEfEi2AcL7C/vwmwcHV0O97eO1E1pxBZuyjlZrx5seTaNBg1U6+o35wpa35Qfcc+7ag==",
"license": "MIT",
"engines": {
"node": ">=18"
@@ -9581,9 +9574,9 @@
}
},
"node_modules/fastify": {
"version": "5.12.5",
"resolved": "https://registry.npmjs.org/fastify/-/fastify-5.12.5.tgz",
"integrity": "sha512-OB2k1dlxs5/NAABqeKV2FUHkSD2BbENsCak8yULVcymn3fHIPDVa9TI3SDnJSWYSllZmSYuZXy2gTnsT+Sut1A==",
"version": "5.12.7",
"resolved": "https://registry.npmjs.org/fastify/-/fastify-5.12.7.tgz",
"integrity": "sha512-v4pEPsTRPuqlDIYiQFB+BHjVN4SdHX9u0ABk0ZDUKbbdY1vsNryUd1j4Z+jQEZajvYIVk047Kimva9/AJ79GYw==",
"funding": [
{
"type": "github",
@@ -16587,9 +16580,9 @@
"license": "MIT"
},
"node_modules/undici": {
"version": "7.30.1",
"resolved": "https://registry.npmjs.org/undici/-/undici-7.30.1.tgz",
"integrity": "sha512-03KrMbjIFDWQR5RNPrSw3bd1r12JpyFQl9/yxjUbJtqkjmllqC5lFjZh6ujeuJCKr8EOQvRW872Oy/az+nLYxg==",
"version": "7.30.2",
"resolved": "https://registry.npmjs.org/undici/-/undici-7.30.2.tgz",
"integrity": "sha512-mLyPQnHwUxaM/rideoK/+KL8DaAJLRl4aBCfmX444aCSLnIaomqPLtNRe2jdCYcsk5wZzXrDIQC8ZeYNRAYyIA==",
"license": "MIT",
"engines": {
"node": ">=20.18.1"
@@ -18141,6 +18134,13 @@
"node": ">=10"
}
},
"node_modules/yallist": {
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/yallist/-/yallist-3.1.1.tgz",
"integrity": "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g==",
"dev": true,
"license": "ISC"
},
"node_modules/yaml": {
"version": "2.9.0",
"resolved": "https://registry.npmjs.org/yaml/-/yaml-2.9.0.tgz",
@@ -18233,9 +18233,9 @@
}
},
"node_modules/zod": {
"version": "4.4.3",
"resolved": "https://registry.npmjs.org/zod/-/zod-4.4.3.tgz",
"integrity": "sha512-ytENFjIJFl2UwYglde2jchW2Hwm4GJFLDiSXWdTrJQBIN9Fcyp7n4DhxJEiWNAJMV1/BqWfW/kkg71UDcHJyTQ==",
"version": "4.6.7",
"resolved": "https://registry.npmjs.org/zod/-/zod-4.6.7.tgz",
"integrity": "sha512-g8W2D2kU6a8untc053fX1uVBKzM7yqAxW/3zwk83+6RyF2+SKOuV+c0wBDhUwzV1PaGapmZH+rmcUzPf+3CGwg==",
"license": "MIT",
"funding": {
"url": "https://github.com/sponsors/colinhacks"
@@ -18300,7 +18300,7 @@
"mime-types": "^2.1.35",
"murmurhash": "^2.0.1",
"mysql2": "^3.24.6",
"nodemailer": "^10.1.4",
"nodemailer": "^10.1.7",
"openai": "^6.34.0",
"otpauth": "^9.5.2",
"pg": "^8.21.0",
@@ -18312,10 +18312,10 @@
"svg-captcha": "^1.4.0",
"together-ai": "^0.33.0",
"ua-parser-js": "^1.0.41",
"undici": "^7.30.1",
"undici": "^7.30.2",
"uuid": "^14.0.3",
"validator": "^13.15.35",
"ws": "^8.22.0"
"ws": "^8.22.2"
},
"devDependencies": {
"@types/bcrypt": "^6.0.0",
@@ -18343,10 +18343,19 @@
"dev": true,
"license": "MIT"
},
"src/backend/node_modules/nodemailer": {
"version": "10.1.7",
"resolved": "https://registry.npmjs.org/nodemailer/-/nodemailer-10.1.7.tgz",
"integrity": "sha512-OQa6FDln5fLttmqSbjDiRYvFDOFNrBqfiOwHPlEbURil3b+vPB1v8xpi+Sp46drzNDlsTnNsS502vthmz56r4w==",
"license": "MIT-0",
"engines": {
"node": ">=20.0.0"
}
},
"src/backend/node_modules/ws": {
"version": "8.22.0",
"resolved": "https://registry.npmjs.org/ws/-/ws-8.22.0.tgz",
"integrity": "sha512-Ydggc987+RO0AnWtZ/7Wq9FtNvcrL1b/RO0ud9mWjUPgDrsAAwQSF51sm2hm1XofbU/4jkpGEsLFsZZxU+1DOg==",
"version": "8.22.2",
"resolved": "https://registry.npmjs.org/ws/-/ws-8.22.2.tgz",
"integrity": "sha512-/rEZ2ejown2N5gxVTHUu9bVVC7ttu0/Tgzf5hbTOixR3Fs7mbbXgn2DbdbcKgJOmpQMtrIUWivtYuVMnp/W/yA==",
"license": "MIT",
"engines": {
"node": ">=10.0.0"
@@ -167,6 +167,38 @@ describe('AbstractDatabaseClient — replica-aware reads', () => {
expect(client.reads).toEqual(['SELECT 1']);
expect(client.preads).toEqual([]);
});
describe('tryHardReadAll', () => {
const reads = [{ query: 'SELECT 1' }, { query: 'SELECT 2' }];
it('keeps replica rows without reading the primary', async () => {
const client = new FixtureDatabaseClient([{ id: 1 }], [{ id: 2 }]);
expect(await client.tryHardReadAll(reads)).toEqual([
[{ id: 1 }],
[{ id: 1 }],
]);
expect(client.preads).toEqual([]);
});
it.each([[], new Error('replica unavailable')])(
'reads every query on the primary when the replica has none',
async (replica) => {
const client = new FixtureDatabaseClient(replica, [{ id: 2 }]);
expect(await client.tryHardReadAll(reads)).toEqual([
[{ id: 2 }],
[{ id: 2 }],
]);
expect(client.preads).toEqual(['SELECT 1', 'SELECT 2']);
},
);
it('issues one read per query when there is no replica', async () => {
const client = new FixtureDatabaseClient([], [{ id: 2 }], false);
expect(await client.tryHardReadAll(reads)).toEqual([[], []]);
expect(client.reads).toEqual(['SELECT 1', 'SELECT 2']);
expect(client.preads).toEqual([]);
});
});
});
describe('AbstractDatabaseClient — SQL generation', () => {
@@ -172,6 +172,34 @@ export class AbstractDatabaseClient extends PuterClient {
return result.value;
}
/** Reads the primary only when every replica read is empty or fails. */
@Span('db.tryHardReadAll', (reads: unknown[]) => ({
'db.batch_size': reads.length,
}))
async tryHardReadAll(
reads: { query: string; params?: unknown[] }[],
): Promise<Record<string, unknown>[][]> {
if (!this.hasReadReplica()) {
return Promise.all(
reads.map(({ query, params = [] }) => this.read(query, params)),
);
}
const onReplica = await Promise.all(
reads.map(async ({ query, params = [] }) => {
try {
return await this.replicaRead(query, params);
} catch {
// A failed replica read counts as a miss.
return [];
}
}),
);
if (onReplica.some((rows) => rows.length > 0)) return onReplica;
return Promise.all(
reads.map(({ query, params = [] }) => this.pread(query, params)),
);
}
/**
* Apply the `.sql` files named `<engineName>*` in each configured migration
* directory, in filename order. Files must be idempotent: nothing records
@@ -27,7 +27,7 @@ import { DatabaseClientFactory } from './index.js';
import { SqliteDatabaseClient } from './SqliteDatabaseClient.js';
/** Highest schema version the migration table can reach. */
const CURRENT_SCHEMA_VERSION = 98;
const CURRENT_SCHEMA_VERSION = 101;
/**
* These suites migrate real files on disk. Idle they finish in well under a
@@ -705,6 +705,65 @@ describe('SqliteDatabaseClient — boot and migrations', { timeout: DISK_MIGRATI
}
});
it('fills a share row missing its entry owner from 0103, invites left alone', async () => {
const dir = mkdtempSync(join(tmpdir(), 'puter-sqlite-share-owner-'));
const path = join(dir, 'puter.sqlite');
try {
const before = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path }),
);
await before.onServerStart();
const [user] = (await before.read(
'SELECT MIN(`id`) AS id FROM `user`',
)) as { id: number }[];
await before.write(
'INSERT INTO `fsentries` (`uuid`, `name`, `user_id`, `modified`) ' +
'VALUES (?, ?, ?, ?)',
['22222222-2222-4222-8222-222222222222', 'owned.txt', user.id, 0],
);
const [entry] = (await before.read(
'SELECT `id` FROM `fsentries` WHERE `uuid` = ?',
['22222222-2222-4222-8222-222222222222'],
)) as { id: number }[];
await before.write(
'INSERT INTO `share` (`uid`, `issuer_user_id`, ' +
'`recipient_email`, `fsentry_id`) VALUES (?, ?, ?, ?)',
['share-no-owner', user.id, '[email protected]', entry.id],
);
await before.write(
'INSERT INTO `share` (`uid`, `issuer_user_id`, ' +
'`recipient_email`) VALUES (?, ?, ?)',
['share-invite', user.id, '[email protected]'],
);
// The full schema as it stood before 0103, so 0103 reruns; 0104
// is undone so it can apply again after it.
await before.write('DROP INDEX idx_user_signup_tls_fingerprint');
await before.write(
'ALTER TABLE `user` DROP COLUMN `signup_tls_fingerprint`',
);
await before.write('PRAGMA user_version = 98');
before.onServerShutdown();
const after = new SqliteDatabaseClient(
sqliteConfig({ inMemory: false, path }),
);
await after.onServerStart();
await expect(
after.read(
'SELECT `uid`, `entry_owner_user_id` FROM `share` ' +
'WHERE `uid` IN (?, ?) ORDER BY `uid`',
['share-invite', 'share-no-owner'],
),
).resolves.toEqual([
{ uid: 'share-invite', entry_owner_user_id: null },
{ uid: 'share-no-owner', entry_owner_user_id: user.id },
]);
after.onServerShutdown();
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it('indexes notification.created_at for the retention sweep', async () => {
const rows = await client.read('PRAGMA index_list(`notification`)');
expect(rows.map((r) => String(r.name))).toContain(
@@ -137,6 +137,9 @@ const AVAILABLE_MIGRATIONS: [number, string[]][] = [
[95, ['0100_notification-send-key.sql']],
[96, ['0101_user-scoped-handlers.sql']],
[97, ['0102_notification-scope-id-index.sql']],
[98, ['0103_share-entry-owner-backfill.sql']],
[99, ['0104_signup-tls-fingerprint.sql']],
[100, ['0105_sessions_kind_debug_app.sql']],
];
export class SqliteDatabaseClient extends AbstractDatabaseClient {
@@ -0,0 +1,25 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Fills `share`.`entry_owner_user_id` on any row that names an entry but was
-- written without the owner. Mirrors SQLite migration 0103. No per-file
-- applied-state tracking; a replay finds nothing to fill.
UPDATE `share` JOIN `fsentries` ON `fsentries`.`id` = `share`.`fsentry_id`
SET `share`.`entry_owner_user_id` = `fsentries`.`user_id`
WHERE `share`.`fsentry_id` IS NOT NULL
AND `share`.`entry_owner_user_id` IS NULL;
@@ -0,0 +1,51 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- `user`.`signup_tls_fingerprint`. See
-- sqlite/0104_signup-tls-fingerprint.sql for the rationale. No per-file
-- applied-state tracking, so the column and index are added only when missing
-- and a replay does nothing.
DROP PROCEDURE IF EXISTS _puter_add_user_signup_tls_fingerprint;
DELIMITER //
CREATE PROCEDURE _puter_add_user_signup_tls_fingerprint()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user'
AND COLUMN_NAME = 'signup_tls_fingerprint'
) THEN
ALTER TABLE `user`
ADD COLUMN `signup_tls_fingerprint` varchar(64) DEFAULT NULL,
ALGORITHM=INSTANT;
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user'
AND INDEX_NAME = 'idx_user_signup_tls_fingerprint'
) THEN
ALTER TABLE `user` ADD INDEX `idx_user_signup_tls_fingerprint`
(`signup_tls_fingerprint`), ALGORITHM=INPLACE, LOCK=NONE;
END IF;
END//
DELIMITER ;
CALL _puter_add_user_signup_tls_fingerprint();
DROP PROCEDURE IF EXISTS _puter_add_user_signup_tls_fingerprint;
@@ -0,0 +1,46 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Add 'debug_app' to the `sessions.kind` ENUM: a short-lived app session an
-- operator creates on a user's behalf. Mirrors SQLite migration 0105.
--
-- Idempotent: guarded against COLUMN_TYPE so re-running the directory is a
-- no-op once 'debug_app' is in the ENUM.
DROP PROCEDURE IF EXISTS _puter_sessions_kind_debug_app;
DELIMITER //
CREATE PROCEDURE _puter_sessions_kind_debug_app()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'sessions'
AND COLUMN_NAME = 'kind'
AND FIND_IN_SET('debug_app', REPLACE(REPLACE(REPLACE(COLUMN_TYPE, 'enum(', ''), ')', ''), '''', '')) > 0
) THEN
ALTER TABLE `sessions`
MODIFY COLUMN `kind`
ENUM('web', 'app', 'access_token', 'asset', 'worker', 'debug_app')
NOT NULL DEFAULT 'web',
ALGORITHM=INSTANT;
END IF;
END//
DELIMITER ;
CALL _puter_sessions_kind_debug_app();
DROP PROCEDURE IF EXISTS _puter_sessions_kind_debug_app;
@@ -0,0 +1,27 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Fills "share".entry_owner_user_id on any row that names an entry but was
-- written without the owner. Mirrors SQLite migration 0103. No per-file
-- applied-state tracking; a replay finds nothing to fill.
UPDATE "share"
SET entry_owner_user_id = f.user_id
FROM fsentries f
WHERE f.id = "share".fsentry_id
AND "share".fsentry_id IS NOT NULL
AND "share".entry_owner_user_id IS NULL;
@@ -0,0 +1,22 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- "user".signup_tls_fingerprint. Mirrors SQLite migration 0104. Idempotent
-- via IF NOT EXISTS.
ALTER TABLE "user" ADD COLUMN IF NOT EXISTS signup_tls_fingerprint varchar(64);
CREATE INDEX IF NOT EXISTS idx_user_signup_tls_fingerprint ON "user" (signup_tls_fingerprint);
@@ -0,0 +1,24 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Add 'debug_app' to the `sessions.kind` CHECK constraint: a short-lived app
-- session an operator creates on a user's behalf. Mirrors SQLite migration
-- 0105. Drop-then-add keeps it idempotent across directory replays.
ALTER TABLE sessions DROP CONSTRAINT IF EXISTS sessions_kind_check;
ALTER TABLE sessions ADD CONSTRAINT sessions_kind_check
CHECK (kind IN ('web', 'app', 'access_token', 'asset', 'worker', 'debug_app'));
@@ -0,0 +1,27 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Fills `share`.`entry_owner_user_id` on any row that names an entry but was
-- written without the owner, so reads can rely on the column alone. Rows that
-- name no entry (invites) keep a NULL owner. A replay finds nothing to fill.
UPDATE `share`
SET `entry_owner_user_id` = (
SELECT `user_id` FROM `fsentries` WHERE `fsentries`.`id` = `share`.`fsentry_id`
)
WHERE `fsentry_id` IS NOT NULL
AND `entry_owner_user_id` IS NULL;
@@ -0,0 +1,22 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- `user`.`signup_tls_fingerprint`: the TLS fingerprint of the request that
-- created the account, when a fronting proxy supplies one. Indexed so the
-- accounts created from one TLS stack can be listed.
ALTER TABLE `user` ADD COLUMN `signup_tls_fingerprint` varchar(64) DEFAULT NULL;
CREATE INDEX IF NOT EXISTS idx_user_signup_tls_fingerprint ON `user` (`signup_tls_fingerprint`);
@@ -0,0 +1,89 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Add 'debug_app' to the `kind` CHECK constraint on `sessions`. Mirrors the
-- MySQL ENUM extension in mysql_mig_59.
--
-- SQLite cannot ALTER a CHECK constraint in place, so we follow the same
-- table-rebuild as migration 0056: create `sessions_new` with the corrected
-- constraint, copy rows, drop the old table, rename, then recreate every
-- index that lived on the original.
CREATE TABLE `sessions_new` (
"id" INTEGER PRIMARY KEY AUTOINCREMENT,
"user_id" INTEGER NOT NULL,
"uuid" TEXT NOT NULL,
"meta" JSON DEFAULT NULL,
"created_at" INTEGER DEFAULT 0,
"last_activity" INTEGER DEFAULT 0,
"kind" TEXT NOT NULL DEFAULT 'web'
CHECK (`kind` IN ('web', 'app', 'access_token', 'asset', 'worker', 'debug_app')),
"label" TEXT,
"parent_session_id" TEXT,
"last_ip" TEXT,
"last_user_agent" TEXT,
"revoked_at" INTEGER,
"expires_at" INTEGER,
"app_uid" TEXT,
"legacy_token_uid" TEXT,
"created_via" TEXT,
"auth_id" TEXT,
"access_token_uid" TEXT,
FOREIGN KEY("user_id") REFERENCES "user" ("id") ON DELETE CASCADE ON UPDATE CASCADE
);
INSERT INTO `sessions_new` (
`id`, `user_id`, `uuid`, `meta`, `created_at`, `last_activity`,
`kind`, `label`, `parent_session_id`, `last_ip`, `last_user_agent`,
`revoked_at`, `expires_at`, `app_uid`, `legacy_token_uid`,
`created_via`, `auth_id`, `access_token_uid`
)
SELECT
`id`, `user_id`, `uuid`, `meta`, `created_at`, `last_activity`,
`kind`, `label`, `parent_session_id`, `last_ip`, `last_user_agent`,
`revoked_at`, `expires_at`, `app_uid`, `legacy_token_uid`,
`created_via`, `auth_id`, `access_token_uid`
FROM `sessions`;
DROP TABLE `sessions`;
ALTER TABLE `sessions_new` RENAME TO `sessions`;
CREATE INDEX IF NOT EXISTS `idx_sessions_user_revoked`
ON `sessions` (`user_id`, `revoked_at`);
CREATE INDEX IF NOT EXISTS `idx_sessions_parent`
ON `sessions` (`parent_session_id`);
CREATE UNIQUE INDEX IF NOT EXISTS `idx_sessions_user_app_active`
ON `sessions` (`user_id`, `app_uid`)
WHERE `kind` = 'app' AND `revoked_at` IS NULL;
CREATE UNIQUE INDEX IF NOT EXISTS `idx_sessions_legacy_token_active`
ON `sessions` (`legacy_token_uid`)
WHERE `legacy_token_uid` IS NOT NULL AND `revoked_at` IS NULL;
CREATE INDEX IF NOT EXISTS `idx_sessions_kind_user`
ON `sessions` (`kind`, `user_id`);
CREATE INDEX IF NOT EXISTS `idx_sessions_access_token_uid`
ON `sessions` (`access_token_uid`)
WHERE `access_token_uid` IS NOT NULL;
CREATE UNIQUE INDEX IF NOT EXISTS `idx_sessions_user_worker_active`
ON `sessions` (`user_id`, IFNULL(`app_uid`, ''), json_extract(`meta`, '$.worker_name'))
WHERE `kind` = 'worker' AND `revoked_at` IS NULL;
@@ -168,6 +168,204 @@ describe('EmailClient — sending', () => {
});
});
describe('EmailClient — streams', () => {
const STREAMS = {
notify: {
from: '"Notices" <[email protected]>',
headers: { 'X-Example-Stream': 'notify', 'X-Example-Kind': 'n' },
tagsHeader: 'X-Example-Tags',
},
plain: { headers: { 'X-Example-Stream': 'plain' } },
};
const streamClient = () =>
startClient({
email: { jsonTransport: true, from: FROM, streams: STREAMS },
} as unknown as Partial<IConfig>);
const base = { to: '[email protected]', subject: 's', text: 't' };
it('adds the stream headers and from address to the message', async () => {
const info = await streamClient().sendRaw({
...base,
stream: 'notify',
});
const message = sentMessage(info);
expect(message.headers).toMatchObject({
'X-Example-Stream': 'notify',
'X-Example-Kind': 'n',
});
expect(message.from.address).toBe('[email protected]');
});
it('lets caller headers and from win over the stream', async () => {
const info = await streamClient().sendRaw({
...base,
stream: 'notify',
from: '[email protected]',
headers: { 'x-example-stream': 'mine' },
});
const message = sentMessage(info);
expect(message.from.address).toBe('[email protected]');
expect(message.headers?.['x-example-stream']).toBe('mine');
expect(message.headers).not.toHaveProperty('X-Example-Stream');
expect(message.headers?.['X-Example-Kind']).toBe('n');
});
it('keeps the default from address for a stream without one', async () => {
const info = await streamClient().sendRaw({ ...base, stream: 'plain' });
expect(sentMessage(info).from.address).toBe('[email protected]');
});
it('leaves the message unchanged for an unknown or missing stream', async () => {
const client = streamClient();
for (const stream of ['nope', undefined]) {
const message = sentMessage(
await client.sendRaw({ ...base, stream, tags: { a: 'b' } }),
);
expect(message.from.address).toBe('[email protected]');
expect(message.headers ?? {}).toEqual({});
}
});
it('renders valid tags into the stream tags header and drops the rest', async () => {
const info = await streamClient().sendRaw({
...base,
stream: 'notify',
tags: {
kind: 'share',
'app.id': 'a-1:x',
'bad key': 'v',
ok: 'bad value',
long: 'x'.repeat(129),
},
});
expect(sentMessage(info).headers?.['X-Example-Tags']).toBe(
'kind=share, app.id=a-1:x',
);
});
it('keeps a tags header the caller set', async () => {
const info = await streamClient().sendRaw({
...base,
stream: 'notify',
headers: { 'X-Example-Tags': 'own=1' },
tags: { kind: 'share' },
});
expect(sentMessage(info).headers?.['X-Example-Tags']).toBe('own=1');
});
it('sends no tags header when the stream names none or no tag survives', async () => {
const client = streamClient();
const plain = sentMessage(
await client.sendRaw({
...base,
stream: 'plain',
tags: { kind: 'share' },
}),
);
expect(plain.headers).toEqual({ 'X-Example-Stream': 'plain' });
const dropped = sentMessage(
await client.sendRaw({
...base,
stream: 'notify',
tags: { 'bad key': 'v' },
}),
);
expect(dropped.headers).not.toHaveProperty('X-Example-Tags');
});
it('does not hand stream or tags to the transport', async () => {
const info = await streamClient().sendRaw({
...base,
stream: 'notify',
tags: { kind: 'share' },
});
const raw = JSON.parse((info as { message: string }).message);
expect(raw).not.toHaveProperty('stream');
expect(raw).not.toHaveProperty('tags');
});
it('keeps streams out of the transport options', async () => {
const nodemailer = (await import('nodemailer')).default;
const create = vi.spyOn(nodemailer, 'createTransport');
try {
streamClient();
expect(create).toHaveBeenCalledTimes(1);
const [options] = create.mock.calls[0] as [Record<string, unknown>];
expect(options).not.toHaveProperty('streams');
expect(options).not.toHaveProperty('from');
expect(options).toMatchObject({ jsonTransport: true });
} finally {
create.mockRestore();
}
});
it('forwards the template send options to sendRaw', async () => {
const client = streamClient();
const sendRaw = vi.spyOn(client, 'sendRaw').mockResolvedValue(null);
await client.send(
'[email protected]',
'email_verification_code',
{ code: '1' },
{ stream: 'notify', replyTo: '[email protected]' },
);
expect(sendRaw).toHaveBeenCalledWith(
expect.objectContaining({
stream: 'notify',
replyTo: '[email protected]',
}),
);
});
it('applies the stream from address to a template send', async () => {
const info = await streamClient().send(
'[email protected]',
'email_verification_code',
{ code: '1' },
{ stream: 'notify' },
);
expect(sentMessage(info).from.address).toBe('[email protected]');
});
it('reads a stream only from its own config entry', async () => {
const inherited = Object.create({
base: { from: '[email protected]', headers: { 'X-Example-B': '1' } },
});
const client = startClient({
email: { jsonTransport: true, from: FROM, streams: inherited },
} as unknown as Partial<IConfig>);
const message = sentMessage(
await client.sendRaw({ ...base, stream: 'base' }),
);
expect(message.from.address).toBe('[email protected]');
expect(message.headers ?? {}).toEqual({});
});
it('prints the stream name and no tags when no transport is configured', async () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
try {
const client = new EmailClient({
port: 0,
extensions: [],
} as unknown as IConfig);
await client.sendRaw({
...base,
stream: 'notify',
tags: { kind: 'share' },
});
const printed = warn.mock.calls[0][1] as Record<string, unknown>;
expect(printed).not.toHaveProperty('tags');
expect(printed.stream).toBe('notify');
} finally {
warn.mockRestore();
}
});
});
describe('EmailClient — template rendering', () => {
it('renders the code into both subject and body', async () => {
const client = startClient();
+65 -8
View File
@@ -69,14 +69,50 @@ export interface SendMailOptions {
attachments?: EmailAttachment[];
/** Extra message headers (e.g. List-Unsubscribe), passed to the transport. */
headers?: Record<string, string>;
/** Names the `email.streams` entry whose from address and headers apply. */
stream?: string;
/** Rendered into the stream's `tagsHeader`; ignored when it has none. */
tags?: Record<string, string>;
}
/** Options a templated {@link EmailClient.send} forwards to `sendRaw`. */
export type SendTemplateOptions = Pick<SendMailOptions, 'replyTo' | 'stream'>;
const TAG_PART = /^[A-Za-z0-9_.:-]{1,128}$/;
interface CompiledTemplate {
subject: ReturnType<typeof template>;
html: ReturnType<typeof template>;
text?: ReturnType<typeof template>;
}
/** Pairs whose key or value falls outside `TAG_PART` are dropped. */
const renderTags = (tags: Record<string, string> | undefined): string =>
Object.entries(tags ?? {})
.filter(([name, value]) => TAG_PART.test(name) && TAG_PART.test(value))
.map(([name, value]) => `${name}=${value}`)
.join(', ');
const hasHeader = (
headers: Record<string, string> | undefined,
name: string,
): boolean =>
Object.keys(headers ?? {}).some(
(key) => key.toLowerCase() === name.toLowerCase(),
);
// Header names are case-insensitive: `x-a` from `over` replaces `X-A` in `base`.
const mergeHeaders = (
base: Record<string, string> | undefined,
over: Record<string, string> | undefined,
): Record<string, string> => {
const merged: Record<string, string> = {};
for (const [key, value] of Object.entries(base ?? {})) {
if (!hasHeader(over, key)) merged[key] = value;
}
return { ...merged, ...over };
};
// -- EmailClient ------------------------------------------------------
/**
@@ -111,7 +147,8 @@ export class EmailClient extends PuterClient {
return;
}
this.transport = nodemailer.createTransport(emailConf);
const { streams: _streams, from: _from, ...transportConf } = emailConf;
this.transport = nodemailer.createTransport(transportConf);
console.log('[email] transport configured');
}
@@ -125,7 +162,8 @@ export class EmailClient extends PuterClient {
/**
* Render a template and send it to `to`. `options.replyTo` sets the
* Reply-To header (e.g. so a recipient can respond to the originator of the
* message rather than the no-reply From address).
* message rather than the no-reply From address); `options.stream` is
* passed to {@link sendRaw}.
*
* Returns the transport's send result, or `null` when none is configured —
* {@link sendRaw} drops the message instead of throwing. A caller that
@@ -135,7 +173,7 @@ export class EmailClient extends PuterClient {
to: string,
template: T,
values: Record<string, unknown> = {},
options: { replyTo?: string } = {},
options: SendTemplateOptions = {},
) {
const compiled = this.compiledTemplates[template];
if (!compiled) {
@@ -143,12 +181,12 @@ export class EmailClient extends PuterClient {
}
return await this.sendRaw({
from: this.defaultFrom(),
to,
subject: compiled.subject(values),
html: compiled.html(values),
...(compiled.text ? { text: compiled.text(values) } : {}),
...(options.replyTo ? { replyTo: options.replyTo } : {}),
...(options.stream ? { stream: options.stream } : {}),
});
}
@@ -159,19 +197,38 @@ export class EmailClient extends PuterClient {
* Returns the transport's send result, or `null` when no transport is
* configured (the send is a no-op in that case — callers that must not
* silently drop mail should check `config.email` before composing, or the
* `null` return after).
* `null` return after). A configured `stream` supplies the from address
* when the caller gives none, and headers the caller's own override.
*/
async sendRaw(options: SendMailOptions) {
const { stream: streamName, tags, ...message } = options;
if (!this.transport) {
console.warn(
'[email] attempted to send email without transport. If you need to send email, configure an SMTP transport in your config file (see docs for details). Email content:',
options,
{ ...message, ...(streamName ? { stream: streamName } : {}) },
);
return null;
}
const streams = this.config.email?.streams;
const stream =
streamName && streams && Object.hasOwn(streams, streamName)
? streams[streamName]
: undefined;
let headers = mergeHeaders(stream?.headers, message.headers);
const tagsValue = stream?.tagsHeader ? renderTags(tags) : '';
if (
stream?.tagsHeader &&
tagsValue &&
!hasHeader(message.headers, stream.tagsHeader)
) {
headers = mergeHeaders(headers, {
[stream.tagsHeader]: tagsValue,
});
}
return await this.transport.sendMail({
...options,
from: options.from ?? this.defaultFrom(),
...message,
...(Object.keys(headers).length > 0 ? { headers } : {}),
from: message.from ?? stream?.from ?? this.defaultFrom(),
});
}
+26
View File
@@ -20,6 +20,7 @@ import type {
Response as ExpressResponse,
} from 'express';
import { Actor } from '../../core';
import type { EmailTemplateName } from '../email/templates';
import type { UsageInput } from '../../services/metering/types';
import { FSEntry } from '../../stores/fs/FSEntry';
@@ -240,6 +241,8 @@ export type EventMap = {
trail_id?: string;
/** Device signal forwarded verbatim from the signup request body. */
fingerprint?: string | null;
/** TLS fingerprint of the signup request, when a proxy supplies one. */
tls_fingerprint?: string | null;
/**
* Set by the abuse harness — require SMS phone verification
* post-signup.
@@ -247,6 +250,12 @@ export type EventMap = {
requires_phone_verification?: boolean;
/** Set by the abuse harness — require card verification post-signup. */
requires_card_verification?: boolean;
/**
* The email stream the confirmation message is sent on: lowercase
* letters, digits, `:`, `_` and `-`, up to 64 characters, starting with
* a letter or digit. Any other value is ignored.
*/
email_stream?: string | null;
[key: string]: unknown;
};
'puter.signup.success': {
@@ -298,6 +307,8 @@ export type EventMap = {
message: string | null;
[key: string]: unknown;
};
/** Before notification emails; a listener clears `allow` per recipient. */
'email.notify.validate': EmailNotifyValidateEvent;
// Inbound mail to a feedback address (`abuse@`, `fbl@`, ...), emitted via
// `emitAndWait`. A listener that takes it sets `handled` and it reaches no
// mailbox; otherwise it is routed like any other message.
@@ -979,6 +990,21 @@ export type EventMap = {
'outer.pubsub.metering.credits-changed': { userUuid: string };
} & IExtensionEventMap;
/** A union of `Names`, each of which has to be an email template. */
type TemplateNames<Names extends EmailTemplateName> = Names;
/** The emails that ask `email.notify.validate`, by template name. */
export type EmailNotifyKind = TemplateNames<
'file_shared_with_you' | 'file_shared_invite' | 'app-user-feedback'
>;
/** Payload for `email.notify.validate`: one kind of mail, all its recipients. */
export type EmailNotifyValidateEvent = {
kind: EmailNotifyKind;
/** `to` is the address as it will be sent; `allow` starts true. */
recipients: Array<{ to: string; allow: boolean }>;
};
/** Payload for `subscription.gate.<surface>` events. */
export type SubscriptionGateEvent = {
actor: Actor;
@@ -132,6 +132,28 @@ describe('PreludeClient', () => {
});
});
it('forwards a JA4 TLS fingerprint and drops any other shape', async () => {
fetchMock.mockResolvedValue(okJson({ id: 'v', status: 'success' }));
const client = makeClient('sk_test');
const ja4 = 't13d1516h2_8daaf6152771_02713d6af862';
await client.createVerification('+14155550123', {
tls_fingerprint: ja4,
});
await client.createVerification('+14155550123', {
ip: '203.0.113.7',
// A JA3 hash.
tls_fingerprint: 'e7d705a3286e19ea42f587b344ee6865',
});
expect(JSON.parse(fetchMock.mock.calls[0][1].body).signals).toEqual({
ja4_fingerprint: ja4,
});
expect(JSON.parse(fetchMock.mock.calls[1][1].body).signals).toEqual({
ip: '203.0.113.7',
});
});
it('omits the signals object entirely when none are supplied', async () => {
fetchMock.mockResolvedValue(okJson({ id: 'v', status: 'success' }));
const client = makeClient('sk_test');
@@ -25,6 +25,8 @@ const PRELUDE_API_BASE = 'https://api.prelude.dev/v2';
const REQUEST_TIMEOUT_MS = 8000;
/** OTP length — matches the 6-box code UI. Prelude allows 4–8. */
const PRELUDE_CODE_SIZE = 6;
const JA4_SHAPE =
/^[tqd][0-9a-z]{2}[di]\d{4}[0-9a-z]{2}_[0-9a-f]{12}_[0-9a-f]{12}$/;
/**
* Channel Prelude prioritizes for delivery. RCS is far cheaper than SMS, so we
* prefer it by default; Prelude routes to the next reachable channel (SMS) when
@@ -134,6 +136,8 @@ export class PreludeClient extends PuterClient {
ip?: string;
device_id?: string;
user_agent?: string;
/** Forwarded only when it is a JA4 value. */
tls_fingerprint?: string;
dispatch_id?: string;
} = {},
): Promise<{
@@ -168,6 +172,8 @@ export class PreludeClient extends PuterClient {
if (signals.ip) sig.ip = signals.ip;
if (signals.device_id) sig.device_id = signals.device_id;
if (signals.user_agent) sig.user_agent = signals.user_agent;
if (signals.tls_fingerprint && JA4_SHAPE.test(signals.tls_fingerprint))
sig.ja4_fingerprint = signals.tls_fingerprint;
if (Object.keys(sig).length > 0) body.signals = sig;
// dispatch_id is a top-level field, not a member of `signals`.
if (signals.dispatch_id) body.dispatch_id = signals.dispatch_id;
@@ -26,6 +26,7 @@ import { FULL_API_ACCESS } from '../../services/permission/consts.js';
import { tokenClaimsDigest } from '../../../gui/src/helpers/guiToken.js';
import {
createTestUser,
lagReadReplica,
setupPuterTestEnv,
type PuterTestEnv,
} from '../../testUtil.js';
@@ -351,6 +352,98 @@ describe('revoke-own-access-token over HTTP', () => {
expect(await tokenReadStatus(file.uid, readToken)).not.toBe(200);
});
const ACCESS_TOKEN_QUERIES =
/FROM `access_token_permissions`|`access_token_uid` = \?/;
const tokenUidOf = (token: string) =>
(
env.server.services.token.verify('auth', token) as {
token_uid: string;
}
).token_uid;
it('serves a read URL fetched right after minting while the replica lags', async () => {
const owner = env.users.user;
const file = await makeFile(owner);
const replica = await lagReadReplica(env.server, ACCESS_TOKEN_QUERIES);
try {
const readToken = await mintReadToken(owner.token, file.uid);
expect(await tokenReadStatus(file.uid, readToken)).toBe(200);
const cached = await env.server.clients.redis.get(
`perms:token:${tokenUidOf(readToken)}`,
);
expect(cached).toBe(JSON.stringify([`fs:${file.uid}:read`]));
} finally {
replica.catchUp();
}
});
it('mints from a just-created personal API token while the replica lags', async () => {
const owner = env.users.user;
const file = await makeFile(owner);
const replica = await lagReadReplica(env.server, ACCESS_TOKEN_QUERIES);
try {
const pat = await mintPersonalToken(owner);
const readToken = await mintReadToken(pat, file.uid);
expect(await tokenReadStatus(file.uid, readToken)).toBe(200);
} finally {
replica.catchUp();
}
});
it('revokes a just-created token by its uuid while the replica lags', async () => {
const owner = env.users.user;
const file = await makeFile(owner);
const { actor } = await env.server.services.auth.authenticate(
owner.token,
);
const replica = await lagReadReplica(env.server, ACCESS_TOKEN_QUERIES);
let readToken: string;
try {
readToken = await mintReadToken(owner.token, file.uid);
await env.server.services.auth.revokeAccessToken(
actor!,
tokenUidOf(readToken),
);
expect(await tokenReadStatus(file.uid, readToken)).not.toBe(200);
} finally {
replica.catchUp();
}
expect(await tokenReadStatus(file.uid, readToken)).not.toBe(200);
expect(await whoamiStatus(readToken)).toBe(401);
});
it('keeps a revoked token dead while the replica still shows it live', async ({
skip,
}) => {
// The frozen replica is a sqlite copy.
skip(process.env.PUTER_TEST_DB_ENGINE?.toLowerCase() === 'postgres');
const owner = env.users.user;
const file = await makeFile(owner);
const { actor } = await env.server.services.auth.authenticate(
owner.token,
);
const readToken = await mintReadToken(owner.token, file.uid);
const tokenUid = tokenUidOf(readToken);
const replica = await lagReadReplica(env.server, ACCESS_TOKEN_QUERIES, {
mode: 'frozen',
});
try {
await env.server.services.auth.revokeAccessToken(actor!, tokenUid);
expect(await tokenReadStatus(file.uid, readToken)).not.toBe(200);
expect(
await env.server.stores.permission.listAccessTokenPerms(
tokenUid,
),
).toEqual([]);
expect(
await env.server.clients.redis.get(`perms:token:${tokenUid}`),
).toBeNull();
} finally {
replica.catchUp();
}
expect(await whoamiStatus(readToken)).toBe(401);
});
// These three run last in the file's shared `env.users.user`: each one
// revokes a session belonging to `owner.token` / `owner.workerToken`, so
// later tests can't rely on those credentials still being live.
@@ -394,7 +487,10 @@ describe('revoke-own-access-token over HTTP', () => {
it('a revoked GUI session still gets a reauth_token', async () => {
const owner = env.users.user;
const decoded = env.server.services.token.verify('auth', owner.token) as {
const decoded = env.server.services.token.verify(
'auth',
owner.token,
) as {
session_uid: string;
};
await env.server.stores.session.removeByUuid(decoded.session_uid);
@@ -560,7 +656,6 @@ describe('2FA enrollment over HTTP', () => {
expect(enabled.status).toBe(200);
expect((await otpState(account.userId)).enabled).toBe(true);
});
});
/**
@@ -30,6 +30,7 @@
import { installSeatStub } from '../../testFixtures/seatStub.js';
import bcrypt from 'bcrypt';
import { inspect } from 'node:util';
import jwt from 'jsonwebtoken';
import { TOTP } from 'otpauth';
import { v4 as uuidv4, v5 as uuidv5 } from 'uuid';
@@ -721,6 +722,36 @@ describe('AuthController.handleSignup', () => {
expect(audit.ip_fwd).toBe(forged);
});
it('records the request TLS fingerprint on the validate hook and the user row', async () => {
const username = `s_${uniq()}`;
const tlsFingerprint = 't13d1516h2_8daaf6152771_02713d6af862';
let seenTlsFingerprint: unknown;
await withSignupValidateOverride(
(event) => {
seenTlsFingerprint = (event as { tls_fingerprint?: unknown })
.tls_fingerprint;
},
async () => {
await controller.handleSignup(
{
...makeReq({
username,
email: `${username}@test.local`,
password: 'correct-horse-battery',
}),
tlsFingerprint,
},
makeRes(),
);
},
);
expect(seenTlsFingerprint).toBe(tlsFingerprint);
const persisted = await server.stores.user.getByUsername(username);
expect(persisted!.signup_tls_fingerprint).toBe(tlsFingerprint);
});
it('forces phone verification on every signup when always_require_phone_verification is set', async () => {
const cfg = (controller as { config: Record<string, unknown> }).config;
const prev = cfg.always_require_phone_verification;
@@ -4765,14 +4796,24 @@ describe('AuthController.handleSendConfirmEmail', () => {
).rejects.toMatchObject({ statusCode: 403 });
});
it('rotates the email_confirm_code and returns {} on success', async () => {
it('rotates the email_confirm_code and reports the send', async () => {
const { user, actor } = await makeUserAndActor();
const before = await server.stores.user.getById(user.id, {
force: true,
});
const send = vi
.spyOn(server.clients.email, 'send')
.mockResolvedValue({ messageId: 'm' } as never);
const res = makeRes();
await controller.handleSendConfirmEmail(makeReq({}, { actor }), res);
expect(res.body).toEqual({});
try {
await controller.handleSendConfirmEmail(
makeReq({}, { actor }),
res,
);
} finally {
send.mockRestore();
}
expect(res.body).toEqual({ sent: true });
const after = await server.stores.user.getById(user.id, {
force: true,
});
@@ -4782,7 +4823,7 @@ describe('AuthController.handleSendConfirmEmail', () => {
// The resend is the recovery path for a lost code, so it must not fail
// silently either.
it('alarms — and still returns {} — when the resend send throws', async () => {
it('alarms, and still answers 200 with sent: false, when the resend send throws', async () => {
const { actor } = await makeUserAndActor();
const alarm = vi
.spyOn(server.clients.alarm, 'create')
@@ -4797,7 +4838,8 @@ describe('AuthController.handleSendConfirmEmail', () => {
res,
);
// Not the caller's fault; still a 200.
expect(res.body).toEqual({});
expect(res.statusCode).toBe(200);
expect(res.body).toEqual({ sent: false });
const raised = alarm.mock.calls.find(
(c) => c[0] === 'auth:confirmation-email-send-failed:resend',
@@ -7518,6 +7560,46 @@ describe('AuthController.handleSaveAccount', () => {
};
};
it('sends the confirmation code on the account confirmation stream', async () => {
const { user, actor } = await makeTempActor();
await server.stores.user.updateMetadata(user.id, {
emailStream: 'review',
});
const send = vi
.spyOn(server.clients.email, 'send')
.mockResolvedValue(null);
const alarm = vi
.spyOn(server.clients.alarm, 'create')
.mockImplementation(() => undefined);
try {
const username = `s_${uniq()}`;
await controller.handleSaveAccount(
makeReq(
{
username,
email: `${username}@test.local`,
password: 'correct-horse-battery',
},
{ actor },
),
makeRes(),
);
expect(send).toHaveBeenCalledWith(
`${username}@test.local`,
'email_verification_code',
expect.anything(),
{ stream: 'review' },
);
// The dropped send is reported under its own stage.
expect(alarm.mock.calls.map((c) => c[0])).toEqual([
'auth:confirmation-email-send-failed:save-account',
]);
} finally {
send.mockRestore();
alarm.mockRestore();
}
});
it('rejects non-temp accounts with 400', async () => {
const { actor } = await makeUserAndActor();
await expect(
@@ -9509,14 +9591,18 @@ describe('AuthController.handleSignup additional branches', () => {
},
async () => {
const username = `cefail_${uniq()}`;
const res = makeRes();
await controller.handleSignup(
makeReq({
username,
email: `${username}@test.local`,
password: 'correct-horse-battery',
}),
makeRes(),
res,
);
expect(res.body).toMatchObject({
confirmation_email_sent: false,
});
},
);
@@ -9537,6 +9623,93 @@ describe('AuthController.handleSignup additional branches', () => {
});
});
describe('AuthController.handleSignup confirmation email outcome', () => {
const signupWithSend = async (send: () => Promise<unknown>) => {
const spy = vi
.spyOn(server.clients.email, 'send')
.mockImplementation(send as never);
const alarm = vi
.spyOn(server.clients.alarm, 'create')
.mockImplementation(() => undefined);
const res = makeRes();
const username = `ceo_${uniq()}`;
let alarms: string[] = [];
try {
await withSignupValidateOverride(
(event) => {
event.requires_email_confirmation = true;
},
() =>
controller.handleSignup(
makeReq({
username,
email: `${username}@test.local`,
password: 'correct-horse-battery',
}),
res,
),
);
alarms = alarm.mock.calls.map((c) => String(c[0]));
} finally {
spy.mockRestore();
alarm.mockRestore();
}
return {
body: res.body as {
proceed: boolean;
confirmation_email_sent?: boolean;
user: { requires_email_confirmation: boolean };
},
alarms,
};
};
it('reports a send that throws and still completes the signup', async () => {
const { body, alarms } = await signupWithSend(async () => {
throw new Error('refused');
});
expect(body.proceed).toBe(true);
expect(body.confirmation_email_sent).toBe(false);
expect(body.user.requires_email_confirmation).toBeTruthy();
expect(alarms).toContain('auth:confirmation-email-send-failed:signup');
});
it('reports a send the transport accepted', async () => {
const { body } = await signupWithSend(async () => ({ messageId: 'm' }));
expect(body.confirmation_email_sent).toBe(true);
});
it('keeps the confirmation stream out of the returned user details', async () => {
const res = makeRes();
const username = `ces_${uniq()}`;
await withSignupValidateOverride(
(event) => {
(event as Record<string, unknown>).email_stream = 'review';
},
() =>
controller.handleSignup(
makeReq({
username,
email: `${username}@test.local`,
password: 'correct-horse-battery',
}),
res,
),
);
const stored = await server.stores.user.getByUsername(username);
expect(stored!.metadata?.emailStream).toBe('review');
const details = (res.body as { user: { metadata?: object } }).user;
expect(details.metadata ?? {}).not.toHaveProperty('emailStream');
});
it('leaves the field out when no confirmation email is due', async () => {
const res = makeRes();
await controller.handleSignup(makeReq({ is_temp: true }), res);
expect(res.body).toMatchObject({ proceed: true });
expect(res.body).not.toHaveProperty('confirmation_email_sent');
});
});
describe('AuthController.handleSendPassRecoveryEmail additional branches', () => {
it('rejects an invalid email format with 400 (when no username supplied)', async () => {
await expect(
@@ -10632,10 +10805,10 @@ describe('AuthController.loginWait audience binding', () => {
// party holding the session id is somewhere else entirely.
const token = await mintAppToken(actor, OPENER);
const { res, outcome } = await waitWithRelay(
uuidv4(),
{ origin: 'https://evil.test' },
token,
const session = uuidv4();
const [{ res, outcome }] = await runWaits(
[{ session, origin: 'https://evil.test' }],
{ during: () => setToken(session, token) },
);
// Same 408 the empty path returns — a mismatched caller must not be
// able to tell "nothing arrived" from "something arrived for someone
@@ -10711,10 +10884,10 @@ describe('AuthController.loginWait audience binding', () => {
);
const guiToken = (loginRes.body as { token: string }).token;
const { res, outcome } = await waitWithRelay(
uuidv4(),
{ origin: OPENER },
guiToken,
const session = uuidv4();
const [{ res, outcome }] = await runWaits(
[{ session, origin: OPENER }],
{ during: () => setToken(session, guiToken) },
);
expect(outcome).toMatchObject({ statusCode: 408 });
expect(res.body).toBeUndefined();
@@ -10729,14 +10902,406 @@ describe('AuthController.loginWait audience binding', () => {
{ keyid: 'v2' },
);
const { res, outcome } = await waitWithRelay(
uuidv4(),
{ origin: OPENER },
forged,
const session = uuidv4();
const [{ res, outcome }] = await runWaits(
[{ session, origin: OPENER }],
{ during: () => setToken(session, forged) },
);
expect(outcome).toMatchObject({ statusCode: 408 });
expect(res.body).toBeUndefined();
});
/** Post a token to `/login/set` with no wait running. */
const setToken = (session: string, token: string) =>
controller.loginSet(makeReq({ session, auth_token: token }), makeRes());
/**
* Run waits on fake timers so the 10 s window passes without real waiting.
* `during` runs once every wait has subscribed; `now` pins the clock.
*/
const runWaits = async (
calls: Array<{ session: string; origin: string; clientGone?: boolean }>,
opts: { now?: number; during?: () => Promise<unknown> } = {},
) => {
const subscribed = vi.spyOn(eventClient, 'on');
vi.useFakeTimers({
toFake:
opts.now === undefined
? ['setTimeout']
: ['setTimeout', 'Date'],
});
if (opts.now !== undefined) vi.setSystemTime(opts.now);
const tick = () => new Promise((r) => setImmediate(r));
try {
let pending = calls.length;
const runs = calls.map(({ session, origin, clientGone }) => {
const res = makeRes();
const req = makeReq({ session }, { headers: { origin } });
if (clientGone) {
Object.assign(req.socket, { destroyed: true });
}
const outcome = controller
.loginWait(req, res)
.then(
() => 'ok' as const,
(e: unknown) => e,
)
.finally(() => {
pending--;
});
return { res, outcome };
});
const allSubscribed = () =>
calls.every(({ session }) =>
subscribed.mock.calls.some(
([key]) => key === `pubsub.login.${session}`,
),
);
for (let i = 0; i < 200 && pending > 0 && !allSubscribed(); i++) {
await tick();
}
for (let i = 0; i < 20; i++) await tick();
await opts.during?.();
for (let i = 0; i < 200 && pending > 0; i++) {
await tick();
await vi.advanceTimersByTimeAsync(1000);
}
return await Promise.all(
runs.map(async ({ res, outcome }) => ({
res,
outcome: await outcome,
})),
);
} finally {
vi.useRealTimers();
subscribed.mockRestore();
}
};
const waitAlone = async (session: string, origin: string, now?: number) =>
(await runWaits([{ session, origin }], { now }))[0];
it('delivers a token set before the wait subscribes', async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const session = uuidv4();
await setToken(session, token);
const { res, outcome } = await waitAlone(session, OPENER);
expect(outcome).toBe('ok');
expect((res.body as { auth_token: string }).auth_token).toBe(token);
});
it('delivers a token set between one wait timing out and the next', async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const session = uuidv4();
const first = await waitAlone(session, OPENER);
expect(first.outcome).toMatchObject({ statusCode: 408 });
await setToken(session, token);
const { res, outcome } = await waitAlone(session, OPENER);
expect(outcome).toBe('ok');
expect((res.body as { auth_token: string }).auth_token).toBe(token);
});
it('hands a token out only once', async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const session = uuidv4();
await setToken(session, token);
expect((await waitAlone(session, OPENER)).outcome).toBe('ok');
const again = await waitAlone(session, OPENER);
expect(again.outcome).toMatchObject({ statusCode: 408 });
expect(again.res.body).toBeUndefined();
});
it('withholds a stored token from a mismatched Origin and keeps it for the matching one', async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const session = uuidv4();
await setToken(session, token);
const other = await waitAlone(session, 'https://other.test');
expect(other.outcome).toMatchObject({ statusCode: 408 });
expect(other.res.body).toBeUndefined();
const { res, outcome } = await waitAlone(session, OPENER);
expect(outcome).toBe('ok');
expect((res.body as { auth_token: string }).auth_token).toBe(token);
});
it('drops a stored token after a couple of minutes', async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const early = uuidv4();
const late = uuidv4();
await setToken(early, token);
await setToken(late, token);
const start = Date.now();
expect((await waitAlone(early, OPENER, start + 110_000)).outcome).toBe(
'ok',
);
const expired = await waitAlone(late, OPENER, start + 121_000);
expect(expired.outcome).toMatchObject({ statusCode: 408 });
});
it('keeps waiting past a stored token for another app and delivers the real one', async () => {
const { actor } = await makeUserAndActor();
const otherToken = await mintAppToken(actor, 'https://other.test');
const token = await mintAppToken(actor, OPENER);
const session = uuidv4();
await setToken(session, otherToken);
const [{ res, outcome }] = await runWaits(
[{ session, origin: OPENER }],
{
during: () => setToken(session, token),
},
);
expect(outcome).toBe('ok');
expect((res.body as { auth_token: string }).auth_token).toBe(token);
});
it('still relays to a subscribed wait when storing the token fails', async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const session = uuidv4();
const failingSet = vi
.spyOn(server.clients.redis, 'set')
.mockRejectedValueOnce(new Error('unavailable'));
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
try {
let setRes: MockRes | undefined;
const [{ res, outcome }] = await runWaits(
[{ session, origin: OPENER }],
{
during: async () => {
setRes = makeRes();
await controller.loginSet(
makeReq({ session, auth_token: token }),
setRes,
);
},
},
);
expect(setRes?.body).toEqual({ success: true });
expect(outcome).toBe('ok');
expect((res.body as { auth_token: string }).auth_token).toBe(token);
} finally {
failingSet.mockRestore();
warn.mockRestore();
}
});
it('removes the stored copy once a subscribed wait takes the token', async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const session = uuidv4();
const { outcome } = await waitWithRelay(
session,
{ origin: OPENER },
token,
);
expect(outcome).toBe('ok');
const later = await waitAlone(session, OPENER);
expect(later.outcome).toMatchObject({ statusCode: 408 });
});
it('hands a stored token to only one of two concurrent waits', async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const session = uuidv4();
await setToken(session, token);
const results = await runWaits([
{ session, origin: OPENER },
{ session, origin: OPENER },
]);
const outcomes = results.map((r) => r.outcome);
expect(outcomes.filter((o) => o === 'ok')).toHaveLength(1);
expect(outcomes.filter((o) => o !== 'ok')).toEqual([
expect.objectContaining({ statusCode: 408 }),
]);
});
it('relays a token to every wait already subscribed', async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const session = uuidv4();
const results = await runWaits(
[
{ session, origin: OPENER },
{ session, origin: OPENER },
],
{ during: () => setToken(session, token) },
);
expect(results.map((r) => r.outcome)).toEqual(['ok', 'ok']);
for (const { res } of results) {
expect((res.body as { auth_token: string }).auth_token).toBe(token);
}
});
it('keeps the stored copy when the subscribed wait has lost its client', async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const session = uuidv4();
await runWaits([{ session, origin: OPENER, clientGone: true }], {
during: () => setToken(session, token),
});
const { res, outcome } = await waitAlone(session, OPENER);
expect(outcome).toBe('ok');
expect((res.body as { auth_token: string }).auth_token).toBe(token);
});
it('still relays to a subscribed wait when the stored copy cannot be read or removed', async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const session = uuidv4();
const failingGet = vi
.spyOn(server.clients.redis, 'get')
.mockRejectedValue(new Error('unavailable'));
const failingEval = vi
.spyOn(server.clients.redis, 'eval')
.mockRejectedValue(new Error('unavailable'));
try {
const [{ res, outcome }] = await runWaits(
[{ session, origin: OPENER }],
{ during: () => setToken(session, token) },
);
expect(outcome).toBe('ok');
expect((res.body as { auth_token: string }).auth_token).toBe(token);
} finally {
failingGet.mockRestore();
failingEval.mockRestore();
}
});
it('answers a subscribed wait when a second token is set right after the first', async () => {
const { actor } = await makeUserAndActor();
const first = await mintAppToken(actor, OPENER);
// A later `iat` makes the second token differ from the first.
await new Promise((r) => setTimeout(r, 1100));
const second = await mintAppToken(actor, OPENER);
expect(second).not.toBe(first);
const session = uuidv4();
const [{ res, outcome }] = await runWaits(
[{ session, origin: OPENER }],
{
during: async () => {
await setToken(session, first);
await setToken(session, second);
},
},
);
expect(outcome).toBe('ok');
expect((res.body as { auth_token: string }).auth_token).toBe(first);
});
it('keeps a subscribed wait listening past a token for another app', async () => {
const { actor } = await makeUserAndActor();
const otherToken = await mintAppToken(actor, 'https://other.test');
const token = await mintAppToken(actor, OPENER);
const session = uuidv4();
const [{ res, outcome }] = await runWaits(
[{ session, origin: OPENER }],
{
during: async () => {
await setToken(session, otherToken);
await setToken(session, token);
},
},
);
expect(outcome).toBe('ok');
expect((res.body as { auth_token: string }).auth_token).toBe(token);
});
it("keeps a stored token when a later set carries another app's token or an unsigned one", async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const otherToken = await mintAppToken(actor, 'https://other.test');
const session = uuidv4();
await setToken(session, token);
await setToken(session, otherToken);
await setToken(session, 'x'.repeat(100_000));
const { res, outcome } = await waitAlone(session, OPENER);
expect(outcome).toBe('ok');
expect((res.body as { auth_token: string }).auth_token).toBe(token);
});
it('stores only a token this server signed for an app', async () => {
const { actor } = await makeUserAndActor();
const real = await mintAppToken(actor, OPENER);
const forged = jwt.sign(
jwt.decode(real) as object,
'not-the-server-secret',
{ keyid: 'v2' },
);
const set = vi.spyOn(server.clients.redis, 'set');
try {
const session = uuidv4();
await setToken(session, forged);
await setToken(session, 'x'.repeat(100_000));
expect(set).not.toHaveBeenCalled();
await setToken(session, real);
expect(set).toHaveBeenCalledTimes(1);
} finally {
set.mockRestore();
}
});
it('leaves the token out of the warning when storing it fails', async () => {
const { actor } = await makeUserAndActor();
const token = await mintAppToken(actor, OPENER);
const session = uuidv4();
const failure = Object.assign(new Error('ERR unavailable'), {
command: { name: 'set', args: [session, token, 'EX', '120'] },
});
const failingSet = vi
.spyOn(server.clients.redis, 'set')
.mockRejectedValueOnce(failure);
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
try {
const res = makeRes();
await controller.loginSet(
makeReq({ session, auth_token: token }),
res,
);
expect(res.body).toEqual({ success: true });
expect(warn).toHaveBeenCalled();
const logged = warn.mock.calls
.flat()
.map((arg) => inspect(arg, { depth: 10 }))
.join(' ');
expect(logged).not.toContain(token);
} finally {
failingSet.mockRestore();
warn.mockRestore();
}
});
it('answers 400 when auth_token is not a string', async () => {
await expect(
controller.loginSet(
makeReq({ session: uuidv4(), auth_token: { nested: true } }),
makeRes(),
),
).rejects.toMatchObject({ statusCode: 400, legacyCode: 'bad_request' });
});
});
// -- Batch grant / revoke + cross-app data grants -----------------------
@@ -11093,3 +11658,100 @@ describe('AuthController — a seat address nobody confirmed', () => {
expect(await emailOf(seat.id)).toBeNull();
});
});
describe('AuthController account mail streams', () => {
const spySend = () =>
vi.spyOn(server.clients.email, 'send').mockResolvedValue(null);
it('sends password recovery on the auth stream', async () => {
const { user } = await makeUserAndActor();
const send = spySend();
try {
await controller.handleSendPassRecoveryEmail(
makeReq({ email: user.email! }),
makeRes(),
);
expect(send).toHaveBeenCalledWith(
user.email,
'email_password_recovery',
expect.anything(),
{ stream: 'auth' },
);
} finally {
send.mockRestore();
}
});
it('sends the password change notice on the auth stream', async () => {
const { user, actor } = await makeUserAndActor();
const req = makeReq({ new_pass: 'a-fresh-password-456' }, { actor });
(req as unknown as { userProtected: unknown }).userProtected = { user };
const send = spySend();
try {
await controller.handleChangePassword(req, makeRes());
expect(send).toHaveBeenCalledWith(
user.email,
'password_change_notification',
expect.anything(),
{ stream: 'auth' },
);
} finally {
send.mockRestore();
}
});
it('sends both change-email messages on the auth stream', async () => {
const { user, actor } = await makeUserAndActor();
const newEmail = `cs_${uniq()}@test.local`;
const send = spySend();
const sendRaw = vi
.spyOn(server.clients.email, 'sendRaw')
.mockResolvedValue(null);
try {
await controller.handleChangeEmail(
makeReq({ new_email: newEmail }, { actor }),
makeRes(),
);
expect(send).toHaveBeenCalledWith(
newEmail,
'email_verification_link',
expect.anything(),
{ stream: 'auth' },
);
expect(sendRaw).toHaveBeenCalledWith(
expect.objectContaining({ to: user.email, stream: 'auth' }),
);
} finally {
send.mockRestore();
sendRaw.mockRestore();
}
});
it('sends the 2FA enable and disable notices on the auth stream', async () => {
const { user, actor } = await makeUserAndActor({ email_confirmed: 1 });
const send = spySend();
try {
const { secret } = await setup2fa(actor);
await enable2fa(actor, { code: liveTotp(user.username, secret) });
expect(send).toHaveBeenCalledWith(
user.email,
'enabled_2fa',
{ username: user.username },
{ stream: 'auth' },
);
await controller.handleDisable2fa(
makeReq({ code: liveTotp(user.username, secret) }, { actor }),
makeRes(),
);
expect(send).toHaveBeenCalledWith(
user.email,
'disabled_2fa',
{ username: user.username },
{ stream: 'auth' },
);
} finally {
send.mockRestore();
}
});
});
+174 -75
View File
@@ -71,6 +71,7 @@ import { sessionCookieFlags } from '../../util/cookieFlags.js';
import { cleanEmail, isStorableEmail } from '../../util/email.js';
import { isGodmodeApp } from '../../util/godmodeApps.js';
import { parsePhone } from '../../util/phone.js';
import { deleteIfEqual } from '../../util/redisLock.js';
import {
bonusCodeInvalidError,
checkSignupBonus,
@@ -143,6 +144,12 @@ const DISPATCH_ID_MAX_LENGTH = 128;
// One name for the flag, so the write and the read cannot drift apart.
const APP_AUTHENTICATED_FLAG = 'flag:app-is-authenticated';
// Long enough to span the gap between one `/login/wait` and the client's next.
const LOGIN_RELAY_TTL_SECONDS = 120;
// Keyed by app too, so a token for one app never replaces another app's.
const loginRelayKey = (session: string, appUid: string) =>
`login:relay:${session}:${appUid}`;
// -- Post-login route limits -----------------------------------------
//
// The credential legs above (login, signup, recovery, confirmation) each
@@ -345,36 +352,43 @@ export class AuthController extends PuterController {
const expectedAppUid =
await this.services.auth.appUidFromOrigin(reqOrigin);
const { resolve, promise } = Promise.withResolvers<void>();
let token: string | null = null;
const listener = (_key: string, value: { authtoken: string }) => {
token = value.authtoken;
resolve();
};
this.clients.event.on(`pubsub.login.${session}`, listener);
const timeout = new Promise<void>((resolve) =>
setTimeout(resolve, 10000),
);
await Promise.race([promise, timeout]);
this.clients.event.off(`pubsub.login.${session}`, listener);
if (!token) {
throw new HttpError(408, 'Request timeout.', {
legacyCode: 'request_timeout',
});
}
const relayKey = loginRelayKey(session, expectedAppUid);
const { resolve, promise } = Promise.withResolvers<string>();
// Audience check. The postMessage hand-off this relay stands in for
// is origin-bound for free — it posts with `targetOrigin`, so a page
// can only ever receive a token minted for *itself*. Delivering
// server-side dropped that binding; this restores it. Without it a
// popup talked into minting for app X (see `trustsOpenerOriginParam`
// in the GUI) hands X's token to whoever holds the session id.
if (!this.#tokenIsForApp(token, expectedAppUid)) {
// Deliberately the same 408 the no-token path returns: a caller
// learns only that nothing arrived for them, not that a token
// for a different app went past.
// server-side dropped that binding; this restores it. Any other token
// is passed over and the wait runs on, so a caller learns only that
// nothing arrived for them.
const listener = (_key: string, value: { authtoken: string }) => {
if (this.#relayedTokenAppUid(value.authtoken) === expectedAppUid) {
resolve(value.authtoken);
}
};
this.clients.event.on(`pubsub.login.${session}`, listener);
let token: string | null;
try {
// Read after subscribing, so a token set before this wait is still delivered.
token = await this.#takeStoredLoginToken(relayKey, expectedAppUid);
if (!token) {
const timeout = new Promise<null>((resolve) =>
setTimeout(resolve, 10000, null),
);
token = await Promise.race([promise, timeout]);
// A wait whose client went away leaves the copy for the next one.
if (token && !req.socket?.destroyed) {
await deleteIfEqual(
this.clients.redis,
relayKey,
token,
).catch(() => false);
}
}
} finally {
this.clients.event.off(`pubsub.login.${session}`, listener);
}
if (!token) {
throw new HttpError(408, 'Request timeout.', {
legacyCode: 'request_timeout',
});
@@ -386,24 +400,46 @@ export class AuthController extends PuterController {
}
/**
* Whether a relayed token is an app-under-user token minted for
* `expectedAppUid`. Verifies the signature — an unverified decode would let
* a caller relay a token whose claims it wrote itself.
* Take the copy `/login/set` kept for this app, once. Null when there is
* none, another wait took it, or Redis fails; the wait then just listens.
*/
#tokenIsForApp(token: string, expectedAppUid: string): boolean {
async #takeStoredLoginToken(
relayKey: string,
expectedAppUid: string,
): Promise<string | null> {
try {
const kept = await this.clients.redis.get(relayKey);
if (!kept || this.#relayedTokenAppUid(kept) !== expectedAppUid) {
return null;
}
const took = await deleteIfEqual(
this.clients.redis,
relayKey,
kept,
);
return took ? kept : null;
} catch {
return null;
}
}
/**
* The app a relayed app-under-user token was minted for, or null. Verifies
* the signature — an unverified decode would let a caller relay a token
* whose claims it wrote itself.
*/
#relayedTokenAppUid(token: string): string | null {
try {
const payload = this.services.token.verify<{
type?: string;
app_uid?: string;
}>('auth', token);
return (
payload.type === 'app-under-user' &&
!!payload.app_uid &&
payload.app_uid === expectedAppUid
);
return payload.type === 'app-under-user' && payload.app_uid
? payload.app_uid
: null;
} catch {
// Malformed, expired, or signed with a key we don't hold.
return false;
return null;
}
}
@Post('/login/set', {
@@ -418,12 +454,34 @@ export class AuthController extends PuterController {
})
async loginSet(req: Request, res: Response) {
const { session, auth_token } = req.body ?? {};
if (!session || !auth_token || !validateUuid(session)) {
if (
!session ||
typeof auth_token !== 'string' ||
!auth_token ||
!validateUuid(session)
) {
throw new HttpError(400, 'session and auth_token are required.', {
legacyCode: 'bad_request',
});
}
// Only a token a wait could deliver is kept: one this server signed for an app.
const appUid = this.#relayedTokenAppUid(auth_token);
if (appUid) {
try {
await this.clients.redis.set(
loginRelayKey(session, appUid),
auth_token,
'EX',
LOGIN_RELAY_TTL_SECONDS,
);
} catch (e) {
console.warn(
'[login-set] storing the relayed token failed:',
e instanceof Error ? e.message : String(e),
);
}
}
this.clients.event.emit(
`pubsub.login.${session}`,
{
@@ -882,14 +940,19 @@ export class AuthController extends PuterController {
if (!bonusCode) throw bonusCodeInvalidError();
}
const user = await this.services.signup.signup({
req,
body,
isTemp: is_temp,
fingerprint,
bonusCode,
const { user, confirmationEmailSent } =
await this.services.signup.signup({
req,
body,
isTemp: is_temp,
fingerprint,
bonusCode,
});
await this.#completeLogin(req, res, user, undefined, {
...(confirmationEmailSent !== undefined
? { confirmation_email_sent: confirmationEmailSent }
: {}),
});
await this.#completeLogin(req, res, user);
}
/**
@@ -1028,10 +1091,12 @@ export class AuthController extends PuterController {
email_confirm_code: code,
});
await this.services.signup.sendConfirmationEmail(user, 'resend', {
code,
});
res.json({});
const sent = await this.services.signup.sendConfirmationEmail(
user,
'resend',
{ code },
);
res.json({ sent });
}
@Post('/confirm-email', {
@@ -1445,6 +1510,7 @@ export class AuthController extends PuterController {
ip,
device_id: req.deviceFingerprint ?? undefined,
user_agent: userAgent,
tls_fingerprint: req.tlsFingerprint,
dispatch_id: dispatchId,
},
);
@@ -2019,6 +2085,7 @@ export class AuthController extends PuterController {
user.email,
'email_password_recovery',
{ link },
{ stream: 'auth' },
);
} catch (e) {
console.warn('[send-pass-recovery-email] send failed:', e);
@@ -2282,6 +2349,7 @@ export class AuthController extends PuterController {
{
username: user.username,
},
{ stream: 'auth' },
);
} catch (e) {
console.warn('[change-password] notification send failed:', e);
@@ -2507,6 +2575,7 @@ export class AuthController extends PuterController {
new_email,
'email_verification_link',
{ link },
{ stream: 'auth' },
);
} catch (e) {
console.warn('[change-email] new-address email failed:', e);
@@ -2514,15 +2583,8 @@ export class AuthController extends PuterController {
// Notify the old address too
if (user.email) {
try {
await (
this.clients.email as unknown as {
sendRaw: (opts: {
to: string;
subject: string;
text: string;
}) => Promise<unknown>;
}
).sendRaw({
await this.clients.email.sendRaw({
stream: 'auth',
to: user.email,
subject: 'Your Puter email change was requested',
text: `A change to ${new_email} was requested on your account. If this wasn't you, please contact support.`,
@@ -2811,18 +2873,11 @@ export class AuthController extends PuterController {
username,
});
// Send confirmation email
if (this.clients.email) {
try {
await this.clients.email.send(
email,
'email_verification_code',
{ code: email_confirm_code },
);
} catch (e) {
console.warn('[save-account] confirmation email failed:', e);
}
}
await this.services.signup.sendConfirmationEmail(
{ ...user, username, email },
'save-account',
{ code: email_confirm_code },
);
try {
this.clients.event?.emit(
@@ -4272,9 +4327,12 @@ export class AuthController extends PuterController {
if (this.clients.email && user.email) {
try {
await this.clients.email.send(user.email, 'enabled_2fa', {
username: user.username,
});
await this.clients.email.send(
user.email,
'enabled_2fa',
{ username: user.username },
{ stream: 'auth' },
);
} catch (e) {
console.warn('[configure-2fa] email send failed:', e);
}
@@ -4348,9 +4406,12 @@ export class AuthController extends PuterController {
if (this.clients.email && user.email) {
try {
await this.clients.email.send(user.email, 'disabled_2fa', {
username: user.username,
});
await this.clients.email.send(
user.email,
'disabled_2fa',
{ username: user.username },
{ stream: 'auth' },
);
} catch (e) {
console.warn('[disable-2fa] email send failed:', e);
}
@@ -4426,6 +4487,34 @@ export class AuthController extends PuterController {
res.json({ token: guiToken });
}
@Post('/auth/first-party-session', {
subdomain: 'api',
rateLimit: SESSION_LIMIT,
// Hands out a session, so only the GUI's own page script may ask, and
// only as the user's sign-in itself.
guiOriginOnly: true,
requireUserActor: true,
})
async handleFirstPartySession(req: Request, res: Response): Promise<void> {
const { origin } = req.body ?? {};
if (typeof origin !== 'string' || !origin) {
throw new HttpError(400, 'Missing `origin`', {
legacyCode: 'bad_request',
});
}
const token = await this.services.auth.createFirstPartySessionToken(
req.actor!,
origin,
{
ip: req.ip || req.socket?.remoteAddress,
user_agent: req.headers?.['user-agent'],
origin: req.headers?.origin,
host: req.headers?.host,
},
);
res.json({ token });
}
@Get('/session/sync-cookie', {
rateLimit: SESSION_LIMIT,
// Installs the session cookie. Only page script on our own origin
@@ -4450,6 +4539,14 @@ export class AuthController extends PuterController {
res.status(400).end();
return;
}
// A session handed to a first-party page never gets a cookie.
if (req.actor.session.parentUid) {
throw new HttpError(
403,
'This session cannot install a session cookie',
{ legacyCode: 'forbidden' },
);
}
// As in `/get-gui-token`: the actor is the user's row as auth read it.
const sessionToken = this.services.auth.createSessionTokenForSession(
req.actor.user as UserRow,
@@ -4766,6 +4863,7 @@ export class AuthController extends PuterController {
res: Response,
user: UserRow,
oidcContinuation?: unknown,
extra: { confirmation_email_sent?: boolean } = {},
): Promise<void> {
const redirectUrl =
oidcContinuation === undefined
@@ -4800,6 +4898,7 @@ export class AuthController extends PuterController {
token: gui_token,
user: details,
...(redirectUrl ? { redirectUrl } : {}),
...extra,
});
}
}
@@ -0,0 +1,282 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import type { IConfig } from '../../types.js';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
const PAGE = 'https://desk.first-party.test';
// Seeded with godmode on.
const DEV_CENTER = 'app-0b37f054-07d4-4627-8765-11bd23e889d4';
/**
* One of the deployment's own pages, served apart from the GUI, signs in with a
* `web` session of its own under the user's sign-in.
*/
describe('first-party page sessions', () => {
let env: PuterTestEnv;
let userId = 0;
beforeAll(async () => {
env = await setupPuterTestEnv({
first_party_session_origins: [`${PAGE}/`],
} as IConfig);
const row = await env.server.stores.user.getByUsername(
env.users.user.username,
);
userId = row!.id;
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const call = (
method: string,
path: string,
token: string,
body?: unknown,
headers: Record<string, string> = {},
) =>
fetch(new URL(path, env.apiOrigin), {
method,
headers: {
'content-type': 'application/json',
authorization: `Bearer ${token}`,
...headers,
},
...(body === undefined ? {} : { body: JSON.stringify(body) }),
});
/** A sign-in of its own, so revoking it leaves the shared one up. */
const signIn = async () => {
const user = await env.server.stores.user.getById(userId);
const { gui_token, session } =
await env.server.services.auth.createSessionToken(user!);
return { token: gui_token, uuid: session.uuid as string };
};
const mint = async (token: string, origin = PAGE) => {
const res = await call('POST', '/auth/first-party-session', token, {
origin,
});
expect(res.status).toBe(200);
return ((await res.json()) as { token: string }).token;
};
const actorOf = async (token: string) =>
(await env.server.services.auth.authenticate(token)).actor;
const sessions = async (token: string) => {
const res = await call('GET', '/auth/list-sessions', token);
expect(res.status).toBe(200);
return (await res.json()) as Array<Record<string, unknown>>;
};
it('hands the page a session of its own under the sign-in', async () => {
const root = await signIn();
const token = await mint(root.token);
const actor = await actorOf(token);
expect(actor?.app ?? null).toBeNull();
expect(actor?.accessToken ?? null).toBeNull();
expect(actor?.session).toMatchObject({
kind: 'web',
parentUid: root.uuid,
});
expect(actor?.session?.uid).not.toBe(root.uuid);
const row = (await sessions(root.token)).find(
(s) => s.uuid === actor!.session!.uid,
);
expect(row).toMatchObject({
kind: 'web',
parent_session_id: root.uuid,
first_party_origin: PAGE,
current: false,
});
});
it('reuses the session on a second ask', async () => {
const root = await signIn();
const first = await actorOf(await mint(root.token));
const second = await actorOf(await mint(root.token));
expect(second?.session?.uid).toBe(first?.session?.uid);
});
it('acts as the account: launches apps, godmode ones included', async () => {
const root = await signIn();
const token = await mint(root.token);
const res = await call('POST', '/auth/get-user-app-token', token, {
app_uid: DEV_CENTER,
});
expect(res.status).toBe(200);
expect(await res.json()).toMatchObject({ godmode: true });
});
it('refuses an origin that is not listed', async () => {
const root = await signIn();
const res = await call(
'POST',
'/auth/first-party-session',
root.token,
{
origin: 'https://elsewhere.test',
},
);
expect(res.status).toBe(403);
});
it('refuses a page on another origin, the listed page included', async () => {
const root = await signIn();
for (const origin of [PAGE, 'https://elsewhere.test']) {
const res = await call(
'POST',
'/auth/first-party-session',
root.token,
{ origin: PAGE },
{ origin },
);
expect(res.status).toBe(403);
}
});
it('refuses app tokens, access tokens and a first-party session', async () => {
const root = await signIn();
const page = await mint(root.token);
const app = (await (
await call('POST', '/auth/get-user-app-token', root.token, {
origin: 'https://some-app.test',
})
).json()) as { token: string };
const godmode = (await (
await call('POST', '/auth/get-user-app-token', root.token, {
app_uid: DEV_CENTER,
})
).json()) as { token: string };
for (const token of [page, app.token, godmode.token]) {
const res = await call('POST', '/auth/first-party-session', token, {
origin: PAGE,
});
expect(res.status).toBe(403);
}
});
it('never gets a session cookie', async () => {
const root = await signIn();
const page = await mint(root.token);
const syncCookie = (token: string) =>
fetch(new URL('/session/sync-cookie', env.origin), {
headers: { authorization: `Bearer ${token}` },
});
expect((await syncCookie(root.token)).status).toBe(204);
const res = await syncCookie(page);
expect(res.status).toBe(403);
expect(res.headers.get('set-cookie')).toBeNull();
});
it('ends with the sign-in it was handed out under', async () => {
const root = await signIn();
const page = await mint(root.token);
expect(await actorOf(page)).toBeDefined();
await env.server.services.auth.revokeSession(root.uuid);
const result = await env.server.services.auth.authenticate(page);
expect(result.actor).toBeUndefined();
expect(result.reauth?.reason).toBe('session_revoked');
});
it('takes the tokens minted under it along when its sign-in ends', async () => {
const root = await signIn();
const page = await mint(root.token);
const godmode = (await (
await call('POST', '/auth/get-user-app-token', page, {
app_uid: DEV_CENTER,
})
).json()) as { token: string };
const scoped = (await (
await call('POST', '/auth/create-access-token', godmode.token, {
permissions: ['service:foo:ii:read'],
expiresIn: '7d',
})
).json()) as { token: string };
expect(await actorOf(scoped.token)).toBeDefined();
await env.server.services.auth.revokeSession(root.uuid);
for (const token of [godmode.token, scoped.token]) {
expect(
(await env.server.services.auth.authenticate(token)).actor,
).toBeUndefined();
}
});
it('is revoked on its own without ending the sign-in', async () => {
const root = await signIn();
const page = await mint(root.token);
const pageUuid = (await actorOf(page))!.session!.uid;
await env.server.services.auth.revokeSession(pageUuid);
expect(
(await env.server.services.auth.authenticate(page)).actor,
).toBeUndefined();
expect(await actorOf(root.token)).toBeDefined();
});
it('keeps the whole current sign-in when revoking all others', async () => {
const root = await signIn();
const page = await mint(root.token);
const other = await signIn();
const otherPage = await mint(other.token);
// Asked from the page: its sign-in stays, and so does the page.
await env.server.services.auth.revokeAllSessions(
(await actorOf(page))!,
);
expect(await actorOf(page)).toBeDefined();
expect(await actorOf(root.token)).toBeDefined();
expect(
(await env.server.services.auth.authenticate(other.token)).actor,
).toBeUndefined();
expect(
(await env.server.services.auth.authenticate(otherPage)).actor,
).toBeUndefined();
// Asked from the sign-in: the page under it stays too.
await env.server.services.auth.revokeAllSessions(
(await actorOf(root.token))!,
);
expect(await actorOf(page)).toBeDefined();
});
it('is not counted as a sign-in', async () => {
const root = await signIn();
const before = await env.server.stores.session.listSignIns(userId, {
limit: 100,
});
await mint(root.token);
const after = await env.server.stores.session.listSignIns(userId, {
limit: 100,
});
expect(after.length).toBe(before.length);
});
});
+2 -1
View File
@@ -18,6 +18,7 @@
*/
import type { Response } from 'express';
import { onResponseClose } from '../../core/http/onResponseClose.js';
// Under the concurrency store's one-hour orphan TTL.
const SLOT_RENEW_MS = 10 * 60 * 1000;
@@ -46,7 +47,7 @@ export class CallSlot {
this.#maybeRelease();
};
res.once('finish', responded);
res.once('close', responded);
onResponseClose(res, responded);
if (res.destroyed || res.writableFinished) responded();
if (!this.#released && this.#work.size > 0) this.#startRenewing();
}
+3 -1
View File
@@ -1850,7 +1850,9 @@ export class FSController extends PuterController {
}
async #assertCanDedupeCreate(actor: Actor, targetPath: string) {
const existing = await this.stores.fsEntry.getEntryByPath(targetPath);
const existing = await this.stores.fsEntry.getEntryByPath(targetPath, {
foldedNames: true,
});
if (!existing) return;
const parent = pathPosix.dirname(targetPath);
await this.#assertAccess(
@@ -168,6 +168,27 @@ describe('legacy file downloads over HTTP', () => {
},
);
it('GET /token-read types a file by its name, and one with no known extension as a byte stream', async () => {
const typed = await requestFor(
'/token-read',
await writeFile('notes.txt', Buffer.from('hello')),
);
expect(typed.headers.get('content-type')).toBe(
'text/plain; charset=utf-8',
);
for (const name of ['README', 'archive.unknownext']) {
const response = await requestFor(
'/token-read',
await writeFile(name, Buffer.from('hello')),
);
expect(response.status).toBe(200);
expect(response.headers.get('content-type')).toBe(
'application/octet-stream',
);
}
});
it('GET /read answers byte_count on an empty file with an empty 200', async () => {
const file = await writeFile('empty-byte-count.bin', Buffer.alloc(0));
for (const query of ['byte_count=512', 'offset=0']) {
@@ -765,8 +765,10 @@ export class LegacyFSController extends PuterController {
normalizedTarget,
);
if (dedupeName) {
const existing =
await this.stores.fsEntry.getEntryByPath(normalizedTarget);
const existing = await this.stores.fsEntry.getEntryByPath(
normalizedTarget,
{ foldedNames: true },
);
if (existing) {
const parent = pathPosix.dirname(normalizedTarget);
await assertAccess(
@@ -1320,7 +1322,7 @@ export class LegacyFSController extends PuterController {
if (options.realMime) {
res.setHeader(
'Content-Type',
contentTypeFromMime(entry.name) as string,
contentTypeFromMime(entry.name) || 'application/octet-stream',
);
} else {
res.setHeader('Content-Type', 'application/octet-stream');
@@ -277,4 +277,124 @@ describe('writes inside a folder shared with write', () => {
expect(JSON.stringify(res.json)).not.toContain(s.realDir);
}
}, 180_000);
describe('a file the recipient opened with an app', () => {
const makeApp = (ownerUserId: number) =>
(
env.server.stores.app.create as unknown as (
fields: Record<string, unknown>,
opts: { ownerUserId: number },
) => Promise<{ uid: string; id: number }>
)(
{
name: `sfw-app-${Math.random().toString(36).slice(2, 9)}`,
title: 'Shared folder app',
index_url: `https://sfw-${Math.random().toString(36).slice(2, 9)}.test/`,
},
{ ownerUserId },
);
/** Opens `note.txt` with a new app, as a launch does. */
const openWithApp = async (
s: Awaited<ReturnType<typeof shareAFolder>>,
) => {
const recipientId = (await env.server.stores.user.getByUsername(
s.recipient.username,
))!.id;
const app = await makeApp(recipientId);
const signed = await post('/sign', s.recipient.token, {
items: [{ uid: s.noteUid, action: 'write' }],
app_uid: app.uid,
});
expect(signed.status).toBe(200);
return { app, appToken: signed.json!.token as string };
};
const read = async (token: string, query: Record<string, string>) => {
const res = await fetch(
new URL(`/read?${new URLSearchParams(query)}`, env.apiOrigin),
{ headers: { authorization: `Bearer ${token}` } },
);
return { status: res.status, text: await res.text() };
};
const write = (token: string, path: string, content: string) =>
post('/fs/write', token, {
fileMetadata: {
path,
size: content.length,
contentType: 'text/plain',
overwrite: true,
},
fileContent: content,
});
it('reads and saves it by path and uid', async () => {
const s = await shareAFolder();
const { appToken } = await openWithApp(s);
const masked = `${s.maskedDir}/note.txt`;
const real = `${s.realDir}/note.txt`;
expect((await write(appToken, masked, 'one')).status).toBe(200);
expect(await read(appToken, { uid: s.noteUid })).toEqual({
status: 200,
text: 'one',
});
expect((await write(appToken, real, 'two')).status).toBe(200);
expect(await read(appToken, { file: real })).toEqual({
status: 200,
text: 'two',
});
expect(
(await post('/stat', appToken, { uid: s.noteUid })).status,
).toBe(200);
}, 180_000);
it('only reads it once the share drops to read', async () => {
const s = await shareAFolder();
const { appToken } = await openWithApp(s);
const dir = await post('/stat', s.owner.token, { path: s.realDir });
const reshared = await post('/share', s.owner.token, {
recipients: [s.recipient.username],
items: [{ uid: dir.json!.uid }],
mode: 'read',
});
expect(reshared.status).toBe(200);
expect((await read(appToken, { uid: s.noteUid })).status).toBe(200);
expect(
(await write(appToken, `${s.maskedDir}/note.txt`, 'x')).status,
).toBe(403);
}, 180_000);
it('gives nothing to another app of the recipient', async () => {
const s = await shareAFolder();
const { appToken } = await openWithApp(s);
const recipientId = (await env.server.stores.user.getByUsername(
s.recipient.username,
))!.id;
const other = await makeApp(recipientId);
const minted = await post(
'/auth/get-user-app-token',
s.recipient.token,
{ app_uid: other.uid },
);
expect(minted.status).toBe(200);
const otherToken = minted.json!.token as string;
// The same requests the opened app gets through.
const real = `${s.realDir}/note.txt`;
expect((await read(appToken, { uid: s.noteUid })).status).toBe(200);
expect((await write(appToken, real, 'one')).status).toBe(200);
expect((await read(otherToken, { uid: s.noteUid })).status).toBe(
404,
);
expect((await write(otherToken, real, 'x')).status).toBe(404);
expect(await read(appToken, { uid: s.noteUid })).toEqual({
status: 200,
text: 'one',
});
}, 180_000);
});
});
+2 -1
View File
@@ -26,6 +26,7 @@
*/
import type { Response } from 'express';
import { onResponseClose } from '../../core/http/onResponseClose.js';
import type { UsageDetails } from '../../drivers/ai-chat/types.js';
import { promoteStopForToolCalls } from '../../drivers/ai-chat/utils/stopReason.js';
import { anthropicUsage } from './anthropicWire.js';
@@ -79,7 +80,7 @@ export const startSse = (res: Response, opts: { ping?: boolean } = {}): Sse => {
timer.unref?.();
}
res.once('close', () => {
onResponseClose(res, () => {
ended = true;
if (timer) clearInterval(timer);
});
@@ -301,6 +301,7 @@ export class SystemController extends PuterController {
if (this.clients.email && req.actor.user?.email) {
try {
await this.clients.email.sendRaw({
stream: 'notify',
to: supportEmail,
replyTo: req.actor.user.email,
subject: `Contact from ${req.actor.user.username}`,
@@ -547,6 +547,7 @@ describe('SystemController POST /contactUs — attachments', () => {
expect(sendRaw).toHaveBeenCalledTimes(1);
const sent = sendRaw.mock.calls[0][0] as {
stream: string;
text: string;
attachments: Array<{
filename: string;
@@ -555,6 +556,7 @@ describe('SystemController POST /contactUs — attachments', () => {
contentDisposition: string;
}>;
};
expect(sent.stream).toBe('notify');
expect(sent.attachments).toHaveLength(1);
expect(sent.attachments[0].filename).toBe('payload.png');
expect(sent.attachments[0].contentType).toBe('image/png');
@@ -0,0 +1,210 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { GetObjectCommand } from '@aws-sdk/client-s3';
import { randomBytes } from 'node:crypto';
import { Readable } from 'node:stream';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import {
RESPONSE_CLOSE_LISTENER_BUDGET,
setupPuterTestEnv,
TEST_USER_CREDENTIALS,
trackResponseCloseListeners,
type PuterTestEnv,
} from '../../testUtil.js';
/** File downloads over WebDAV: the bytes, ranges, and either end failing. */
describe('WebDAV GET over HTTP', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 60_000);
afterAll(async () => {
await env?.shutdown();
});
const dav = (path: string, init: RequestInit = {}) => {
const port = new URL(env.apiOrigin).port;
const auth = Buffer.from(
`${TEST_USER_CREDENTIALS.username}:${TEST_USER_CREDENTIALS.password}`,
).toString('base64');
return fetch(`http://dav.puter.localhost:${port}${encodeURI(path)}`, {
...init,
headers: {
Authorization: `Basic ${auth}`,
// Uncompressed, so the bytes on the wire are the stored ones.
'Accept-Encoding': 'identity',
...(init.headers as Record<string, string> | undefined),
},
});
};
const writeFile = async (name: string, content: Buffer) => {
const { username } = env.users.user;
const user = await env.server.stores.user.getByUsername(username);
const path = `/${username}/Documents/${name}`;
const { fsEntry } = await env.server.services.fs.write(user!.id, {
fileMetadata: {
path,
size: content.byteLength,
contentType: 'application/octet-stream',
},
fileContent: content,
});
return { path, uuid: fsEntry.uuid };
};
it('returns the stored bytes, whole or as a range', async () => {
const content = randomBytes(128 * 1024 + 3);
const file = await writeFile('dav-get.bin', content);
const whole = await dav(file.path);
expect(whole.status).toBe(200);
expect(whole.headers.get('content-length')).toBe(
String(content.byteLength),
);
expect(Buffer.from(await whole.arrayBuffer())).toEqual(content);
const range = await dav(file.path, {
headers: { Range: 'bytes=10-19' },
});
expect(range.status).toBe(206);
expect(range.headers.get('content-range')).toBe(
`bytes 10-19/${content.byteLength}`,
);
expect(Buffer.from(await range.arrayBuffer())).toEqual(
content.subarray(10, 20),
);
});
/**
* Storage hands over one chunk of `file` and then goes quiet, as a large
* object does while the client is still reading.
*/
const stallStorage = (file: { uuid: string }) => {
const s3 = env.server.stores.s3Object;
const client = env.server.clients.s3.get(s3.resolveRegion(null));
const send = client.send.bind(client);
const bodies: Readable[] = [];
const spy = vi.spyOn(client, 'send').mockImplementation(((
command: unknown,
) => {
if (
!(command instanceof GetObjectCommand) ||
command.input.Key !== file.uuid
) {
return send(command as never);
}
let pushed = false;
const body = new Readable({
read() {
if (!pushed) {
pushed = true;
this.push(Buffer.alloc(16 * 1024, 1));
}
},
});
bodies.push(body);
return Promise.resolve({
Body: body,
ContentLength: 1024 * 1024,
ContentType: 'application/octet-stream',
});
}) as never);
return { bodies, restore: () => spy.mockRestore() };
};
const expectFullDownload = async (path: string) => {
const response = await dav(path);
expect(response.status).toBe(200);
expect((await response.arrayBuffer()).byteLength).toBe(1024 * 1024);
};
it('releases the storage stream when the client aborts', async () => {
const file = await writeFile(
'dav-abort.bin',
Buffer.alloc(1024 * 1024, 1),
);
const storage = stallStorage(file);
try {
const abort = new AbortController();
const response = await dav(file.path, { signal: abort.signal });
expect(response.status).toBe(200);
await response.body!.getReader().read();
abort.abort();
expect(storage.bodies).toHaveLength(1);
await vi.waitFor(() =>
expect(storage.bodies[0]!.destroyed).toBe(true),
);
} finally {
storage.restore();
}
await expectFullDownload(file.path);
});
it('cuts the download off when storage fails mid-stream', async () => {
const file = await writeFile(
'dav-reset.bin',
Buffer.alloc(1024 * 1024, 1),
);
const storage = stallStorage(file);
try {
const response = await dav(file.path);
expect(response.status).toBe(200);
const reader = response.body!.getReader();
await reader.read();
storage.bodies[0]!.destroy(new Error('upstream reset'));
const outcome = await (async () => {
try {
while (!(await reader.read()).done);
return 'ended';
} catch {
return 'cut off';
}
})();
expect(outcome).toBe('cut off');
} finally {
storage.restore();
}
await expectFullDownload(file.path);
});
it('stays within the close listener budget per response', async () => {
const file = await writeFile('dav-listeners.bin', randomBytes(4096));
const tracker = trackResponseCloseListeners();
try {
const response = await dav(file.path);
expect(response.status).toBe(200);
await response.arrayBuffer();
} finally {
tracker.restore();
}
const peak = tracker.peak(
(req) => req.method === 'GET' && req.url === file.path,
);
expect(peak).toBeGreaterThan(0);
expect(peak).toBeLessThanOrEqual(RESPONSE_CLOSE_LISTENER_BUDGET);
});
});
@@ -1365,6 +1365,7 @@ describe('WebDAVController verbs', () => {
callback();
},
});
sink.on('error', () => {});
sink.on('finish', () => {
captured.ended = true;
});
@@ -21,7 +21,6 @@ import type { Team2faLookup } from '../../core/http/middleware/gates.js';
import { compare as bcryptCompare } from 'bcrypt';
import type { Request, Response } from 'express';
import { posix as pathPosix } from 'node:path';
import { pipeline } from 'node:stream/promises';
import type { Actor } from '../../core/actor.js';
import { HttpError } from '../../core/http/HttpError.js';
import { pipeToResponse } from '../../core/http/pipeToResponse.js';
@@ -551,12 +550,7 @@ export class WebDAVController extends PuterController {
'Content-Length': String(result.contentLength ?? 0),
});
}
try {
await pipeline(result.body, res);
} catch {
// Client disconnect or upstream stream error: pipeline already
// tore down both ends, so nothing is left dangling to crash on.
}
await pipeToResponse(result.body, res);
}
// -- PROPFIND ----------------------------------------------------
@@ -813,7 +807,10 @@ export class WebDAVController extends PuterController {
const userId = actor.user!.id as number;
await this.#assertCreate(actor, davPath);
const existing = await this.stores.fsEntry.getEntryByPath(davPath);
// A name only the collation matches is taken too.
const existing = await this.stores.fsEntry.getEntryByPath(davPath, {
foldedNames: true,
});
if (existing)
throw new HttpError(405, 'Already exists', {
legacyCode: 'bad_request',
@@ -937,7 +934,9 @@ export class WebDAVController extends PuterController {
await this.#assertCreate(actor, destPath);
const overwrite = req.headers.overwrite !== 'F';
const destExists = await this.stores.fsEntry.getEntryByPath(destPath);
const destExists = await this.stores.fsEntry.getEntryByPath(destPath, {
foldedNames: true,
});
if (destExists && !overwrite)
throw new HttpError(412, 'Destination exists and Overwrite=F', {
legacyCode: 'conflict',
@@ -985,7 +984,9 @@ export class WebDAVController extends PuterController {
await this.#assertCreate(actor, destPath);
const overwrite = req.headers.overwrite !== 'F';
const destExists = await this.stores.fsEntry.getEntryByPath(destPath);
const destExists = await this.stores.fsEntry.getEntryByPath(destPath, {
foldedNames: true,
});
if (destExists && !overwrite)
throw new HttpError(412, 'Destination exists and Overwrite=F', {
legacyCode: 'conflict',
+7 -1
View File
@@ -76,8 +76,14 @@ export interface Actor {
/**
* Set when authenticated by a session token or an app-under-user token
* carrying one. `kind` mirrors the session row (`web`, `app`, `worker`).
* `parentUid` is the sign-in a `web` session was handed out under, set only
* on a first-party page's session.
*/
session?: { uid: string; kind?: string | null } | null;
session?: {
uid: string;
kind?: string | null;
parentUid?: string | null;
} | null;
/**
* Events handler runs behind this actor's writes; set only from an events
* handler token or a token one minted. Provenance, not authority, so
+2 -1
View File
@@ -19,6 +19,7 @@
import type { Response } from 'express';
import { HttpError } from './HttpError.js';
import { onResponseClose } from './onResponseClose.js';
/**
* A signal that fires when the caller hangs up before its response is done, so
@@ -38,6 +39,6 @@ export const abortOnDisconnect = (res: Response): AbortSignal => {
}
};
if (res.destroyed) onClose();
else res.once('close', onClose);
else onResponseClose(res, onClose);
return abort.signal;
};
@@ -104,6 +104,13 @@ declare global {
*/
deviceFingerprint?: string;
/**
* The client's TLS fingerprint, read from the header named by
* `tlsFingerprintHeader`. Shared by every client on the same TLS
* stack, so it identifies software, not a person.
*/
tlsFingerprint?: string;
/**
* Parsed cookies, populated by the global `cookie-parser`
* middleware.
@@ -28,6 +28,7 @@ import {
} from '../../../services/metering/costs.js';
import type { UsageInput } from '../../../services/metering/types';
import '../expressAugmentation';
import { onResponseClose } from '../onResponseClose';
/**
* Subset of the metering service this middleware needs. Metering is optional
@@ -195,7 +196,7 @@ export const createEgressMeteringMiddleware = (
// 'close' rather than 'finish': it fires for a response that completed
// AND for one whose connection died mid-stream, and an aborted download
// still sent whatever it sent.
res.once('close', () => {
onResponseClose(res, () => {
try {
if (!isMeteredHost(req)) return;
const actor = resolveEgressActor(req);
@@ -23,8 +23,11 @@ import { createFingerprintMiddleware } from './fingerprint';
const VALID_FP = 'dev-fingerprint-abc123';
const run = (reqPartial: Partial<Request> & Record<string, unknown>) => {
const middleware = createFingerprintMiddleware();
const run = (
reqPartial: Partial<Request> & Record<string, unknown>,
opts?: Parameters<typeof createFingerprintMiddleware>[0],
) => {
const middleware = createFingerprintMiddleware(opts);
const req = {
headers: {},
...reqPartial,
@@ -90,4 +93,21 @@ describe('createFingerprintMiddleware', () => {
const { req } = run({ ip: '203.0.113.1' });
expect(req.deviceFingerprint).toBeUndefined();
});
it('reads the TLS fingerprint from the configured header only', () => {
const ja4 = 't13d1516h2_8daaf6152771_02713d6af862';
const headers = { 'x-tls-fp': ja4 };
expect(run({ headers }).req.tlsFingerprint).toBeUndefined();
expect(
run({ headers }, { tlsFingerprintHeader: 'X-TLS-FP' }).req
.tlsFingerprint,
).toBe(ja4);
expect(
run(
{ headers: { 'x-tls-fp': 'not a fingerprint!' } },
{ tlsFingerprintHeader: 'x-tls-fp' },
).req.tlsFingerprint,
).toBeUndefined();
});
});
@@ -33,6 +33,20 @@ const DEVICE_FINGERPRINT_SHAPE = /^[A-Za-z0-9._-]{8,128}$/;
/** Header the GUI may send the device fingerprint on for non-signup requests. */
const DEVICE_FINGERPRINT_HEADER = 'x-puter-device-fingerprint';
/** JA3 / JA4-style values; anything else is dropped. */
const TLS_FINGERPRINT_SHAPE = /^[A-Za-z0-9_]{10,64}$/;
function readTlsFingerprint(
req: { headers?: Record<string, unknown> },
header: string | undefined,
): string | undefined {
if (!header) return undefined;
const value = req.headers?.[header];
return typeof value === 'string' && TLS_FINGERPRINT_SHAPE.test(value)
? value
: undefined;
}
function readDeviceFingerprint(req: {
body?: unknown;
headers?: Record<string, unknown>;
@@ -65,15 +79,22 @@ function readDeviceFingerprint(req: {
* and spoofable, but stable per real device across IP rotation. The rate
* limiter's 'fingerprint' strategy appends it to the network hash so each
* device behind a shared network gets its own bucket.
* - `req.tlsFingerprint` — set only when `tlsFingerprintHeader` is configured and
* the request carries a well-shaped value in that header. It names the
* client's TLS stack, so every client on the same build shares it.
*
* Install AFTER the body parsers (so the body fingerprint is readable) and
* before `requestContext` (so the snapshot into ALS already carries them).
* Never rejects — a missing/invalid device fingerprint is simply absent.
*/
export const createFingerprintMiddleware = (): RequestHandler => {
export const createFingerprintMiddleware = (
opts: { tlsFingerprintHeader?: string } = {},
): RequestHandler => {
const tlsHeader = opts.tlsFingerprintHeader?.toLowerCase();
return (req, _res, next) => {
req.networkFingerprint = computeNetworkFingerprint(req);
req.deviceFingerprint = readDeviceFingerprint(req);
req.tlsFingerprint = readTlsFingerprint(req, tlsHeader);
next();
};
};
@@ -20,6 +20,7 @@
import type { Request, RequestHandler, Response } from 'express';
import { actorUid } from '../../actor';
import { HttpError } from '../HttpError';
import { onResponseClose } from '../onResponseClose';
import { createJsonBodyParser } from './jsonBody';
/** The global JSON parser's limit, which applies before auth. */
@@ -152,7 +153,7 @@ export const createLargeJsonBodyParser = ({
}
inFlightBytes += declared;
inFlightByUser.set(user, userBytes + declared);
res.once('close', () => {
onResponseClose(res, () => {
inFlightBytes -= declared;
const left = (inFlightByUser.get(user) ?? 0) - declared;
if (left > 0) inFlightByUser.set(user, left);
@@ -26,7 +26,7 @@ import type { IConfig } from '../../../types';
* Browsers send none of those variations, but config values do — a hand-edited
* `origin` with a trailing slash would otherwise reject every real request.
*/
const normalizeOrigin = (raw: string | undefined): string =>
export const normalizeOrigin = (raw: string | undefined): string =>
(raw ?? '').trim().replace(/\/+$/, '').toLowerCase();
/**
@@ -21,6 +21,7 @@
import crypto from 'node:crypto';
import { withSpan } from '../../../util/span.js';
import { HttpError } from '../HttpError.js';
import { onResponseClose } from '../onResponseClose.js';
import { subscriptionOverride } from '../../../services/metering/consts.js';
/**
@@ -1169,7 +1170,7 @@ export function concurrencyGate(opts) {
const { release } = guardSlot(slot, '[concurrent]');
const onDone = () => void release();
res.once('finish', onDone);
res.once('close', onDone);
onResponseClose(res, onDone);
if (res.destroyed || res.writableFinished) onDone();
next();
};
@@ -0,0 +1,105 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { EventEmitter } from 'node:events';
import { createServer, get, type Server } from 'node:http';
import type { AddressInfo } from 'node:net';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { onResponseClose } from './onResponseClose.js';
let server: Server | undefined;
afterEach(async () => {
server?.closeAllConnections();
await new Promise((resolve) => server?.close(resolve) ?? resolve(null));
server = undefined;
});
describe('onResponseClose', () => {
it('shares one close listener and runs every callback once, in order', () => {
const res = new EventEmitter();
const calls: string[] = [];
onResponseClose(res, () => calls.push('a'));
onResponseClose(res, () => calls.push('b'));
onResponseClose(res, () => calls.push('c'));
expect(res.listenerCount('close')).toBe(1);
res.emit('close');
res.emit('close');
expect(calls).toEqual(['a', 'b', 'c']);
expect(res.listenerCount('close')).toBe(0);
});
it('keeps responses apart', () => {
const first = new EventEmitter();
const second = new EventEmitter();
const onFirst = vi.fn();
const onSecond = vi.fn();
onResponseClose(first, onFirst);
onResponseClose(second, onSecond);
first.emit('close');
expect(onFirst).toHaveBeenCalledOnce();
expect(onSecond).not.toHaveBeenCalled();
});
it('runs the rest when one callback throws, and still surfaces the error', async () => {
const res = new EventEmitter();
const after = vi.fn();
const failure = new Error('callback broke');
const uncaught = vi.fn();
const listeners = process.listeners('uncaughtException');
process.removeAllListeners('uncaughtException');
process.on('uncaughtException', uncaught);
try {
onResponseClose(res, () => {
throw failure;
});
onResponseClose(res, after);
res.emit('close');
expect(after).toHaveBeenCalledOnce();
await vi.waitFor(() =>
expect(uncaught).toHaveBeenCalledWith(
failure,
expect.anything(),
),
);
} finally {
process.off('uncaughtException', uncaught);
for (const listener of listeners)
process.on('uncaughtException', listener);
}
});
it('fires when a client hangs up on a real response', async () => {
const closed = new Promise<boolean>((resolve) => {
server = createServer((_req, res) => {
onResponseClose(res, () => resolve(res.writableFinished));
res.write('partial');
});
});
await new Promise<void>((resolve) => server!.listen(0, resolve));
const { port } = server!.address() as AddressInfo;
const req = get(`http://127.0.0.1:${port}/`, (res) => {
res.once('data', () => req.destroy());
});
req.on('error', () => {});
expect(await closed).toBe(false);
});
});
+57
View File
@@ -0,0 +1,57 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/** Only what's used here, so test doubles and express's Response both fit. */
interface ClosingResponse {
once(event: 'close', listener: () => void): unknown;
}
const callbacksByResponse = new WeakMap<ClosingResponse, Array<() => void>>();
/**
* Runs `callback` once when the response closes. Every callback on a response
* shares one `close` listener, since Node warns past ten and request tracing
* adds its own. Callbacks run in the order added; one that throws doesn't stop
* the rest.
*/
export const onResponseClose = (
res: ClosingResponse,
callback: () => void,
): void => {
const callbacks = callbacksByResponse.get(res);
if (callbacks) {
callbacks.push(callback);
return;
}
const pending = [callback];
callbacksByResponse.set(res, pending);
res.once('close', () => {
for (const run of pending.splice(0)) {
try {
run();
} catch (e) {
// Surfaces as an uncaught error, as it would from a listener
// of its own.
queueMicrotask(() => {
throw e;
});
}
}
});
};
@@ -0,0 +1,147 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import {
RESPONSE_CLOSE_LISTENER_BUDGET,
setupPuterTestEnv,
trackResponseCloseListeners,
type PuterTestEnv,
} from '../../testUtil.js';
/**
* The routes that hold a response open the longest, where middleware, gates and
* the handler each want to hear when it closes.
*/
describe('close listeners per response over HTTP', () => {
let env: PuterTestEnv;
let filePath = '';
beforeAll(async () => {
env = await setupPuterTestEnv({
meteringEnforcement: { subscriptions: false },
} as never);
const { username } = env.users.user;
const user = await env.server.stores.user.getByUsername(username);
filePath = `/${username}/Documents/close-listeners.txt`;
await env.server.services.fs.write(user!.id, {
fileMetadata: {
path: filePath,
size: 4096,
contentType: 'text/plain',
},
fileContent: Buffer.alloc(4096, 97),
});
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const streamedChat = {
model: 'fake',
stream: true,
messages: [{ role: 'user', content: 'hi' }],
max_tokens: 16,
};
const post = (path: string, token: string, body: unknown) =>
fetch(new URL(path, env.apiOrigin), {
method: 'POST',
headers: {
Authorization: `Bearer ${token}`,
'Content-Type': 'application/json',
Origin: env.apiOrigin,
},
body: JSON.stringify(body),
});
const routes: Array<[string, string, () => Promise<Response>]> = [
[
'a streamed driver call',
'/drivers/call',
() =>
post('/drivers/call', env.users.user.token, {
interface: 'puter-chat-completion',
service: 'ai-chat',
method: 'complete',
args: streamedChat,
}),
],
[
'an Anthropic-style streamed completion',
'/puterai/anthropic/v1/messages',
() =>
post(
'/puterai/anthropic/v1/messages',
env.users.user.apiToken,
streamedChat,
),
],
[
'an OpenAI-style streamed completion',
'/puterai/openai/v1/chat/completions',
() =>
post(
'/puterai/openai/v1/chat/completions',
env.users.user.apiToken,
streamedChat,
),
],
[
'a file read',
'/read',
() => {
const url = new URL('/read', env.apiOrigin);
url.searchParams.set('file', filePath);
return fetch(url, {
headers: {
Authorization: `Bearer ${env.users.user.token}`,
},
});
},
],
[
'a v2 file read',
'/fs/read',
() => {
const url = new URL('/fs/read', env.apiOrigin);
url.searchParams.set('path', filePath);
return fetch(url, {
headers: {
Authorization: `Bearer ${env.users.user.token}`,
},
});
},
],
];
it.each(routes)('%s stays within the budget', async (_name, path, call) => {
const tracker = trackResponseCloseListeners();
try {
const response = await call();
expect(response.status).toBe(200);
await response.arrayBuffer();
} finally {
tracker.restore();
}
const peak = tracker.peak((req) => req.url?.split('?')[0] === path);
expect(peak).toBeGreaterThan(0);
expect(peak).toBeLessThanOrEqual(RESPONSE_CLOSE_LISTENER_BUDGET);
});
});
+2 -1
View File
@@ -21,6 +21,7 @@ import type { Request, RequestHandler } from 'express';
import { actorUid } from '../actor';
import type { EventClient } from '../../clients/event/EventClient';
import type { LifecycleRejectStatus } from '../../clients/event/types';
import { onResponseClose } from './onResponseClose';
import type { RouteMethod } from './types';
type RoutePathValue = string | RegExp | Array<string | RegExp> | undefined;
@@ -179,7 +180,7 @@ export const createRouteLifecycleMiddleware = (
);
};
res.once('finish', () => settle(false));
res.once('close', () => settle(!res.writableFinished));
onResponseClose(res, () => settle(!res.writableFinished));
next();
};
};
@@ -2128,6 +2128,29 @@ describe('KVStoreDriver', () => {
await expect(inCtx(() => target.flush({}))).resolves.toBe(true);
});
it('removes paths without handing the value back', async () => {
await inCtx(() =>
target.set({ key: 'obj', value: { a: 1, b: 2 } }),
);
expect(
await inCtx(() => target.remove({ key: 'obj', paths: ['a'] })),
).toEqual({ b: 2 });
await exhaust(actor);
// A path the value doesn't have would otherwise read it all back.
expect(
await inCtx(() => target.remove({ key: 'obj', paths: ['x'] })),
).toBeNull();
expect(
await inCtx(() => target.remove({ key: 'obj', paths: ['b'] })),
).toBeNull();
const stored = await server.stores.kv.get(
{ key: 'obj' },
{ actor },
);
expect(stored.res).toEqual({});
});
it('exempts a worker session', async () => {
const worker = makeActor({
session: { uid: 'worker-session', kind: 'worker' },
+15 -2
View File
@@ -42,7 +42,10 @@ import {
type KVUsage,
} from '../../stores/systemKv/SystemKVStore.js';
import { KV_SHARE_HANDLE_WIDTHS } from '../../stores/events/columnWidths.js';
import { assertActorHasCredits } from '../../services/metering/enforcement.js';
import {
actorHasCredits,
assertActorHasCredits,
} from '../../services/metering/enforcement.js';
import { isKvHandleId } from '../../services/events/subjects.js';
import {
KV_HANDLE_KEY_MAX_BYTES,
@@ -263,7 +266,7 @@ const relativizeError = (
* `list` is here for the step before that: deleting a key means knowing it
* exists, and `flush` — the only alternative — takes everything. It is gated
* again inside the method for the forms that return values, which are a read
* like any other.
* like any other. `remove` withholds the value it would return instead.
*/
const CREDIT_UNGATED_KV_METHODS = new Set(['del', 'remove', 'flush', 'list']);
@@ -1231,6 +1234,16 @@ export class KVStoreDriver extends PuterDriver {
) {
return null;
}
// Nor does an account out of credits, whose other reads are refused.
if (
!(await actorHasCredits(
this.services.metering,
opts.actor,
this.config,
))
) {
return null;
}
return res;
}
}
@@ -504,3 +504,95 @@ describe.runIf(process.env.PUTER_TEST_DB_ENGINE === 'postgres')(
});
},
);
describe('an immediate send answers with the row it wrote', () => {
/** What the driver returns for a row. */
const clientShape = (row: Record<string, unknown> | null) => ({
uid: row?.uid,
value: row?.value,
app_uid: row?.app_uid ?? null,
shown: row?.shown ?? null,
acknowledged: row?.acknowledged ?? null,
created_at: row?.created_at ?? null,
});
/** `fn`'s result, with the notification writes and uid lookups it made. */
const traced = async <T>(fn: () => Promise<T>) => {
const reads = [vi.spyOn(db(), 'read'), vi.spyOn(db(), 'pread')];
const write = vi.spyOn(db(), 'write');
try {
const result = await fn();
const sql = (calls: unknown[][]) =>
calls
.map(([q]) => String(q))
.filter((q) => /`notification`/u.test(q));
return {
result: result as Record<string, unknown>,
writes: sql(write.mock.calls),
uidReads: reads
.flatMap((spy) => sql(spy.mock.calls))
.filter((q) =>
/FROM `notification` WHERE `uid` = \?/u.test(q),
),
};
} finally {
for (const spy of [...reads, write]) spy.mockRestore();
}
};
const stored = (uid: unknown) =>
server.stores.notification.getByUid(String(uid), {
userId: holder.user!.id,
primary: true,
});
it('a new row: one insert and no read of it', async () => {
setSchedules(undefined);
const { result, writes, uidReads } = await traced(() =>
notify({ text: 'new' }),
);
expect(writes).toHaveLength(1);
expect(writes[0]).toMatch(/^INSERT INTO `notification`/u);
expect(uidReads).toEqual([]);
expect(result).toEqual(clientShape(await stored(result.uid)));
});
it('a group replacement: one update and no read of the row after it', async () => {
setSchedules(undefined);
const first = (await notify({ group: 'g', text: 'one' })) as {
uid: string;
};
const { result, writes, uidReads } = await traced(() =>
notify({ group: 'g', text: 'two' }),
);
expect(result.uid).toBe(first.uid);
expect(writes).toHaveLength(1);
expect(writes[0]).toMatch(/^UPDATE `notification`/u);
expect(uidReads).toEqual([]);
expect(result).toEqual(clientShape(await stored(first.uid)));
expect(result.value).toMatchObject({ text: 'two' });
});
it('a replacement of the row a delayed send wrote', async () => {
setSchedules(undefined);
const token = uuidv4();
const key = uuidv4();
const id = await claimed(token);
await write(key, txFor(id, token), {
payload: JSON.stringify({
message: { title: 'Later', group: 'g' },
}),
});
const { result, uidReads } = await traced(() =>
notify({ group: 'g', text: 'now' }),
);
expect(result.uid).toBe(key);
expect(uidReads).toEqual([]);
const row = await stored(key);
expect(row?.send_key).toBeNull();
expect(result).toEqual(clientShape(row));
});
});
@@ -161,6 +161,44 @@ describe('NotificationDriver.create', () => {
expect(result?.value).toEqual({ type: NOTIF_TYPE, title: 'lagged' });
});
it('writes once and returns the row without reading it back', async () => {
const { actor, userId } = await makeUser();
const db = server.clients.db;
const read = vi.spyOn(db, 'read');
const pread = vi.spyOn(db, 'pread');
const write = vi.spyOn(db, 'write');
let result: Record<string, unknown> | null;
let writes: unknown[][];
let reads: unknown[][];
try {
result = (await withActor(actor, () =>
driver.create({
object: { value: { type: NOTIF_TYPE, title: 'once' } },
}),
)) as Record<string, unknown> | null;
} finally {
writes = write.mock.calls;
reads = [...read.mock.calls, ...pread.mock.calls];
for (const spy of [read, pread, write]) spy.mockRestore();
}
const touches = (q: unknown) => /`notification`/u.test(String(q));
expect(writes.filter(([q]) => touches(q))).toHaveLength(1);
expect(reads.filter(([q]) => touches(q))).toEqual([]);
const row = await server.stores.notification.getByUid(
result!.uid as string,
{ userId, primary: true },
);
expect(result).toEqual({
uid: row?.uid,
value: row?.value,
app_uid: row?.app_uid ?? null,
shown: row?.shown ?? null,
acknowledged: row?.acknowledged ?? null,
created_at: row?.created_at ?? null,
});
});
it('rejects a `value` with no `type` with 400', async () => {
const { actor } = await makeUser();
await expect(
@@ -64,6 +64,7 @@ import { PuterDriver } from '../types.js';
import type { Actor } from '../../core/actor.js';
import type { DriverRateLimitConfig } from '../meta.js';
import { CRUD_CONCURRENT } from '../util/crudLimits.js';
import type { NotificationRow } from '../../services/notification/NotificationService.js';
const MAX_SELECT_LIMIT = 200;
const MINUTE_MS = 60_000;
@@ -289,12 +290,12 @@ export class NotificationDriver extends PuterDriver {
// Silent: a driver `create` has never pushed to the live socket, only
// ever written the row — moving through the registry does not change
// that.
const uid = await this.#write(actor.user.id, value, {
const row = await this.#write(actor.user.id, value, {
type: registeredType,
appUid,
silent: true,
});
return this.#readBack(uid, actor.user.id);
return this.#toClient(row);
}
async read(args: Record<string, unknown>): Promise<unknown> {
@@ -447,12 +448,9 @@ export class NotificationDriver extends PuterDriver {
return this.#scheduleAppMessage(actor, message, handle, timing);
if (handle === undefined) {
const uid = await this.#sendAppMessage(
actor.user.id,
appUid,
message,
return this.#toClient(
await this.#sendAppMessage(actor.user.id, appUid, message),
);
return this.#readBack(uid, actor.user.id);
}
try {
const recipient =
@@ -484,7 +482,7 @@ export class NotificationDriver extends PuterDriver {
recipientId: number,
appUid: string,
message: AppMessage,
): Promise<string> {
): Promise<NotificationRow> {
return this.#oneSendPerGroup(recipientId, appUid, message, () =>
this.#replaceOrWriteAppMessage(recipientId, appUid, message),
);
@@ -675,7 +673,7 @@ export class NotificationDriver extends PuterDriver {
recipientId: number,
appUid: string,
message: AppMessage,
): Promise<string> {
): Promise<NotificationRow> {
// Primary: a lagging replica would turn a replacement into a new
// notification, which spends a day slot.
const open = message.group
@@ -698,17 +696,13 @@ export class NotificationDriver extends PuterDriver {
);
if (open) {
const uid = String(open.uid);
if (
await this.services.notification.notifyUpdate(
uid,
recipientId,
message,
{ type: APP_MESSAGE_TYPE, replace: true },
)
) {
return uid;
}
const replaced = await this.services.notification.replaceUnseen(
open,
recipientId,
message,
{ type: APP_MESSAGE_TYPE },
);
if (replaced) return replaced;
// Shown or dismissed since the lookup, so this one is new after all.
await this.#spendAppMessageLimits(recipientId, appUid, ['day']);
}
@@ -760,7 +754,7 @@ export class NotificationDriver extends PuterDriver {
}
}
/** Writes for `userId`, who need not be the actor, and names the row. */
/** Writes for `userId`, who need not be the actor, and returns the row. */
async #write(
userId: number,
value: Record<string, unknown>,
@@ -770,31 +764,18 @@ export class NotificationDriver extends PuterDriver {
silent: boolean;
group?: string | null;
},
): Promise<string> {
const uid = await this.services.notification.notify(
): Promise<NotificationRow> {
const [row] = await this.services.notification.notifyRows(
[userId],
value,
opts,
);
if (!uid) {
if (!row) {
throw new HttpError(500, 'Notification could not be created', {
legacyCode: 'internal_error',
});
}
return uid;
}
/** The service hands back only the uid; the primary has the rest. */
async #readBack(
uid: string,
userId: number,
): Promise<Record<string, unknown> | null> {
return this.#toClient(
await this.stores.notification.getByUid(uid, {
userId,
primary: true,
}),
);
return row;
}
// -- Permissions -------------------------------------------------
@@ -129,9 +129,9 @@ describe('pipeStreamResult', () => {
expect(warn).not.toHaveBeenCalled();
});
it('adds at most two close listeners to the response', async () => {
// The response already carries close listeners from metering,
// compression and tracing; Node warns past ten.
it('adds no close listener beyond the one pipe adds', async () => {
// The response already carries close listeners from middleware and
// request tracing; Node warns past ten.
let added = -1;
server = createServer((_req, res) => {
const before = res.listenerCount('close');
@@ -146,8 +146,7 @@ describe('pipeStreamResult', () => {
res.on('end', () => resolve());
}),
);
expect(added).toBeGreaterThan(0);
expect(added).toBeLessThanOrEqual(2);
expect(added).toBe(1);
});
it('drops the source, without throwing, when the client left before the result was ready', async () => {
+7 -8
View File
@@ -33,9 +33,9 @@ type ResponseSink = NodeJS.WritableStream & {
* than a clean end, and the failure is logged once instead of surfacing as an
* unhandled stream error. A client that hangs up stops the source.
*
* `pipe` rather than `pipeline`: pipeline puts seven close listeners on the
* response, and with the ones metering, compression and tracing add, every
* streamed response tripped Node's ten-listener warning.
* `pipe` rather than `pipeline`, and `unpipe` rather than `close`: Node warns
* past ten close listeners on the response, and other middleware and request
* tracing add their own.
*/
export function pipeStreamResult(
stream: Readable,
@@ -52,11 +52,10 @@ export function pipeStreamResult(
if (!res.destroyed) res.destroy();
console.warn(`[drivers] ${label} stream failed mid-response:`, err);
});
// A response that closes before the source ended is a client hanging up.
res.once('close', () => {
if (stream.readableEnded) return;
stream.unpipe(res);
stream.destroy();
// `pipe` unpipes once the response finishes, closes or fails; a source
// that hasn't ended by then has a client that hung up.
res.once('unpipe', () => {
if (!stream.readableEnded) stream.destroy();
});
res.once('error', () => stream.destroy());
stream.pipe(res);
+3 -3
View File
@@ -53,7 +53,7 @@
"mime-types": "^2.1.35",
"murmurhash": "^2.0.1",
"mysql2": "^3.24.6",
"nodemailer": "^10.1.4",
"nodemailer": "^10.1.7",
"openai": "^6.34.0",
"otpauth": "^9.5.2",
"pg": "^8.21.0",
@@ -65,10 +65,10 @@
"svg-captcha": "^1.4.0",
"together-ai": "^0.33.0",
"ua-parser-js": "^1.0.41",
"undici": "^7.30.1",
"undici": "^7.30.2",
"uuid": "^14.0.3",
"validator": "^13.15.35",
"ws": "^8.22.0"
"ws": "^8.22.2"
},
"devDependencies": {
"@types/bcrypt": "^6.0.0",
+53
View File
@@ -645,6 +645,59 @@ describe('PuterServer host header validation — permissive modes', () => {
});
});
describe('PuterServer behind a trusted proxy', () => {
let server: PuterServer;
let port: number;
beforeAll(async () => {
port = await allocateEphemeralPort();
server = await setupTestServer(
{
port,
domain: 'puter.localhost',
origin: `http://puter.localhost:${port}`,
api_base_url: `http://api.puter.localhost:${port}`,
allow_all_host_values: false,
allow_no_host_header: false,
custom_domains_enabled: false,
static_hosting_domain: 'site.puter.localhost',
trust_proxy: 1,
} as unknown as IConfig,
{ listen: true },
);
});
afterAll(async () => {
await server?.shutdown();
});
it('serves the main domain whatever X-Forwarded-Host names', async () => {
const res = await rawRequest(port, '/healthcheck', {
host: `puter.localhost:${port}`,
'x-forwarded-host': 'nosuchsite.site.puter.localhost',
});
expect(res.status).toBe(200);
expect(res.headers['x-frame-options']).toBe('SAMEORIGIN');
});
it('keeps a hosted-site host on site hosting', async () => {
const res = await rawRequest(port, '/healthcheck', {
host: `nosuchsite.site.puter.localhost:${port}`,
'x-forwarded-host': 'api.puter.localhost',
});
expect(res.status).toBe(404);
});
it('reads subdomains from Host', async () => {
const res = await rawRequest(port, '/healthcheck', {
host: `api.puter.localhost:${port}`,
'x-forwarded-host': 'puter.localhost',
});
expect(res.status).toBe(200);
expect(res.headers['x-frame-options']).toBeUndefined();
});
});
describe('PuterServer host header validation — custom domains with every host accepted', () => {
let server: PuterServer;
let port: number;
+16 -1
View File
@@ -416,6 +416,17 @@ export class PuterServer {
* `#materializeRoute` as those options ship.
*/
#installGlobalMiddleware() {
// -- Forwarded host ------------------------------------------
// With `trust proxy` set, express derives `req.hostname` and
// `req.subdomains` from X-Forwarded-Host, which any client can send.
// Dropping it keeps them on Host, the header host validation checks,
// so a request can't pick a different site or subdomain router than
// the host it was sent to.
this.#app.use((req, _res, next) => {
delete req.headers['x-forwarded-host'];
next();
});
// -- Egress metering -----------------------------------------
// First, so the byte counter wraps `res.write` before compression
// does and therefore counts what actually goes out rather than what
@@ -625,7 +636,11 @@ export class PuterServer {
// `req.deviceFingerprint` (client-supplied, from body/header). Runs
// AFTER body parsing so the body fingerprint is readable, and before
// the ALS context so the snapshot carries them.
this.#app.use(createFingerprintMiddleware());
this.#app.use(
createFingerprintMiddleware({
tlsFingerprintHeader: this.#config.tlsFingerprintHeader,
}),
);
// -- Per-request ALS context ---------------------------------
// Runs AFTER auth probe so `req.actor` is already populated when
@@ -64,7 +64,10 @@ describe('ACL checks read the ancestor chain lazily', () => {
) => {
if (isSystemActor(actor)) return true;
if (path === '/') return ['read', 'list', 'see'].includes(mode);
const chain = await server.stores.fsEntry.getAncestorChain(path);
const chain =
typeof path === 'string'
? await server.stores.fsEntry.getAncestorChain(path)
: [];
return acl.check(
actor,
{ path, resolveAncestors: async () => chain },
@@ -190,9 +193,8 @@ describe('ACL checks read the ancestor chain lazily', () => {
}
}
}
// Grants, refusals, the 400 for a refused path and the failure for a
// missing one all came up.
expect([...seen].sort()).toEqual(['400', 'false', 'true', 'undefined']);
// Grants, refusals and the 400 for a refused path all came up.
expect([...seen].sort()).toEqual(['400', 'false', 'true']);
});
it("answers an owner's own path without reading the chain", async () => {
+304 -4
View File
@@ -83,6 +83,8 @@ function makeService() {
permission: {
// Default: token carries no explicit fs grant rows.
hasAnyAccessTokenPerm: vi.fn().mockResolvedValue(false),
// Default: the user granted the app nothing.
readUserAppPerms: vi.fn().mockResolvedValue([]),
},
user: {
getByUsername: vi.fn().mockResolvedValue(null),
@@ -1306,16 +1308,16 @@ describe('ACLService.statUserUser / setUserUser (integration)', () => {
[tokenUid, top],
);
const get = vi.spyOn(server.clients.redis, 'get');
const list = vi.spyOn(server.stores.permission, 'listAccessTokenPerms');
try {
expect(await acl.check(token, deep, 'read')).toBe(true);
expect(await acl.check(token, deep, 'write')).toBe(false);
const tokenReads = get.mock.calls.filter(
([key]) => key === `perms:token:${tokenUid}`,
const tokenReads = list.mock.calls.filter(
([uid]) => uid === tokenUid,
);
expect(tokenReads).toHaveLength(2);
} finally {
get.mockRestore();
list.mockRestore();
}
});
@@ -1450,6 +1452,252 @@ describe('ACLService.statUserUser / setUserUser (integration)', () => {
).toBe(false);
});
describe('a file opened with the app inside a shared folder', () => {
/** `Documents/<name>` of an issuer, shared with a holder at `mode`. */
const sharedFile = async (mode: 'read' | 'write') => {
const issuer = await makeUser();
const holder = await makeUser();
const folder = await ownedResource(issuer);
const path = `${folder.path}/note-${uuidv4()}.txt`;
const entry = await server.services.fs.touch(issuer.user.id!, {
path,
});
await runWithContext({ actor: issuer }, () =>
acl.setUserUser(issuer, holder, folder, mode),
);
return { issuer, holder, folder, path, uid: entry.uuid };
};
const open = (user: Actor, app: { uid: string }, uid: string) =>
runWithContext({ actor: user }, () =>
server.services.permission.grantUserAppPermission(
user,
app.uid,
`fs:${uid}:write`,
),
);
it('reads and writes it', async () => {
const { holder, path, uid } = await sharedFile('write');
const app = await makeApp(holder.user.id!);
await open(holder, app, uid);
const file = () => acl.fsDescriptor(path);
expect(
await acl.check(asApp(holder, app), file(), 'read'),
).toBe(true);
expect(
await acl.check(asApp(holder, app), file(), 'write'),
).toBe(true);
});
it('can only read it once the share drops to read', async () => {
const { issuer, holder, folder, path, uid } =
await sharedFile('write');
const app = await makeApp(holder.user.id!);
await open(holder, app, uid);
await runWithContext({ actor: issuer }, () =>
acl.setUserUser(issuer, holder, folder, 'read'),
);
const file = () => acl.fsDescriptor(path);
expect(await acl.check(holder, file(), 'write')).toBe(false);
expect(
await acl.check(asApp(holder, app), file(), 'read'),
).toBe(true);
expect(
await acl.check(asApp(holder, app), file(), 'write'),
).toBe(false);
});
it('gives nothing to another app of the same user', async () => {
const { holder, path, uid } = await sharedFile('write');
const opened = await makeApp(holder.user.id!);
const other = await makeApp(holder.user.id!);
await open(holder, opened, uid);
expect(
await acl.check(
asApp(holder, other),
acl.fsDescriptor(path),
'read',
),
).toBe(false);
});
it('reaches it through a token the app issued', async () => {
const { holder, path, uid } = await sharedFile('write');
const app = await makeApp(holder.user.id!);
await open(holder, app, uid);
const tokenUid = `tok-${uuidv4()}`;
await server.clients.db.write(
'INSERT INTO `access_token_permissions` (`token_uid`, `permission`) VALUES (?, ?)',
[tokenUid, `fs:${uid}:read`],
);
const token = makeActor({
user: holder.user,
accessToken: {
uid: tokenUid,
issuer: asApp(holder, app),
authorized: null,
fullAccess: false,
},
});
const file = () => acl.fsDescriptor(path);
expect(await acl.check(token, file(), 'read')).toBe(true);
expect(await acl.check(token, file(), 'write')).toBe(false);
});
it('reaches a file below a folder it was granted, and no sibling of a granted file', async () => {
const { issuer, holder, folder, path, uid } =
await sharedFile('write');
const sub = `${folder.path}/sub-${uuidv4()}`;
const dir = await server.services.fs.mkdir(issuer.user.id!, {
path: sub,
} as never);
const below = `${sub}/below.txt`;
await server.services.fs.touch(issuer.user.id!, {
path: below,
});
const folderApp = await makeApp(holder.user.id!);
const fileApp = await makeApp(holder.user.id!);
await open(holder, folderApp, (dir as { uuid: string }).uuid);
await open(holder, fileApp, uid);
const at = (p: string) => acl.fsDescriptor(p);
expect(
await acl.check(
asApp(holder, folderApp),
at(below),
'write',
),
).toBe(true);
expect(
await acl.check(asApp(holder, fileApp), at(path), 'write'),
).toBe(true);
expect(
await acl.check(asApp(holder, fileApp), at(below), 'read'),
).toBe(false);
});
it('is refused once the share is withdrawn, though its grant stays', async () => {
const { issuer, holder, folder, path, uid } =
await sharedFile('write');
const app = await makeApp(holder.user.id!);
await open(holder, app, uid);
await runWithContext({ actor: issuer }, () =>
server.services.permission.revokeUserUserPermission(
issuer,
holder.user.username!,
`fs:${folder.uid}:write`,
),
);
const file = () => acl.fsDescriptor(path);
expect(await acl.check(holder, file(), 'read')).toBe(false);
expect(
await acl.check(asApp(holder, app), file(), 'read'),
).toBe(false);
});
it('cannot write with a read grant while its user can write', async () => {
const { holder, path, uid } = await sharedFile('write');
const app = await makeApp(holder.user.id!);
await runWithContext({ actor: holder }, () =>
server.services.permission.grantUserAppPermission(
holder,
app.uid,
`fs:${uid}:read`,
),
);
const file = () => acl.fsDescriptor(path);
expect(await acl.check(holder, file(), 'write')).toBe(true);
expect(
await acl.check(asApp(holder, app), file(), 'read'),
).toBe(true);
expect(
await acl.check(asApp(holder, app), file(), 'write'),
).toBe(false);
});
const shareFile = (issuer: Actor, holder: Actor, path: string) =>
runWithContext({ actor: issuer }, () =>
acl.setUserUser(
issuer,
holder,
acl.fsDescriptor(path),
'write',
),
);
it('counts a folder grant only while the user holds that folder', async () => {
const { issuer, holder, folder, path } =
await sharedFile('write');
const app = await makeApp(holder.user.id!);
await open(holder, app, folder.uid);
await runWithContext({ actor: issuer }, () =>
server.services.permission.revokeUserUserPermission(
issuer,
holder.user.username!,
`fs:${folder.uid}:write`,
),
);
await shareFile(issuer, holder, path);
const file = () => acl.fsDescriptor(path);
expect(await acl.check(holder, folder, 'see')).toBe(false);
expect(await acl.check(holder, file(), 'write')).toBe(true);
expect(
await acl.check(asApp(holder, app), file(), 'read'),
).toBe(false);
await runWithContext({ actor: issuer }, () =>
acl.setUserUser(issuer, holder, folder, 'write'),
);
expect(
await acl.check(asApp(holder, app), file(), 'write'),
).toBe(true);
});
it('does not count a grant on a folder the user never held', async () => {
const issuer = await makeUser();
const holder = await makeUser();
const folder = await ownedResource(issuer);
const path = `${folder.path}/note-${uuidv4()}.txt`;
await server.services.fs.touch(issuer.user.id!, { path });
const app = await makeApp(holder.user.id!);
await open(holder, app, folder.uid);
await shareFile(issuer, holder, path);
const file = () => acl.fsDescriptor(path);
expect(await acl.check(holder, file(), 'write')).toBe(true);
expect(
await acl.check(asApp(holder, app), file(), 'read'),
).toBe(false);
});
it('leaves a file the user owns as it was', async () => {
const user = await makeUser();
const path = `/${user.user.username}/Documents/own-${uuidv4()}.txt`;
const entry = await server.services.fs.touch(user.user.id!, {
path,
});
const opened = await makeApp(user.user.id!);
const other = await makeApp(user.user.id!);
await open(user, opened, entry.uuid);
const file = () => acl.fsDescriptor(path);
expect(
await acl.check(asApp(user, opened), file(), 'write'),
).toBe(true);
expect(
await acl.check(asApp(user, other), file(), 'read'),
).toBe(false);
});
});
describe("another app's AppData", () => {
/** `/<user>/AppData/<appUid>/inner`, as the app would leave it. */
const seedAppData = async (user: Actor, appUid: string) => {
@@ -1499,6 +1747,36 @@ describe('ACLService.statUserUser / setUserUser (integration)', () => {
).toBe(true);
});
it('stays closed to an app granted the home or AppData of a user who shared them', async () => {
const issuer = await makeUser();
const holder = await makeUser();
const closed = await makeApp(issuer.user.id!, {
metadata: JSON.stringify({ share_app_data: false }),
});
const root = await seedAppData(issuer, closed.uid);
const inner = acl.fsDescriptor(`${root}/inner`);
const home = `/${issuer.user.username}`;
await runWithContext({ actor: issuer }, () =>
acl.setUserUser(
issuer,
holder,
acl.fsDescriptor(home),
'write',
),
);
expect(await acl.check(holder, inner, 'read')).toBe(true);
for (const target of [home, `${home}/AppData`]) {
const entry =
await server.stores.fsEntry.getEntryByPath(target);
const reader = await makeApp(holder.user.id!);
await grantTo(holder, reader, `fs:${entry!.uuid}:read`);
expect(
await acl.check(asApp(holder, reader), inner, 'read'),
).toBe(false);
}
});
it('opens to an fs grant while that app shares', async () => {
const user = await makeUser();
const open = await makeApp(user.user.id!);
@@ -1802,6 +2080,28 @@ describe('ACLService filesystem descriptors and refusals', () => {
});
});
it('answers an entry with no path as absent', async () => {
const { service } = withChains();
const pathless = null as unknown as string;
await expect(
service.check(
issuerActor,
{ path: pathless, resolveAncestors: async () => [] },
'see',
),
).resolves.toBe(false);
await expect(
refusedWith(() =>
service.assertFsAccess(issuerActor, pathless, 'list'),
),
).resolves.toEqual({
status: 404,
message: 'Subject does not exist',
legacyCode: 'subject_does_not_exist',
});
});
it('lets a create through on write to the parent or to the target', async () => {
const { service, services } = withChains();
grant(services, [['/victim/shared', 'write']]);
+47 -5
View File
@@ -57,11 +57,7 @@ export interface ResourceDescriptor {
}
export type AclMode =
| 'see'
| 'list'
| 'read'
| 'write'
| typeof MANAGE_PERM_PREFIX;
'see' | 'list' | 'read' | 'write' | typeof MANAGE_PERM_PREFIX;
/** Duck-typed error shape compatible with APIError consumers (fsv2). */
export interface AclError {
@@ -183,6 +179,7 @@ export class ACLService extends PuterService {
mode: AclMode,
): Promise<boolean> {
if (isSystemActor(actor)) return true;
if (typeof resource?.path !== 'string') return false;
if (resource.path === '/') {
return (PUBLIC_READ_MODES as AclMode[]).includes(mode);
@@ -296,6 +293,10 @@ export class ACLService extends PuterService {
) {
return true;
}
// Capped by the user check above, given the app's own grant.
if (await this.#userAppGrantReaches(actor, await ancestors(), mode))
return true;
}
// Fall back to the permission scan: walk ancestors, any hit wins.
@@ -329,6 +330,47 @@ export class ACLService extends PuterService {
return this.#anyoneWithLinkAllows(actor, ancestry, mode);
}
/**
* Whether the user granted this app `mode` on the node, or on a folder
* above it that the user holds `mode` on too.
*/
async #userAppGrantReaches(
actor: Actor,
ancestors: ReadonlyArray<{ uid: string; path: string }>,
mode: AclMode,
): Promise<boolean> {
const appId = actor.app?.id;
const userId = actor.user?.id;
if (typeof appId !== 'number' || typeof userId !== 'number') {
return false;
}
const rows = await this.stores.permission.readUserAppPerms(
userId,
appId,
ancestors.flatMap((ancestor) =>
this.permissionsFor(ancestor.uid, mode),
),
);
if (rows.length === 0) return false;
const granted = new Set(rows.map((row) => row.permission));
const userActor: Actor = { user: actor.user, effectiveApp: null };
for (const [depth, ancestor] of ancestors.entries()) {
const grantedHere = this.permissionsFor(ancestor.uid, mode).some(
(permission) => granted.has(permission),
);
if (!grantedHere) continue;
// The caller already checked the user on the node itself.
if (depth === 0) return true;
const folder: ResourceDescriptor = {
path: ancestor.path,
resolveAncestors: async () => ancestors.slice(depth),
};
if (await this.check(userActor, folder, mode)) return true;
}
return false;
}
/**
* "Anyone with the link": the owner switched the node open to every
* signed-in account, at `read` or `write`. Honoured only while the owner's
@@ -124,6 +124,106 @@ describe('AuthService (integration)', () => {
return u;
};
const makeDebugApp = async (): Promise<string> => {
const uid = `app-${uuidv4()}`;
await server.clients.db.write(
'INSERT INTO `apps` (`uid`, `name`, `title`, `index_url`, `owner_user_id`) VALUES (?, ?, ?, ?, ?)',
[uid, `n-${uid}`, `t-${uid}`, `https://${uid}.example/`, 1],
);
return uid;
};
describe('createDebugAppToken', () => {
it('mints an app-under-user token on its own expiring, hidden session', async () => {
const user = await makeUser();
const appUid = await makeDebugApp();
const { token, sessionUuid, expiresAt } =
await authService.createDebugAppToken(
user as never,
{ uid: appUid },
{
ttlSeconds: 3600,
label: 'debug',
meta: { debug_admin_username: 'op' },
},
);
// Its own dedicated row, not reusing a kind='app' session.
const row = (
(await server.clients.db.read(
'SELECT `kind`, `app_uid`, `expires_at` FROM `sessions` WHERE `uuid` = ?',
[sessionUuid],
)) as Array<Record<string, unknown>>
)[0];
expect(row?.kind).toBe('debug_app');
expect(row?.app_uid).toBe(appUid);
expect(Number(row?.expires_at)).toBeGreaterThan(
Math.floor(Date.now() / 1000) + 3500,
);
expect(expiresAt).toBeGreaterThan(Date.now() + 3500 * 1000);
// Authenticates as the user acting in the app.
const { actor } = await authService.authenticate(token);
expect(actor?.user.uuid).toBe(user.uuid);
expect(actor?.app?.uid).toBe(appUid);
// Hidden from the user's own session list.
const listed = await authService.listSessions({
user: { id: user.id },
} as Actor);
expect(listed.some((s) => s.uuid === sessionUuid)).toBe(false);
});
it('carries an exp, so the token lapses on its own', async () => {
const user = await makeUser();
const appUid = await makeDebugApp();
const { token } = await authService.createDebugAppToken(
user as never,
{ uid: appUid },
{ ttlSeconds: 1234 },
);
const claims = server.services.token.verify('auth', token) as {
iat: number;
exp: number;
};
expect(claims.exp - claims.iat).toBe(1234);
});
it('records no activity for the user', async () => {
const user = await makeUser();
const appUid = await makeDebugApp();
const { token } = await authService.createDebugAppToken(
user as never,
{ uid: appUid },
{ ttlSeconds: 600 },
);
const touch = vi.spyOn(server.stores.session, 'touch');
try {
const ctx = { ip: '9.9.9.9', userAgent: 'debug' };
expect(
(await authService.authenticate(token, ctx)).actor,
).toBeTruthy();
expect(touch).not.toHaveBeenCalled();
} finally {
touch.mockRestore();
}
});
it('is evicted by a password reset', async () => {
const user = await makeUser();
const appUid = await makeDebugApp();
const { token } = await authService.createDebugAppToken(
user as never,
{ uid: appUid },
{ ttlSeconds: 600 },
);
await authService.revokeInteractiveSessionsForUserId(user.id);
const result = await authService.authenticate(token);
expect(result.actor).toBeUndefined();
expect(result.reauth).toEqual({ reason: 'session_revoked' });
});
});
describe('authenticateFromToken', () => {
it('returns null for a malformed/unverifiable token', async () => {
const actor = await authService.authenticateFromToken('not-a-jwt');
+152 -8
View File
@@ -26,11 +26,13 @@ import {
type Actor,
} from '../../core/actor';
import { HttpError } from '../../core/http/HttpError.js';
import { normalizeOrigin } from '../../core/http/middleware/originGate.js';
import { WEB_AND_EXTENSION_PROTOCOLS } from '../../util/validation.js';
import { awaitLater } from '../../util/concurrency.js';
import { appGenerationStart } from '../../util/appGeneration.js';
import {
ASSET_WINDOW_SECONDS,
DEBUG_APP_SESSION_KIND,
GODMODE_TOKEN_MAX_AGE_SECONDS,
GODMODE_TOKEN_WINDOW_SECONDS,
WEB_WINDOW_SECONDS,
@@ -392,6 +394,68 @@ export class AuthService extends PuterService {
);
}
/**
* Whether `origin` is one of the deployment's own pages that sign in with a
* session of their own (`first_party_session_origins`).
*/
isFirstPartySessionOrigin(origin: unknown): boolean {
if (typeof origin !== 'string' || !origin) return false;
const wanted = normalizeOrigin(origin);
return (this.config.first_party_session_origins ?? []).some(
(listed) => normalizeOrigin(listed) === wanted,
);
}
/**
* Mint the GUI token one of the deployment's own pages signs in with: a
* `web` session of its own under the caller's sign-in, so it is listed and
* revoked apart from that sign-in and ends with it. Only a sign-in itself
* may ask, never a session handed out this way. No session cookie is ever
* issued for the row.
*/
async createFirstPartySessionToken(
actor: Actor,
origin: string,
meta: Record<string, unknown> = {},
): Promise<string> {
if (
!isPlainUserActor(actor) ||
!actor.user?.id ||
!actor.user.uuid ||
actor.session?.kind !== 'web' ||
actor.session.parentUid
) {
throw new HttpError(
403,
'Only a signed-in browser session can sign this page in',
{ legacyCode: 'forbidden' },
);
}
if (!this.isFirstPartySessionOrigin(origin)) {
throw new HttpError(403, 'This origin cannot be signed in', {
legacyCode: 'forbidden',
});
}
const auth_id = this.#authIdFor(actor.user as UserRow);
const row = await this.stores.session.getOrCreateFirstPartySession(
actor.user.id,
{
parentSessionUid: actor.session.uid,
origin: normalizeOrigin(origin),
expiresAt: nowSeconds() + WEB_WINDOW_SECONDS,
meta,
auth_id,
},
);
if (!row?.uuid) {
throw new HttpError(500, 'Could not sign this page in', {
legacyCode: 'internal_error',
});
}
return this.createGuiToken(actor.user as UserRow, row.uuid);
}
/** Sign a session token for an existing session (upgrade from GUI token). */
createSessionTokenForSession(user: UserRow, sessionUuid: string): string {
return this.#signSessionTypeToken(
@@ -787,7 +851,9 @@ export class AuthService extends PuterService {
const visible = [...live.values()].filter(
(row) =>
row.kind !== 'asset' &&
(row.kind !== 'access_token' ||
// Operator-created debug sessions are never the user's own.
row.kind !== DEBUG_APP_SESSION_KIND &&
((row.kind !== 'access_token' && row.kind !== 'web') ||
!row.parent_session_id ||
live.has(row.parent_session_id)),
);
@@ -1075,7 +1141,12 @@ export class AuthService extends PuterService {
});
await this.#revokeCascades(
rows
.filter((row) => row.kind === 'web' || row.kind === 'app')
.filter(
(row) =>
row.kind === 'web' ||
row.kind === 'app' ||
row.kind === DEBUG_APP_SESSION_KIND,
)
.map((row) => row.uuid as string),
);
}
@@ -1089,16 +1160,24 @@ export class AuthService extends PuterService {
legacyCode: 'forbidden',
});
}
const currentUuid = actor.session?.uid;
const rows = await this.stores.session.getByUserId(
// The current sign-in is the session, the sign-in it was handed out
// under, and the sessions handed out under either.
const current = new Set(
[actor.session?.uid, actor.session?.parentUid].filter(Boolean),
);
const rows = (await this.stores.session.getByUserId(
actor.user.id as number,
{ primary: true },
);
)) as SessionRow[];
await this.#revokeCascades(
rows
.filter((row) =>
row.kind === 'web'
? opts.includeCurrent || row.uuid !== currentUuid
? opts.includeCurrent ||
!(
current.has(row.uuid) ||
current.has(row.parent_session_id ?? '')
)
: row.kind === 'app' && opts.includeApps,
)
.map((row) => row.uuid as string),
@@ -1858,6 +1937,49 @@ export class AuthService extends PuterService {
});
}
/**
* An app token acting as `user` in `app` on its own `debug_app` session,
* which expires after `ttlSeconds` and never touches the user's real
* sessions. The caller authorizes who may ask; `meta` lands on the row.
*/
async createDebugAppToken(
user: UserRow,
app: { uid: string },
opts: {
ttlSeconds: number;
label?: string | null;
meta?: Record<string, unknown>;
},
): Promise<{ token: string; sessionUuid: string; expiresAt: number }> {
const authId = this.#authIdFor(user);
const expiresAtSeconds = nowSeconds() + opts.ttlSeconds;
const session = await this.stores.session.create(user.id as number, {
kind: DEBUG_APP_SESSION_KIND,
app_uid: app.uid,
expires_at: expiresAtSeconds,
auth_id: authId,
label: opts.label ?? null,
meta: opts.meta ?? {},
});
const token = this.services.token.sign(
'auth',
{
type: 'app-under-user',
version: '2',
user_uid: user.uuid,
app_uid: app.uid,
session_uid: session.uuid,
auth_id: authId,
},
{ expiresIn: opts.ttlSeconds },
);
return {
token,
sessionUuid: session.uuid as string,
expiresAt: expiresAtSeconds * 1000,
};
}
/**
* Mint the full-access token a godmode app runs on. Only the desktop's own
* web session may ask, and only asking extends it: using the token never
@@ -2702,6 +2824,18 @@ export class AuthService extends PuterService {
const session: SessionRow | null = rawRow;
if (!session) return { invalid: true };
// A session handed to a first-party page lives no longer than the
// sign-in it was handed out under.
if (session.kind === 'web' && session.parent_session_id) {
const parent = (await this.stores.session.getByUuidAny(
session.parent_session_id,
)) as SessionRow | null;
if (!parent) {
return { reauth: { reason: 'session_revoked', auth_id } };
}
const parentEnded = this.#endedSession(parent, auth_id);
if (parentEnded) return parentEnded;
}
if (!(await deployed)) return { invalid: true };
void this.#touchSession(session.uuid, user.id, ctx);
@@ -2760,7 +2894,11 @@ export class AuthService extends PuterService {
// An events handler's calls are background work, not the user being
// active, and its session never expires, so they record no activity.
if (decoded.worker_name !== EVENTS_WORKER_SESSION_NAME)
// Nor do calls on a debug session, which aren't the user's own.
if (
decoded.worker_name !== EVENTS_WORKER_SESSION_NAME &&
session.kind !== DEBUG_APP_SESSION_KIND
)
void this.#touchSession(session.uuid, user.id, ctx);
const actor = this.#buildAppUnderUserActor(user, app, session);
@@ -2990,7 +3128,13 @@ export class AuthService extends PuterService {
return makeActor({
user: this.#actorUserFromRow(user),
session: session
? { uid: session.uuid, kind: session.kind ?? null }
? {
uid: session.uuid,
kind: session.kind ?? null,
...(session.kind === 'web' && session.parent_session_id
? { parentUid: session.parent_session_id }
: {}),
}
: null,
});
}
+1
View File
@@ -644,6 +644,7 @@ export class OIDCService extends PuterService {
// forwarded chain stays in `audit_metadata.ip_fwd`.
signup_ip_forwarded: clientIp,
signup_user_agent: req?.headers?.['user-agent'] ?? null,
signup_tls_fingerprint: req?.tlsFingerprint ?? null,
signup_origin: req?.headers?.origin,
signup_server: this.config.serverId,
referrer: referrer ?? null,
@@ -103,6 +103,11 @@ export interface ForwardableDelivery {
ackId?: string;
/** Client skips its handler: the event is past the handler-chain limit. */
skipHandler?: true;
/**
* Set on a queued delivery with a worker behind it: only connections that
* run the subscription's handler get it, and with none it is declined.
*/
toHandlers?: true;
}
/** What stands in for events a full queue could not carry across. */
@@ -608,10 +613,22 @@ export class EventForwardService extends PuterService {
to: this.region,
});
// Handler rooms are looked up up front, once each however many of the
// batch's deliveries name one.
let heldRooms: Map<string, boolean> | undefined;
try {
heldRooms = await this.services.events.heldHandlerRooms(
items.filter(
(item): item is ForwardDelivery => item.kind === 'delivery',
),
);
} catch (err) {
console.warn('[events] handler room probe failed', err);
}
for (const item of items) {
if (item.kind !== 'delivery') continue;
try {
await this.services.events.deliverForwarded(item);
await this.services.events.deliverForwarded(item, heldRooms);
} catch (err) {
console.warn('[events] forwarded delivery failed', err);
}
@@ -698,6 +715,7 @@ export class EventForwardService extends PuterService {
nack.reason === 'retriable' || nack.reason === 'terminal'
? nack.reason
: 'no_handler',
from,
),
);
for (const outcome of declined)
@@ -797,6 +815,7 @@ export class EventForwardService extends PuterService {
}
: {}),
...(delivery.skipHandler ? { skipHandler: true as const } : {}),
...(delivery.toHandlers ? { toHandlers: true as const } : {}),
};
this.#queueFor().push(region, item);
}
+409 -44
View File
@@ -96,6 +96,10 @@ import type {
ClaimedDelivery,
PendingShed,
} from '../../stores/events/PendingDeliveryStore.js';
import {
HANDLER_RENEW_MS,
type HandledSubscription,
} from '../../stores/events/PresenceStore.js';
import {
isScheduleRow,
runsInWorker,
@@ -732,6 +736,11 @@ export const EVENTS_UNSUBSCRIBE_VERB = 'events.unsubscribe';
export const EVENTS_ACK_VERB = 'events.ack';
/** Client→server: hand a queued delivery back, so the worker takes it now. */
export const EVENTS_NACK_VERB = 'events.nack';
/**
* Client→server: every persistent subscription this connection runs a function
* handler for, replacing what it sent before.
*/
export const EVENTS_HANDLE_VERB = 'events.handle';
export const EVENTS_DELIVERY_CHANNEL = 'events.delivery';
/** Server→client, no ack: the server ended a session subscription itself. */
export const EVENTS_ENDED_CHANNEL = 'events.ended';
@@ -888,8 +897,17 @@ const NO_CREDIT_SWEEP_LEASE_MS = 10 * 60 * 1000;
/** The part of a socket this service uses, so tests need not build one. */
export interface EventSocket {
id: string;
connected: boolean;
rooms: ReadonlySet<string>;
on(event: string, listener: (...args: never[]) => void): unknown;
once(event: string, listener: (...args: never[]) => void): unknown;
join(room: string): unknown;
leave(room: string): unknown;
}
/** Body of the `events.handle` verb. */
export interface HandleRequest {
subIds?: unknown;
}
// -- Errors -----------------------------------------------------------
@@ -1147,6 +1165,37 @@ const deliveryTarget = (row: DispatchSubscription): SocketSpecifier => {
};
};
/**
* The room, inside a row's delivery room, of the connections that said they run
* its handler. A queue row with a worker is offered only to these: any other
* connection would decline it while one of these is still running it.
*/
export const handlerSocketRoom = (room: string, subId: string): string =>
`${room}:sub:${subId}`;
const handlerTarget = (row: DispatchSubscription): string =>
handlerSocketRoom(String(deliveryTarget(row).room), row.subId);
/** Handler rooms one forwarded batch looks up at once. */
const HANDLER_PROBE_CONCURRENCY = 16;
/** One subscription a connection runs the handler for. */
interface HandledRoom extends HandledSubscription {
room: string;
}
/** The handler room a forwarded delivery must reach, if it is limited to one. */
const forwardedHandlerRoom = (item: ForwardDelivery): string | null =>
item.ackRequired && item.toHandlers && item.ackId
? handlerSocketRoom(
String(forwardTarget(item.userId, item.appUid).room),
item.subId,
)
: null;
/** Rows one `events.handle` reads at once. */
const HANDLER_ANNOUNCE_CONCURRENCY = 8;
/** Whether this deployment can address an events worker at all. */
const workerRuntimeReady = (config: IConfig): boolean =>
config.events?.workerRuntime === true && !!config.events?.internalSecret;
@@ -1798,6 +1847,8 @@ export class EventsService extends PuterService {
});
}) as (...args: never[]) => void);
const releaseHandlers = this.#trackHandlers(actor, userId, socket);
// Presence is per (user, app) and per region, so it is the connection
// rather than the subscription that moves it — a client that has not
// subscribed yet is still somewhere, and that is what a peer needs.
@@ -1808,6 +1859,7 @@ export class EventsService extends PuterService {
});
socket.once('disconnect', (() => {
releaseHandlers();
void this.reapSocket(userId, socket.id);
void this.services.eventForward
.noteDisconnect(actor, socket.id)
@@ -1817,13 +1869,199 @@ export class EventsService extends PuterService {
}) as (...args: never[]) => void);
}
/**
* Install `events.handle` on one connection and keep the region's record of
* the handlers it runs renewed. Announces, renewals and the release on
* disconnect take turns, so a slower earlier one can't undo a later one.
* Returns the release.
*/
#trackHandlers(
actor: Actor,
userId: number,
socket: EventSocket,
): () => void {
const handling = new Map<string, HandledRoom>();
let turns: Promise<void> = Promise.resolve();
const takeTurn = (run: () => Promise<void>): Promise<void> => {
const previous = turns;
const turn = (async () => {
try {
await previous;
} catch {
// Answered on its own turn.
}
await run();
})();
turns = turn;
return turn;
};
let renewTimer: ReturnType<typeof setInterval> | null = null;
const stopRenewing = () => {
if (renewTimer) clearInterval(renewTimer);
renewTimer = null;
};
const renew = async () => {
if (!socket.connected || handling.size === 0) return;
try {
await this.stores.presence.addHandlerSocket(userId, socket.id, [
...handling.values(),
]);
} catch (err) {
console.warn('[events] handler renewal failed', err);
}
};
socket.on(EVENTS_HANDLE_VERB, ((
payload: HandleRequest,
ack: unknown,
) => {
const turn = takeTurn(async () => {
try {
await this.#announceHandlers(
actor,
socket,
handling,
payload,
);
} finally {
if (handling.size === 0 || !socket.connected)
stopRenewing();
else if (!renewTimer) {
renewTimer = setInterval(
() => void takeTurn(renew),
HANDLER_RENEW_MS,
);
renewTimer.unref?.();
}
}
});
void this.#answer(ack, async () => {
await turn;
return {};
});
}) as (...args: never[]) => void);
return () => {
stopRenewing();
void takeTurn(async () => {
stopRenewing();
const released = [...handling.values()];
handling.clear();
await this.#forgetHandlers(userId, socket.id, released);
});
};
}
/** Take a connection out of the region's record for these subscriptions. */
async #forgetHandlers(
userId: number,
socketId: string,
handled: readonly HandledRoom[],
): Promise<void> {
try {
await this.stores.presence.removeHandlerSocket(
userId,
socketId,
handled,
);
} catch (err) {
// Lapses on its own once nothing renews it.
console.warn('[events] handler release failed', err);
}
}
/**
* Whether a connection in this region runs the subscription's handler: one
* on this node, or one the region's record holds. A failed read is a no.
*/
async #handlerHeld(
userId: number,
appUid: string | null,
subId: string,
room: string,
): Promise<boolean> {
if (this.services.socket.has({ room })) return true;
try {
return await this.stores.presence.holdsHandler(
userId,
presenceApp(appUid),
subId,
);
} catch (err) {
console.warn('[events] handler lookup failed', err);
return false;
}
}
/**
* Whether each handler room the forwarded deliveries name has a connection
* in this region, for `deliverForwarded`, each room looked up once.
*/
async heldHandlerRooms(
items: readonly ForwardDelivery[],
): Promise<Map<string, boolean>> {
const held = new Map<string, boolean>();
if (!this.enabled) return held;
const rooms = new Map<string, ForwardDelivery>();
for (const item of items) {
const room = forwardedHandlerRoom(item);
if (room && !rooms.has(room)) rooms.set(room, item);
}
if (rooms.size === 0) return held;
const list = [...rooms];
const outcomes = await runWithConcurrencyLimitSettled(
list,
HANDLER_PROBE_CONCURRENCY,
([room, item]) =>
this.#handlerHeld(item.userId, item.appUid, item.subId, room),
);
list.forEach(([room], i) => {
const outcome = outcomes[i];
held.set(
room,
outcome?.status === 'fulfilled' && outcome.value === true,
);
});
return held;
}
/**
* Put down one delivery another region handed us. Nothing is metered or
* re-checked: the region that emitted it did both, and this one holds no
* state about it at all.
*/
async deliverForwarded(item: ForwardDelivery): Promise<void> {
async deliverForwarded(
item: ForwardDelivery,
heldRooms?: ReadonlyMap<string, boolean>,
): Promise<void> {
if (!this.enabled) return;
let target = forwardTarget(item.userId, item.appUid);
const room = forwardedHandlerRoom(item);
if (room && item.ackId) {
const held =
heldRooms?.get(room) ??
(await this.#handlerHeld(
item.userId,
item.appUid,
item.subId,
room,
));
// Nobody here runs its handler: hand it straight back, so the
// worker takes it without waiting out the lease.
if (!held) {
if (item.origin)
this.services.eventForward.relayNack(
item.origin,
item.userId,
item.subId,
item.ackId,
'no_handler',
);
return;
}
target = { room };
}
const envelope: DeliveryEnvelope = {
subId: item.subId,
event: item.event,
@@ -1837,7 +2075,7 @@ export class EventsService extends PuterService {
...(item.skipHandler ? { skipHandler: true as const } : {}),
};
await this.services.socket.send(
forwardTarget(item.userId, item.appUid),
target,
EVENTS_DELIVERY_CHANNEL,
envelope,
);
@@ -1862,17 +2100,18 @@ export class EventsService extends PuterService {
if (row.suspendedAt === null) await this.#drain(row);
}
/** Apply a decline whose client gave it in another region. */
/** Apply a decline whose client, or `fromRegion` itself, gave it there. */
async declineRelayed(
holderUserId: number,
subId: string,
entryId: string,
reason: DeclineReason,
fromRegion: string,
): Promise<void> {
if (!this.enabled || !subId || !entryId) return;
const row = await this.stores.durableSubscription.getBySubId(subId);
if (!row || row.holderUserId !== holderUserId) return;
await this.#declined(row, entryId, reason);
await this.#declined(row, entryId, reason, fromRegion);
}
async #answer<T extends object>(
@@ -2488,7 +2727,90 @@ export class EventsService extends PuterService {
);
return;
}
await this.#declined(row, entryId, reason);
await this.#declined(row, entryId, reason, null);
}
/**
* Put a connection in the handler room of each queue row it names and out
* of the ones it no longer names. A row it can't be shown to hold, or whose
* delivery room it isn't in, is skipped without saying so.
*/
async #announceHandlers(
actor: Actor,
socket: EventSocket,
handling: Map<string, HandledRoom>,
request: HandleRequest,
): Promise<void> {
if (!this.enabled) throw disabled();
const holderUserId = actor.user?.id;
if (holderUserId === undefined) throw disabled();
const raw = request?.subIds;
if (!Array.isArray(raw))
throw badRequest('subIds must be an array', 'bad_request');
const wanted = new Set(
raw
.filter(
(subId): subId is string =>
typeof subId === 'string' && subId.length > 0,
)
.slice(0, EVENTS_DURABLE_SUBSCRIPTIONS_PER_APP.limit),
);
await spendUserWindow(EVENTS_ACK_LIMIT, holderUserId);
const dropped: HandledRoom[] = [];
for (const [subId, handled] of handling) {
if (wanted.has(subId)) continue;
handling.delete(subId);
dropped.push(handled);
}
// Out of the record before the room: a record naming a connection
// that left sends events to nobody until their lease lapses.
await this.#forgetHandlers(holderUserId, socket.id, dropped);
for (const { room } of dropped) await socket.leave(room);
const added = [...wanted].filter((subId) => !handling.has(subId));
const rows = await runWithConcurrencyLimitSettled(
added,
HANDLER_ANNOUNCE_CONCURRENCY,
(subId) => this.stores.durableSubscription.getBySubId(subId),
);
const joined: HandledRoom[] = [];
for (const outcome of rows) {
if (outcome.status !== 'fulfilled') continue;
const row = outcome.value;
if (
!row ||
row.holderUserId !== holderUserId ||
!rowInActorScope(actor, row) ||
row.delivery !== 'single' ||
!runsInWorker(row)
)
continue;
if (!socket.rooms.has(String(deliveryTarget(row).room))) continue;
// Joined after a disconnect, the id would stay in the room.
if (!socket.connected) return;
const handled: HandledRoom = {
room: handlerTarget(row),
appUid: presenceApp(row.appUid),
subId: row.subId,
};
handling.set(row.subId, handled);
await socket.join(handled.room);
joined.push(handled);
}
// Recorded only once in the room. A failed write is retried by the
// next renewal; until then this connection is not offered the row.
if (joined.length === 0) return;
try {
await this.stores.presence.addHandlerSocket(
holderUserId,
socket.id,
joined,
);
} catch (err) {
console.warn('[events] handler record failed', err);
}
}
/** The row an ack or decline names, and the region it belongs to. */
@@ -2530,18 +2852,22 @@ export class EventsService extends PuterService {
/**
* Take a declined delivery off the client's lease. A failed client run
* counts like a failed worker run: retried after the backoff, or dropped. A
* client with no handler costs nothing and the worker runs it now; a row
* with no worker has nothing to hand it to, so its lease is left to lapse.
* client with no handler costs nothing: the next region with the app open
* that hasn't had it gets it, and once none is left the worker runs it now.
* A row with no worker has nothing to hand it to, so its lease is left to
* lapse. `fromRegion` is where the decline came from, `null` for here.
*/
async #declined(
row: DurableSubscription,
entryId: string,
reason: DeclineReason,
fromRegion: string | null,
): Promise<void> {
if (reason === 'no_handler' && !runsInWorker(row)) return;
const claimed = await this.stores.pendingDelivery.takeFromSocket(
row.subId,
entryId,
fromRegion,
);
if (!claimed) return;
@@ -2558,8 +2884,17 @@ export class EventsService extends PuterService {
// later drain asks again.
const serving = await this.#stillOwed(row);
if (!serving) return;
if (await this.#runInWorker(serving, claimedAsServed(serving, claimed)))
await this.#drain(row);
const served = claimedAsServed(serving, claimed);
const region = await this.services.eventForward.candidateRegion(
serving.holderUserId,
serving.appUid,
served.triedRegions,
);
if (region) {
await this.#handToRegion(serving, served, region, true);
return;
}
if (await this.#runInWorker(serving, served)) await this.#drain(row);
}
// -- Handlers ----------------------------------------------------
@@ -6233,16 +6568,25 @@ export class EventsService extends PuterService {
// suspension's window and goes out if the balance comes back.
if (!(await this.#chargeable(meter, row.subId))) return false;
const target = deliveryTarget(row);
const hasWorkerFallback = runsInWorker(row);
const appTarget = deliveryTarget(row);
const target: SocketSpecifier = hasWorkerFallback
? { room: handlerTarget(row) }
: appTarget;
// Region-wide: `has` only sees this node's sockets, and the room send
// below reaches a sibling node's just the same.
const heldHere =
this.services.socket.has(target) ||
(await this.services.eventForward.heldInRegion(
row.holderUserId,
row.appUid,
));
const heldHere = hasWorkerFallback
? await this.#handlerHeld(
row.holderUserId,
row.appUid,
row.subId,
String(target.room),
)
: this.services.socket.has(appTarget) ||
(await this.services.eventForward.heldInRegion(
row.holderUserId,
row.appUid,
));
// A row with no worker to fall back to has nothing left to try once
// its socket budget is spent. Without this, a socket that disappeared
@@ -6273,17 +6617,15 @@ export class EventsService extends PuterService {
)
: null;
if (here || region) {
if (region) {
await this.#handToRegion(row, claimed, region);
return false;
}
if (here) {
await this.stores.pendingDelivery.leaseToSocket(
row.subId,
claimed.entryId,
);
if (region)
await this.stores.pendingDelivery.recordRemoteAttempt(
row.subId,
claimed.entryId,
region,
);
const envelope: DeliveryEnvelope = {
subId: row.subId,
event: claimed.event,
@@ -6291,28 +6633,13 @@ export class EventsService extends PuterService {
ackId: claimed.entryId,
};
const bill = await this.#firstAttempt(row.subId, claimed);
if (region) {
this.services.eventForward.handOff(region, {
holderUserId: row.holderUserId,
appUid: row.appUid,
subId: row.subId,
event: claimed.event,
ackRequired: true,
ackId: claimed.entryId,
});
this.#delivered(envelope, meter, bill);
} else {
await this.#send({
target,
envelope,
meter,
bill,
});
}
singleAttempt.add(1, {
target: region ? 'remote-socket' : 'local-socket',
result: 'sent',
await this.#send({
target,
envelope,
meter,
bill,
});
singleAttempt.add(1, { target: 'local-socket', result: 'sent' });
return false;
}
@@ -6325,6 +6652,44 @@ export class EventsService extends PuterService {
return this.#runInWorker(row, claimed);
}
/**
* Forward one claimed delivery to a region's connections under a socket
* lease, `reoffer` after another region declined it.
*/
async #handToRegion(
row: DispatchSubscription,
claimed: ClaimedDelivery,
region: string,
reoffer = false,
): Promise<void> {
await this.stores.pendingDelivery.leaseToSocket(
row.subId,
claimed.entryId,
{ region, reoffer },
);
const bill = await this.#firstAttempt(row.subId, claimed);
this.services.eventForward.handOff(region, {
holderUserId: row.holderUserId,
appUid: row.appUid,
subId: row.subId,
event: claimed.event,
ackRequired: true,
ackId: claimed.entryId,
...(runsInWorker(row) ? { toHandlers: true as const } : {}),
});
this.#delivered(
{
subId: row.subId,
event: claimed.event,
ackRequired: true,
ackId: claimed.entryId,
},
meterFor(row),
bill,
);
singleAttempt.add(1, { target: 'remote-socket', result: 'sent' });
}
/**
* Run one claimed delivery in the events worker. Returns whether it
* settled, which is what says the next one may go out now.
@@ -57,6 +57,11 @@ export interface ForwardDelivery {
origin?: string;
/** Client skips its handler: the event is past the handler-chain limit. */
skipHandler?: true;
/**
* Set on a queued delivery with a worker behind it: only connections that
* run the subscription's handler get it, and with none it is declined.
*/
toHandlers?: true;
}
/** A client's settle, on its way back to the region that owns the lease. */
+253 -2
View File
@@ -44,7 +44,11 @@ import type { DurableSubscription } from '../../stores/events/types.js';
import type { FSEntry } from '../../stores/fs/FSEntry.js';
import type { IConfig } from '../../types.js';
import { EventForwardService } from './EventForwardService.js';
import { EventsService, type DeliveryEnvelope } from './EventsService.js';
import {
EventsService,
handlerSocketRoom,
type DeliveryEnvelope,
} from './EventsService.js';
import { appSocketRoom } from '../socket/SocketService.js';
import type {
ForwardBatch,
@@ -199,6 +203,9 @@ interface Region {
subscriptions: EventSubscriptionStore;
forward: EventForwardService;
events: EventsService;
socket: {
send: ReturnType<typeof vi.fn>;
};
/** Every addressed POST this region made, whether or not it arrived. */
posts: Array<{ peerId: string; batch: ForwardBatch }>;
/** Rooms this region terminates a socket for. */
@@ -368,6 +375,7 @@ const makeRegion = (
region.sent.push(data as DeliveryEnvelope);
}),
};
region.socket = socket;
const broadcast = {
get regionId() {
@@ -1474,7 +1482,8 @@ describe('a delivery owed to exactly one consumer', () => {
it('sends a decline home, and the emitting region runs the worker', async () => {
const west = makeRegion('west', ['east']);
const east = makeRegion('east', ['west']);
east.rooms.add(appSocketRoom(userId, APP_UID));
const room = appSocketRoom(userId, APP_UID);
east.rooms.add(room);
await east.forward.noteConnect(actorFor(APP_UID), 'tab-1');
workerOutcome = 'settled';
const row = await register(west, {
@@ -1482,6 +1491,7 @@ describe('a delivery owed to exactly one consumer', () => {
delivery: 'single',
handlerName: 'onWrite',
});
east.rooms.add(handlerSocketRoom(room, row.subId));
await dispatch(west);
await posted(west);
@@ -1508,6 +1518,124 @@ describe('a delivery owed to exactly one consumer', () => {
});
});
it('is declined home by a region where no connection runs its handler', async () => {
const west = makeRegion('west', ['east']);
const east = makeRegion('east', ['west']);
east.rooms.add(appSocketRoom(userId, APP_UID));
await east.forward.noteConnect(actorFor(APP_UID), 'tab-1');
workerOutcome = 'settled';
const row = await register(west, {
appUid: APP_UID,
delivery: 'single',
handlerName: 'onWrite',
});
await dispatch(west);
await posted(west);
await vi.waitFor(() => expect(west.invoked).toHaveLength(1), {
timeout: 3_000,
interval: 20,
});
// No client there saw it, so none can also be running it.
expect(east.sent).toEqual([]);
expect(deliveriesIn(west)[0]).toMatchObject({ toHandlers: true });
expect(east.posts.at(-1)?.batch.items[0]).toMatchObject({
kind: 'nack',
subId: row.subId,
reason: 'no_handler',
});
await vi.waitFor(
async () => expect(await west.pending.depth(row.subId)).toBe(0),
{ timeout: 3_000, interval: 20 },
);
});
it('reaches the region whose client runs the handler after a nearer one without it declines', async () => {
const west = makeRegion('west', ['east', 'south']);
const east = makeRegion('east', ['west', 'south']);
const south = makeRegion('south', ['west', 'east']);
const room = appSocketRoom(userId, APP_UID);
south.rooms.add(room);
await south.forward.noteConnect(actorFor(APP_UID), 'tab-s');
await quiet(5);
east.rooms.add(room);
await east.forward.noteConnect(actorFor(APP_UID), 'tab-e');
workerOutcome = 'settled';
const row = await register(west, {
appUid: APP_UID,
delivery: 'single',
handlerName: 'onWrite',
});
// South's handler connection is on another of its nodes.
await south.presence.addHandlerSocket(userId, 'tab-s', [
{ appUid: APP_UID, subId: row.subId },
]);
await dispatch(west);
await arrived(south);
expect(deliveriesIn(west).map((item) => item.subId)).toEqual([
row.subId,
row.subId,
]);
expect(west.posts.map((post) => post.peerId)).toEqual([
'east',
'south',
]);
expect(east.sent).toEqual([]);
expect(south.socket.send.mock.calls[0][0]).toEqual({
room: handlerSocketRoom(room, row.subId),
});
const envelope = south.sent[0];
await south.events.ackDelivery(actorFor(APP_UID), {
subId: row.subId,
id: envelope.ackId,
origin: envelope.origin,
});
await vi.waitFor(
async () => expect(await west.pending.depth(row.subId)).toBe(0),
{ timeout: 3_000, interval: 20 },
);
expect(west.invoked).toEqual([]);
expect(south.sent).toHaveLength(1);
expect(west.metered).toHaveLength(1);
});
it('runs the worker once after every region with the app open declines', async () => {
const west = makeRegion('west', ['east', 'south']);
const east = makeRegion('east', ['west', 'south']);
const south = makeRegion('south', ['west', 'east']);
const room = appSocketRoom(userId, APP_UID);
for (const region of [east, south]) {
region.rooms.add(room);
await region.forward.noteConnect(actorFor(APP_UID), 'tab-1');
}
workerOutcome = 'settled';
const row = await register(west, {
appUid: APP_UID,
delivery: 'single',
handlerName: 'onWrite',
});
await dispatch(west);
await vi.waitFor(() => expect(west.invoked).toHaveLength(1), {
timeout: 3_000,
interval: 20,
});
expect(deliveriesIn(west)).toHaveLength(2);
expect(east.sent).toEqual([]);
expect(south.sent).toEqual([]);
await vi.waitFor(
async () => expect(await west.pending.depth(row.subId)).toBe(0),
{ timeout: 3_000, interval: 20 },
);
await quiet(100);
expect(west.invoked).toHaveLength(1);
});
it('never settles a lease in the region the client happened to reach', async () => {
const west = makeRegion('west', ['east']);
const east = makeRegion('east', ['west']);
@@ -1803,6 +1931,129 @@ describe('receiving a batch', () => {
});
});
describe('receiving queued deliveries that only a handler connection takes', () => {
const queued = (subId: string): ForwardDelivery => ({
kind: 'delivery',
userId,
appUid: APP_UID,
subId,
event: {
id: `e-${subId}`,
subject: 'fs:/u7/Documents',
op: 'write',
uid: 'node-1',
path: '/u7/Documents/notes.txt',
self: true,
seq: 0,
ts: 1,
},
ackRequired: true,
ackId: `entry-${subId}`,
origin: 'west',
toHandlers: true,
});
const plain = (subId: string): ForwardDelivery => ({
kind: 'delivery',
userId,
appUid: APP_UID,
subId,
event: queued(subId).event,
});
const handlerRoom = (subId: string) =>
handlerSocketRoom(appSocketRoom(userId, APP_UID), subId);
/** A connection on another node of east that runs the handler. */
const handledElsewhere = (east: Region, subId: string, socketId = 'tab-1') =>
east.presence.addHandlerSocket(userId, socketId, [
{ appUid: APP_UID, subId },
]);
it("sends what this region's record names a handler connection for, and declines the rest", async () => {
const east = makeRegion('east', ['west']);
const nack = vi
.spyOn(east.forward, 'relayNack')
.mockImplementation(() => {});
await handledElsewhere(east, 'h-0');
east.rooms.add(handlerRoom('h-2'));
await east.forward.receive(
{
from: 'west',
items: [
queued('h-0'),
queued('h-1'),
queued('h-2'),
queued('h-3'),
plain('plain'),
],
},
'west',
);
// Still sent in the batch's order, and each empty room declined.
expect(east.sent.map((envelope) => envelope.subId)).toEqual([
'h-0',
'h-2',
'plain',
]);
expect(east.socket.send.mock.calls[0][0]).toEqual({
room: handlerRoom('h-0'),
});
expect(nack.mock.calls.map((call) => call[2])).toEqual(['h-1', 'h-3']);
expect(nack.mock.calls.every((call) => call[4] === 'no_handler')).toBe(
true,
);
});
it('looks a room up once however many of its deliveries the batch carries', async () => {
const east = makeRegion('east', ['west']);
await handledElsewhere(east, 'same');
const lookup = vi.spyOn(east.presence, 'holdsHandler');
await east.forward.receive(
{
from: 'west',
items: [queued('same'), queued('same'), queued('same')],
},
'west',
);
expect(lookup).toHaveBeenCalledTimes(1);
expect(east.sent).toHaveLength(3);
});
it('declines what it could not look up', async () => {
const east = makeRegion('east', ['west']);
const nack = vi
.spyOn(east.forward, 'relayNack')
.mockImplementation(() => {});
await handledElsewhere(east, 'h-0');
vi.spyOn(east.presence, 'holdsHandler').mockRejectedValue(
new Error('redis down'),
);
await east.forward.receive(
{ from: 'west', items: [queued('h-0')] },
'west',
);
expect(east.sent).toEqual([]);
expect(nack).toHaveBeenCalledTimes(1);
});
it('never looks anything up for a batch without such a delivery', async () => {
const east = makeRegion('east', ['west']);
const lookup = vi.spyOn(east.presence, 'holdsHandler');
await east.forward.receive(
{ from: 'west', items: [plain('a'), plain('b')] },
'west',
);
expect(lookup).not.toHaveBeenCalled();
expect(east.sent).toHaveLength(2);
});
});
// -- Observability ------------------------------------------------------
describe('presence write metrics', () => {
@@ -31,12 +31,24 @@ import {
import type { Actor } from '../../core/actor.js';
import { EventSubscriptionStore } from '../../stores/events/EventSubscriptionStore.js';
import { PendingDeliveryStore } from '../../stores/events/PendingDeliveryStore.js';
import {
HANDLER_LIVE_MS,
HANDLER_RENEW_MS,
PresenceStore,
} from '../../stores/events/PresenceStore.js';
import type { DurableSubscription } from '../../stores/events/types.js';
import type { FSEntry } from '../../stores/fs/FSEntry.js';
import type { IConfig } from '../../types.js';
import type { UsageInput } from '../metering/types.js';
import { EVENTS_COSTS } from './costs.js';
import { EventsService, type DeliveryEnvelope } from './EventsService.js';
import type { SocketSpecifier } from '../socket/SocketService.js';
import {
EVENTS_ACK_VERB,
EVENTS_HANDLE_VERB,
EVENTS_NACK_VERB,
EventsService,
type DeliveryEnvelope,
} from './EventsService.js';
import { fsAnchorToken } from './subjects.js';
import type {
WorkerInvocation,
@@ -60,6 +72,20 @@ let subscriptions: EventSubscriptionStore;
let pending: PendingDeliveryStore;
let service: EventsService;
let sent: DeliveryEnvelope[];
/** Where each socket send was addressed, aligned with `sent`. */
let sentTo: SocketSpecifier[];
/** Connections modelled one by one, for the tests that need several. */
let tabs: FakeTab[];
/**
* A cluster-wide room count, which costs a request to every node. Answers
* truly, so a test can only fail on it by asking.
*/
const askedEveryNode = vi.fn(async (room: string) =>
tabs.length === 0
? socketConnected
: tabs.some((tab) => tab.rooms.has(room)),
);
let presence: PresenceStore;
let delivered: DeliveryEnvelope[];
let invoked: WorkerInvocation[];
let rows: Map<string, DurableSubscription>;
@@ -95,6 +121,8 @@ let candidateRegionImpl: (
) => Promise<string | null> = async () => null;
/** The regions already tried that each `candidateRegion` call carried. */
let candidateRegionCalls: string[][] = [];
/** Each delivery forwarded to another region, and where. */
let handOffs: Array<{ region: string; delivery: { ackId?: string } }> = [];
const entry = (over: Partial<FSEntry> = {}): FSEntry =>
({
@@ -200,7 +228,11 @@ beforeEach(async () => {
workerRetryAfterMs = undefined;
candidateRegionImpl = async () => null;
candidateRegionCalls = [];
handOffs = [];
sent = [];
sentTo = [];
tabs = [];
askedEveryNode.mockClear();
delivered = [];
invoked = [];
rows = new Map();
@@ -224,6 +256,7 @@ beforeEach(async () => {
{ redis } as never,
{} as never,
);
presence = new PresenceStore({} as IConfig, { redis } as never, {} as never);
entries.set(`uid:${anchorUid()}`, entry({ uid: anchorUid(), isDir: true }));
@@ -237,6 +270,7 @@ beforeEach(async () => {
{
eventSubscription: subscriptions,
pendingDelivery: pending,
presence,
durableSubscription: {
warmRegion: async () => false,
getBySubId: async (subId: string) => {
@@ -295,7 +329,9 @@ beforeEach(async () => {
},
heldInRegion: async () => false,
fanOut: async () => undefined,
handOff: () => undefined,
handOff: (region: string, delivery: { ackId?: string }) => {
handOffs.push({ region, delivery });
},
relayAck: () => undefined,
relayNack: () => undefined,
announceWatch: () => undefined,
@@ -303,10 +339,24 @@ beforeEach(async () => {
announceGeneration: () => undefined,
},
socket: {
send: vi.fn(async (_spec, _key, data) => {
send: vi.fn(async (spec: SocketSpecifier, _key, data) => {
sent.push(data as DeliveryEnvelope);
sentTo.push(spec);
for (const tab of tabs)
if (tab.rooms.has(String(spec.room)))
tab.receive(data as DeliveryEnvelope);
}),
has: () => socketConnected,
// With no tabs modelled, a connection here is one that runs
// the handler. Only this node's tabs count.
has: (spec: SocketSpecifier) =>
tabs.length === 0
? socketConnected
: tabs.some(
(tab) =>
!tab.elsewhere &&
tab.rooms.has(String(spec.room)),
),
occupied: askedEveryNode,
},
acl: {
check: async () => true,
@@ -1291,3 +1341,516 @@ describe('the budgets a `single` is delivered under', () => {
await expect(pending.depth(row.subId)).resolves.toBe(1);
});
});
/**
* One app connection as puter.js drives it: a queued delivery for a
* subscription it runs a handler for is kept (the test acks it), and one it
* has no handler for is declined at once.
*/
class FakeTab {
readonly rooms = new Set<string>();
readonly verbs = new Map<string, (payload: unknown, ack: unknown) => void>();
readonly received: DeliveryEnvelope[] = [];
readonly answers: Promise<unknown>[] = [];
connected = true;
readonly handlers = new Set<string>();
readonly #onDisconnect: Array<() => void> = [];
constructor(
readonly id: string,
readonly actor: Actor,
/** On another node in the region: `has` can't see it, a room send can. */
readonly elsewhere = false,
) {
this.rooms.add(id);
this.rooms.add(`u${actor.user!.id}:a${actor.effectiveApp!.uid}`);
}
on(event: string, listener: (...args: never[]) => void): void {
this.verbs.set(
event,
listener as unknown as (payload: unknown, ack: unknown) => void,
);
}
once(event: string, listener: (...args: never[]) => void): void {
if (event === 'disconnect')
this.#onDisconnect.push(listener as unknown as () => void);
}
join(room: string): void {
if (this.connected) this.rooms.add(room);
}
leave(room: string): void {
this.rooms.delete(room);
}
disconnect(): void {
this.vanish();
for (const listener of this.#onDisconnect) listener();
}
/** Gone with the node it was on: nothing on the server hears it go. */
vanish(): void {
this.connected = false;
this.rooms.clear();
}
/** A verb the server doesn't install goes unanswered, as on the wire. */
verb(name: string, payload: unknown): Promise<unknown> {
const listener = this.verbs.get(name);
const answer = listener
? new Promise((resolve) => listener(payload, resolve))
: Promise.resolve(undefined);
this.answers.push(answer);
return answer;
}
receive(envelope: DeliveryEnvelope): void {
this.received.push(envelope);
if (envelope.ackRequired && !this.handlers.has(envelope.subId))
void this.verb(EVENTS_NACK_VERB, {
subId: envelope.subId,
id: envelope.ackId,
reason: 'no_handler',
});
}
/** Register a handler and tell the server, as puter.js does. */
async handle(...subIds: string[]): Promise<void> {
for (const subId of subIds) this.handlers.add(subId);
await this.verb(EVENTS_HANDLE_VERB, { subIds: [...this.handlers] });
}
async ack(envelope: DeliveryEnvelope): Promise<void> {
await this.verb(EVENTS_ACK_VERB, {
subId: envelope.subId,
id: envelope.ackId,
});
}
}
const appActorFor = (): Actor =>
({
user: { id: userId, uuid: `user-${userId}`, username: `u${userId}` },
effectiveApp: { uid: `app-${seq}` },
}) as unknown as Actor;
const openTab = (id: string, elsewhere = false): FakeTab => {
const tab = new FakeTab(`${id}-${seq}`, appActorFor(), elsewhere);
service.attachSocket(tab, tab.actor);
tabs.push(tab);
return tab;
};
describe('a queued delivery with several connections of the app open', () => {
it('runs only in the tab that has the handler while another tab is open', async () => {
workerOutcome = 'settled';
const row = await register();
const withHandler = openTab('a');
const without = openTab('b');
await withHandler.handle(row.subId);
await dispatch();
await Promise.all(without.answers);
await handedOver();
// The other tab never sees it, so it can't decline it into the worker.
expect(invoked).toEqual([]);
expect(withHandler.received).toHaveLength(1);
expect(without.received).toEqual([]);
await withHandler.ack(withHandler.received[0]);
await expect(pending.depth(row.subId)).resolves.toBe(0);
expect(invoked).toEqual([]);
expect(metered).toHaveLength(1);
});
it('asks no other node about a pair with no connection in the region', async () => {
socketConnected = false;
workerOutcome = 'settled';
const row = await register();
await dispatch();
expect(askedEveryNode).not.toHaveBeenCalled();
expect(sent).toEqual([]);
expect(invoked).toHaveLength(1);
await expect(pending.depth(row.subId)).resolves.toBe(0);
});
it('goes straight to the worker when no open tab has the handler, asking no other node', async () => {
workerOutcome = 'settled';
const row = await register();
const tab = openTab('a');
const sibling = openTab('b', true);
await dispatch();
expect(askedEveryNode).not.toHaveBeenCalled();
expect(tab.received).toEqual([]);
expect(sibling.received).toEqual([]);
expect(invoked).toHaveLength(1);
await expect(pending.depth(row.subId)).resolves.toBe(0);
expect(metered).toHaveLength(1);
});
it('is not handed to the worker as well when two tabs both have the handler', async () => {
workerOutcome = 'settled';
const row = await register();
const first = openTab('a');
const second = openTab('b');
await first.handle(row.subId);
await second.handle(row.subId);
await dispatch();
await first.ack(first.received[0]);
await second.ack(second.received[0]);
expect(invoked).toEqual([]);
await expect(pending.depth(row.subId)).resolves.toBe(0);
expect(metered).toHaveLength(1);
});
it('stops sending to a tab once it drops the handler', async () => {
workerOutcome = 'settled';
const row = await register();
const tab = openTab('a');
await tab.handle(row.subId);
tab.handlers.clear();
await tab.verb(EVENTS_HANDLE_VERB, { subIds: [] });
await dispatch();
expect(tab.received).toEqual([]);
expect(invoked).toHaveLength(1);
});
it('takes the delivery from a tab that disconnected holding it, once its lease lapses', async () => {
workerOutcome = 'settled';
const row = await register();
const tab = openTab('a');
await tab.handle(row.subId);
await dispatch();
expect(tab.received).toHaveLength(1);
tab.disconnect();
jump(EVENTS_SOCKET_LEASE_MS + 1);
await service.sweepPending();
expect(invoked).toHaveLength(1);
await expect(pending.depth(row.subId)).resolves.toBe(0);
// Its ack, sent late from a new connection, settles nothing more.
const back = openTab('a2');
await back.ack(tab.received[0]);
expect(invoked).toHaveLength(1);
expect(metered).toHaveLength(1);
});
it('ignores a handler announce for a row the connection does not hold', async () => {
workerOutcome = 'settled';
const row = await register();
const stranger = new FakeTab(`x-${seq}`, {
user: { id: userId + 1, uuid: 'other', username: 'other' },
effectiveApp: { uid: `app-${seq}` },
} as unknown as Actor);
service.attachSocket(stranger, stranger.actor);
tabs.push(stranger);
await stranger.handle(row.subId);
expect([...stranger.rooms].some((room) => room.includes(':sub:'))).toBe(
false,
);
await dispatch();
expect(stranger.received).toEqual([]);
expect(invoked).toHaveLength(1);
});
it('still sends a row with no worker to the whole app room', async () => {
const row = await register({ handlerName: null });
const tab = openTab('a');
await dispatch();
expect(sentTo).toEqual([{ room: `u${userId}:aapp-${seq}` }]);
expect(tab.received).toHaveLength(1);
expect(row.subId).toBe(tab.received[0].subId);
});
});
describe("the region's record of handler connections", () => {
/** Connections the region's record names for the row, live or not. */
const recorded = async (subId: string): Promise<string[]> => {
const [key] = (await redis.keys('*ev:phs:*')).filter((one: string) =>
one.endsWith(`:${subId}`),
);
return key ? redis.zrange(key, 0, -1) : [];
};
it('offers it to a handler connection on another node in the region, asking no other node', async () => {
workerOutcome = 'settled';
const row = await register();
const sibling = openTab('a', true);
await sibling.handle(row.subId);
await dispatch();
expect(askedEveryNode).not.toHaveBeenCalled();
expect(sibling.received).toHaveLength(1);
await sibling.ack(sibling.received[0]);
await expect(pending.depth(row.subId)).resolves.toBe(0);
expect(invoked).toEqual([]);
expect(metered).toHaveLength(1);
});
it('runs it in the worker once when the record names a connection whose node died', async () => {
workerOutcome = 'settled';
const row = await register();
const sibling = openTab('a', true);
await sibling.handle(row.subId);
sibling.vanish();
await dispatch();
// Sent to a room nobody is in; the lease is what hands it on.
expect(sent).toHaveLength(1);
expect(sibling.received).toEqual([]);
expect(invoked).toEqual([]);
jump(EVENTS_SOCKET_LEASE_MS + 1);
await service.sweepPending();
expect(invoked).toHaveLength(1);
await expect(pending.depth(row.subId)).resolves.toBe(0);
expect(metered).toHaveLength(1);
});
it('stops counting a dead node’s connection once its renewals stop', async () => {
workerOutcome = 'settled';
const row = await register();
const sibling = openTab('a', true);
await sibling.handle(row.subId);
sibling.vanish();
jump(HANDLER_LIVE_MS + 1);
await dispatch();
expect(sent).toEqual([]);
expect(invoked).toHaveLength(1);
await expect(pending.depth(row.subId)).resolves.toBe(0);
});
it('keeps counting a live connection past that window while it renews', async () => {
vi.useFakeTimers({
toFake: ['Date', 'setInterval', 'clearInterval'],
});
workerOutcome = 'settled';
const row = await register();
const sibling = openTab('a', true);
await sibling.handle(row.subId);
await vi.advanceTimersByTimeAsync(HANDLER_LIVE_MS + HANDLER_RENEW_MS);
await dispatch();
expect(sibling.received).toHaveLength(1);
expect(invoked).toEqual([]);
});
it('drops a connection from the record when it disconnects', async () => {
workerOutcome = 'settled';
const row = await register();
const sibling = openTab('a', true);
await sibling.handle(row.subId);
await expect(recorded(row.subId)).resolves.toEqual([sibling.id]);
sibling.disconnect();
await handedOver();
await expect(recorded(row.subId)).resolves.toEqual([]);
await dispatch();
expect(sent).toEqual([]);
expect(invoked).toHaveLength(1);
});
it('drops a connection from the record when it stops running the handler', async () => {
workerOutcome = 'settled';
const row = await register();
const sibling = openTab('a', true);
await sibling.handle(row.subId);
sibling.handlers.clear();
await sibling.verb(EVENTS_HANDLE_VERB, { subIds: [] });
await expect(recorded(row.subId)).resolves.toEqual([]);
await dispatch();
expect(sent).toEqual([]);
expect(invoked).toHaveLength(1);
});
it('treats a connection that never announced, as on a node without the record, as having no handler', async () => {
workerOutcome = 'settled';
const row = await register();
const sibling = openTab('a', true);
await dispatch();
expect(sibling.received).toEqual([]);
expect(invoked).toHaveLength(1);
await expect(pending.depth(row.subId)).resolves.toBe(0);
});
it('records nothing for a row the connection does not hold', async () => {
const row = await register();
const stranger = new FakeTab(`x-${seq}`, {
user: { id: userId + 1, uuid: 'other', username: 'other' },
effectiveApp: { uid: `app-${seq}` },
} as unknown as Actor);
service.attachSocket(stranger, stranger.actor);
await stranger.handle(row.subId, 'made-up');
const written = (await redis.keys('*ev:phs:*')).filter((key: string) =>
key.includes(`:app-${seq}:`),
);
expect(written).toEqual([]);
});
it('goes to the worker when the record cannot be read', async () => {
workerOutcome = 'settled';
const row = await register();
const sibling = openTab('a', true);
await sibling.handle(row.subId);
vi.spyOn(presence, 'holdsHandler').mockRejectedValueOnce(
new Error('redis down'),
);
await dispatch();
expect(sibling.received).toEqual([]);
expect(invoked).toHaveLength(1);
await expect(pending.depth(row.subId)).resolves.toBe(0);
});
});
describe('a queued delivery declined for want of a handler', () => {
/** Regions with the app open, nearest first, as presence names them. */
const openIn = (...regions: string[]) => {
candidateRegionImpl = async (_holder, _app, tried) =>
regions.find((region) => !tried.includes(region)) ?? null;
};
const declineFrom = (
row: DurableSubscription,
entryId: string,
region: string,
) =>
service.declineRelayed(userId, row.subId, entryId, 'no_handler', region);
it('goes to the next region with the app open, and to the worker only once none is left', async () => {
workerOutcome = 'settled';
socketConnected = false;
openIn('east', 'south');
const row = await register();
await dispatch();
expect(handOffs.map((one) => one.region)).toEqual(['east']);
const entryId = String(handOffs[0].delivery.ackId);
await declineFrom(row, entryId, 'east');
expect(handOffs.map((one) => one.region)).toEqual(['east', 'south']);
expect(handOffs[1].delivery).toMatchObject({ toHandlers: true });
expect(invoked).toEqual([]);
await declineFrom(row, entryId, 'south');
expect(invoked).toHaveLength(1);
await expect(pending.depth(row.subId)).resolves.toBe(0);
expect(metered).toHaveLength(1);
});
it('settles once when the next region’s client takes it', async () => {
workerOutcome = 'settled';
socketConnected = false;
openIn('east', 'south');
const row = await register();
await dispatch();
const entryId = String(handOffs[0].delivery.ackId);
await declineFrom(row, entryId, 'east');
await service.settleRelayedAck(userId, row.subId, entryId);
await expect(pending.depth(row.subId)).resolves.toBe(0);
expect(invoked).toEqual([]);
expect(metered).toHaveLength(1);
// A late decline from either region settles nothing more.
await declineFrom(row, entryId, 'east');
await declineFrom(row, entryId, 'south');
expect(invoked).toEqual([]);
});
it('ignores a second decline from a region it already moved on from', async () => {
workerOutcome = 'settled';
socketConnected = false;
openIn('east', 'south');
const row = await register();
await dispatch();
const entryId = String(handOffs[0].delivery.ackId);
await declineFrom(row, entryId, 'east');
// East's other client, declining what east already gave back.
await declineFrom(row, entryId, 'east');
await service.declineDelivery(actorFor(), {
subId: row.subId,
id: entryId,
reason: 'no_handler',
});
expect(handOffs.map((one) => one.region)).toEqual(['east', 'south']);
expect(invoked).toEqual([]);
await expect(pending.depth(row.subId)).resolves.toBe(1);
});
it('goes to the worker once when the next region never answers', async () => {
workerOutcome = 'settled';
socketConnected = false;
openIn('east', 'south');
const row = await register();
await dispatch();
await declineFrom(row, String(handOffs[0].delivery.ackId), 'east');
jump(EVENTS_SOCKET_LEASE_MS + 1);
await service.sweepPending();
expect(handOffs).toHaveLength(2);
expect(invoked).toHaveLength(1);
await expect(pending.depth(row.subId)).resolves.toBe(0);
});
it('goes to another region after a tab here declined it', async () => {
workerOutcome = 'settled';
openIn('east');
const row = await register();
const tab = openTab('a');
await tab.handle(row.subId);
// Dropped its handler without telling the server yet.
tab.handlers.clear();
await dispatch();
await Promise.all(tab.answers);
await handedOver();
expect(tab.received).toHaveLength(1);
expect(handOffs.map((one) => one.region)).toEqual(['east']);
expect(invoked).toEqual([]);
});
it('still lets a row with no worker lapse rather than move on', async () => {
socketConnected = false;
openIn('east', 'south');
const row = await register({ handlerName: null });
await dispatch();
await declineFrom(row, String(handOffs[0].delivery.ackId), 'east');
expect(handOffs).toHaveLength(1);
expect(invoked).toEqual([]);
await expect(pending.depth(row.subId)).resolves.toBe(1);
});
});
@@ -29,6 +29,7 @@ import {
import { v4 as uuidv4 } from 'uuid';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import type { EmailNotifyValidateEvent } from '../../clients/event/types.js';
import { AppFeedbackService } from './AppFeedbackService.js';
// Drives the real wired service against the real stores and in-memory
@@ -544,7 +545,7 @@ describe('AppFeedbackService owner email', () => {
app_title: app.title,
message: 'hello dev',
}),
expect.objectContaining({ replyTo: sender.email }),
{ replyTo: sender.email, stream: 'notify' },
);
expect(Boolean((await feedbackRows(userId))[0].email_sent)).toBe(true);
});
@@ -598,9 +599,7 @@ describe('AppFeedbackService owner email', () => {
expect(send).toHaveBeenCalledTimes(1);
const [, , values, options] = send.mock.calls[0];
expect((values as Record<string, unknown>).sender_email).toBeNull();
expect(
(options as { replyTo?: string } | undefined)?.replyTo,
).toBeUndefined();
expect(options).toEqual({ stream: 'notify' });
// Delivery still happens; only the reply path is withheld.
expect(Boolean((await feedbackRows(userId))[0].email_sent)).toBe(true);
});
@@ -690,6 +689,93 @@ describe('AppFeedbackService owner email', () => {
).toBe(AppFeedbackService.PER_APP_DAILY_EMAIL_LIMIT);
});
describe('email.notify.validate', () => {
type Listener = (key: string, data: EmailNotifyValidateEvent) => void;
let listener: Listener | null = null;
const listen = (fn: Listener) => {
listener = fn;
server.clients.event.on('email.notify.validate', fn);
};
afterEach(() => {
if (listener) {
server.clients.event.off('email.notify.validate', listener);
}
listener = null;
});
it('skips only the owner a listener declines, without claiming a cap slot', async () => {
const send = mockEmailReady();
const markSent = vi.spyOn(
server.stores.appFeedback,
'markEmailSent',
);
const declinedId = await makeDeliverableOwner();
const declined = (await server.stores.user.getById(declinedId))!;
const acceptedId = await makeDeliverableOwner();
const accepted = (await server.stores.user.getById(acceptedId))!;
const seen: EmailNotifyValidateEvent[] = [];
listen((_key, data) => {
seen.push(structuredClone(data));
for (const recipient of data.recipients) {
if (recipient.to === declined.email) recipient.allow = false;
}
});
const declinedApp = await makeApp(declinedId, {
feedbackEnabled: true,
});
const acceptedApp = await makeApp(acceptedId, {
feedbackEnabled: true,
});
const userId = await makeUser();
await service.submit({
userId,
app: declinedApp.name,
message: 'a',
});
await service.submit({
userId,
app: acceptedApp.name,
message: 'b',
});
// One entry per ask: the app's owner.
expect(seen).toEqual(
[declined.email, accepted.email].map((to) => ({
kind: 'app-user-feedback',
recipients: [{ to, allow: true }],
})),
);
expect(send).toHaveBeenCalledTimes(1);
expect(send.mock.calls[0][0]).toBe(accepted.email);
expect(markSent).toHaveBeenCalledTimes(1);
const rows = await feedbackRows(userId);
expect(rows).toHaveLength(2);
expect(rows.map((row) => Boolean(row.email_sent))).toEqual([
false,
true,
]);
});
it('still emails the owner when a listener throws', async () => {
const send = mockEmailReady();
vi.spyOn(console, 'error').mockImplementation(() => {});
const ownerId = await makeDeliverableOwner();
const app = await makeApp(ownerId, { feedbackEnabled: true });
const userId = await makeUser();
const calls: string[] = [];
listen((_key, data) => {
calls.push(...data.recipients.map((r) => r.to));
throw new Error('listener failed');
});
await service.submit({ userId, app: app.name, message: 'hi' });
expect(calls).toHaveLength(1);
expect(send).toHaveBeenCalledTimes(1);
});
});
it('keeps the feedback and releases the email slot when the send fails', async () => {
mockEmailConfigured();
vi.spyOn(server.clients.email, 'send').mockRejectedValue(
@@ -18,7 +18,7 @@
*/
import { HttpError } from '../../core/http/HttpError.js';
import { isBlockedEmail } from '../../util/email.js';
import { isBlockedEmail, mayEmailNotify } from '../../util/email.js';
import { PuterService } from '../types.js';
/**
@@ -271,7 +271,7 @@ export class AppFeedbackService extends PuterService {
* precondition holds; otherwise silently skip (the row stays stored with
* `email_sent = 0`). Preconditions: transport configured, owner exists with
* a confirmed non-blocklisted email, owner not suspended and not
* unsubscribed, per-app daily email cap not reached.
* unsubscribed, not declined by a listener, per-app email cap not reached.
*/
async #emailOwner({
app,
@@ -302,6 +302,14 @@ export class AppFeedbackService extends PuterService {
}
if (isBlockedEmail(owner.email, this.config.blockedEmailDomains))
return;
if (
!(
await mayEmailNotify(this.clients.event, 'app-user-feedback', [
owner.email,
])
).has(owner.email)
)
return;
// Claim an email-cap slot *before* sending: flip email_sent, recount
// with the claim included, and release the slot if a concurrent
@@ -350,7 +358,10 @@ export class AppFeedbackService extends PuterService {
dev_center_link: `${this.config.origin}/app/dev-center`,
message,
},
senderEmail ? { replyTo: senderEmail } : {},
{
stream: 'notify',
...(senderEmail ? { replyTo: senderEmail } : {}),
},
);
} catch (e) {
// Release the claimed slot — the mail never went out.
File diff suppressed because it is too large. Load diff
+70 -11
View File
@@ -3732,7 +3732,9 @@ describe('FSService mkdir, touch, rename and shortcuts', () => {
);
expect(error.statusCode).toBe(status);
// The target was looked up, and that lookup changed nothing.
expect(lookups).toHaveBeenCalledWith(path);
expect(lookups).toHaveBeenCalledWith(path, {
foldedNames: true,
});
expect(
writes.mock.calls.filter(([sql]) =>
/fsentries/i.test(String(sql)),
@@ -4424,6 +4426,16 @@ describe('FSService home directory guard', () => {
});
});
// The column itself: a path lookup would heal a stale row on the way.
const storedPaths = async (...entries: FSEntry[]) => {
const rows = (await server.clients.db.read(
`SELECT uuid, path FROM fsentries WHERE uuid IN (${entries.map(() => '?').join(', ')})`,
entries.map((entry) => entry.uuid),
)) as Array<{ uuid: string; path: string }>;
const byUuid = new Map(rows.map((row) => [row.uuid, row.path]));
return entries.map((entry) => byUuid.get(entry.uuid) ?? null);
};
describe('FSService names starting with a supplementary character', () => {
let user: TestUser;
beforeAll(async () => {
@@ -4445,16 +4457,6 @@ describe('FSService names starting with a supplementary character', () => {
return { dir, inner, file };
};
// The column itself: a path lookup would heal a stale row on the way.
const storedPaths = async (...entries: FSEntry[]) => {
const rows = (await server.clients.db.read(
`SELECT uuid, path FROM fsentries WHERE uuid IN (${entries.map(() => '?').join(', ')})`,
entries.map((entry) => entry.uuid),
)) as Array<{ uuid: string; path: string }>;
const byUuid = new Map(rows.map((row) => [row.uuid, row.path]));
return entries.map((entry) => byUuid.get(entry.uuid) ?? null);
};
it('rewrites the subtree when the directory is renamed', async () => {
const { dir, inner, file } = await seed('🔥 rename');
@@ -4510,6 +4512,63 @@ describe('FSService names starting with a supplementary character', () => {
});
});
describe('FSService sibling folders a prefix pattern cannot tell apart', () => {
let user: TestUser;
beforeAll(async () => {
user = await makeUser();
});
// `Notes/a.txt` beside `notes/b.txt`: SQLite's LIKE folds ASCII case.
const seed = async (top: string) => {
const base = `${user.home}/Documents/${top}`;
const upper = await fs.mkdir(user.userId, {
path: `${base}/Notes`,
createMissingParents: true,
});
const lower = await fs.mkdir(user.userId, { path: `${base}/notes` });
expect(lower.uuid).not.toBe(upper.uuid);
const inUpper = await writeFile(user, `${base}/Notes/a.txt`, 'a');
const inLower = await writeFile(user, `${base}/notes/b.txt`, 'b');
return { base, upper, inUpper, inLower };
};
it('renames one and leaves the other subtree where it was', async () => {
const { base, upper, inUpper, inLower } = await seed('twin-rename');
await fs.rename(user.userId, upper, 'Archive');
expect(await storedPaths(inUpper, inLower)).toEqual([
`${base}/Archive/a.txt`,
`${base}/notes/b.txt`,
]);
const kept = await entryAt(user, '/Documents/twin-rename/notes/b.txt');
expect(kept?.uuid).toBe(inLower.uuid);
expect(await readBack(kept!)).toBe('b');
});
it('moves one, nested folders included, and leaves the other subtree where it was', async () => {
const { base, upper, inUpper, inLower } = await seed('twin-move');
await fs.mkdir(user.userId, { path: `${base}/Notes/deep` });
const nested = await writeFile(user, `${base}/Notes/deep/c.txt`, 'c');
const destination = (await entryAt(user, '/Desktop'))!;
await fs.move(user.userId, {
source: upper,
destinationParent: destination,
newName: 'twin-moved',
});
expect(await storedPaths(inUpper, nested, inLower)).toEqual([
`${user.home}/Desktop/twin-moved/a.txt`,
`${user.home}/Desktop/twin-moved/deep/c.txt`,
`${base}/notes/b.txt`,
]);
expect(
await entryAt(user, '/Documents/twin-move/Notes/a.txt'),
).toBeNull();
});
});
describe('FSService move', () => {
let user: TestUser;
beforeAll(async () => {
+130 -31
View File
@@ -73,6 +73,7 @@ import {
runWithConcurrencyLimitSettled,
} from '../../util/concurrency.js';
import { isUniqueViolation } from '../../util/dbError.js';
import { entryNamesMatch } from '../../util/entryNames.js';
import { AclMode } from '../acl/ACLService.js';
import type { puterServices } from '../index.js';
import {
@@ -159,6 +160,25 @@ const foreignAppDataOwner = (
const childPath = (parentPath: string, name: string): string =>
parentPath === '/' ? `/${name}` : `${parentPath}/${name}`;
/**
* The name is held by a differently named entry the database folds onto it (see
* `entryNamesMatch`). That entry is never replaced or returned instead.
*/
const foldedNameConflict = (name: string): HttpError =>
new HttpError(
409,
`An entry whose name conflicts with \`${name}\` already exists`,
{ legacyCode: 'conflict' },
);
/**
* `occupant`, found at `path`, is a different name the database folded. The
* store resolves a stored path spelled otherwise through the parent chain.
*/
const holdsFoldedName = (occupant: FSEntry, path: string): boolean =>
!entryNamesMatch(occupant.name, pathPosix.basename(path)) ||
!entryNamesMatch(occupant.path, path);
/** `/<username>/AppData/<appUid>`: the directory an app's launch provisions. */
const isAppDataRootPath = (path: string): boolean => {
const segments = path.split('/');
@@ -805,6 +825,7 @@ export class FSService extends PuterService {
// granted access to, even when those live outside the
// writer's own namespace.
crossNamespace: true,
foldedNames: true,
},
);
for (let index = 0; index < initialPaths.length; index++) {
@@ -829,6 +850,7 @@ export class FSService extends PuterService {
const readPromise = this.stores.fsEntry.getEntryByPath(path, {
useTryHardRead: true,
skipCache: true,
foldedNames: true,
});
existingEntryCache.set(path, readPromise);
return await readPromise;
@@ -838,6 +860,17 @@ export class FSService extends PuterService {
for (const input of inputs) {
let normalizedInput = input.normalizedInput;
let existingEntry = await loadExistingEntry(normalizedInput.path);
if (
existingEntry &&
holdsFoldedName(existingEntry, normalizedInput.path)
) {
if (normalizedInput.overwrite && !normalizedInput.dedupeName) {
throw foldedNameConflict(
pathPosix.basename(normalizedInput.path),
);
}
normalizedInput = { ...normalizedInput, overwrite: false };
}
const pathReservedInBatch = reservedPaths.has(normalizedInput.path);
let pendingDirectory =
!existingEntry &&
@@ -885,16 +918,19 @@ export class FSService extends PuterService {
if (existingEntry && !normalizedInput.overwrite) {
// v1 wire contract: clients (the GUI's save dialogs among
// them) key on `item_with_same_name_exists` + `entry_name`
// to offer an overwrite prompt.
// to offer an overwrite prompt, naming the entry they show.
throw new HttpError(
409,
'A file already exists at this path and overwrite was not requested',
{
legacyCode: 'item_with_same_name_exists',
fields: {
entry_name: pathPosix.basename(
entry_name: holdsFoldedName(
existingEntry,
normalizedInput.path,
),
)
? existingEntry.name
: pathPosix.basename(normalizedInput.path),
},
},
);
@@ -2991,9 +3027,10 @@ export class FSService extends PuterService {
if (!liveKey || liveKey === session.objectKey) return;
// The completion was allowed against a path, so that is where the
// entry has to still be; a move since would put the bytes elsewhere.
// The upload may spell the entry's name in another case or form.
const target =
await this.stores.fsEntry.getEntryByUuidFromPrimary(liveKey);
if (!target || target.path !== session.targetPath) {
if (!target || !entryNamesMatch(target.path, session.targetPath)) {
throw new HttpError(409, 'Upload target has moved', {
legacyCode: 'conflict',
});
@@ -4062,7 +4099,9 @@ export class FSService extends PuterService {
parentPath === '/'
? `/${candidate}`
: `${parentPath}/${candidate}`;
const existing = await repo.getEntryByPath(candidatePath);
const existing = await repo.getEntryByPath(candidatePath, {
foldedNames: true,
});
if (!existing) return candidate;
}
throw new HttpError(
@@ -4129,13 +4168,21 @@ export class FSService extends PuterService {
// reads as missing; a concurrent create lands in the unique-violation
// handler below.
const existingLookup = awaitLater(
this.stores.fsEntry.getEntryByPath(targetPath),
this.stores.fsEntry.getEntryByPath(targetPath, {
foldedNames: true,
}),
);
const parent = await parentLookup;
let name = pathPosix.basename(targetPath);
const existing = await existingLookup;
if (existing) {
if (
existing &&
holdsFoldedName(existing, childPath(parent.path, name))
) {
if (!input.dedupeName) throw foldedNameConflict(name);
name = await this.#findDedupedName(parent, name);
} else if (existing) {
if (existing.isDir) {
if (input.dedupeName) {
name = await this.#findDedupedName(parent, name);
@@ -4182,8 +4229,11 @@ export class FSService extends PuterService {
// would surface the raw ER_DUP_ENTRY as a 500. Read primary too.
const raced = await this.stores.fsEntry.getEntryByPath(
insertedPath,
{ useTryHardRead: true },
{ useTryHardRead: true, foldedNames: true },
);
if (raced && holdsFoldedName(raced, insertedPath)) {
throw foldedNameConflict(name);
}
if (raced?.isDir) return raced;
if (raced) {
throw new HttpError(
@@ -4219,7 +4269,15 @@ export class FSService extends PuterService {
!!input.createMissingParents,
);
const name = pathPosix.basename(targetPath);
const existing = await this.stores.fsEntry.getEntryByPath(targetPath);
const existing = await this.stores.fsEntry.getEntryByPath(targetPath, {
foldedNames: true,
});
if (
existing &&
holdsFoldedName(existing, childPath(parent.path, name))
) {
throw foldedNameConflict(name);
}
if (existing) {
return this.stores.fsEntry.touchEntryTimestamps(existing.uuid, {
setAccessed: input.setAccessed,
@@ -4258,25 +4316,39 @@ export class FSService extends PuterService {
parentPath === '/' ? `/${newName}` : `${parentPath}/${newName}`;
// Reject if another entry already owns the target path.
const collision = await this.stores.fsEntry.getEntryByPath(newPath);
const collision = await this.stores.fsEntry.getEntryByPath(newPath, {
foldedNames: true,
});
const taken = (name: string) =>
new HttpError(409, `An entry named \`${name}\` already exists`, {
legacyCode: 'conflict',
});
if (collision && collision.uuid !== entry.uuid) {
throw new HttpError(
409,
`An entry named \`${newName}\` already exists`,
{ legacyCode: 'conflict' },
throw taken(
holdsFoldedName(collision, newPath) ? collision.name : newName,
);
}
const updated = await this.stores.fsEntry.updateEntry(entry.uuid, {
name: newName,
path: newPath,
});
let updated: FSEntry;
try {
updated = await this.stores.fsEntry.updateEntry(entry.uuid, {
name: newName,
path: newPath,
});
} catch (err) {
// Taken after the probe, or by a sibling it can't see through
// another spelling of the folder's path.
if (isUniqueViolation(err)) throw taken(newName);
throw err;
}
if (entry.isDir) {
await this.stores.fsEntry.updatePathPrefixForUser(
entry.userId,
entry.path,
newPath,
undefined,
entry.uuid,
);
}
this.#emitFsEvent(
@@ -4312,7 +4384,9 @@ export class FSService extends PuterService {
input.parent.path === '/'
? `/${name}`
: `${input.parent.path}/${name}`;
const collision = await this.stores.fsEntry.getEntryByPath(childPath);
const collision = await this.stores.fsEntry.getEntryByPath(childPath, {
foldedNames: true,
});
if (collision) {
if (input.dedupeName) {
name = await this.#findDedupedName(input.parent, name);
@@ -5162,7 +5236,9 @@ export class FSService extends PuterService {
input.newMetadata,
);
const collision = await this.stores.fsEntry.getEntryByPath(targetPath);
const collision = await this.stores.fsEntry.getEntryByPath(targetPath, {
foldedNames: true,
});
if (collision && collision.uuid !== source.uuid) {
name = await this.#resolveMoveCollision(
userId,
@@ -5214,7 +5290,7 @@ export class FSService extends PuterService {
if (!isUniqueViolation(err)) throw err;
const raced = await this.stores.fsEntry.getEntryByPath(
pathIn(name),
{ useTryHardRead: true },
{ useTryHardRead: true, foldedNames: true },
);
if (raced && raced.uuid !== source.uuid) {
name = await this.#resolveMoveCollision(
@@ -5237,6 +5313,7 @@ export class FSService extends PuterService {
source.path,
finalPath,
newOwnerId,
source.uuid,
);
}
@@ -5293,7 +5370,11 @@ export class FSService extends PuterService {
if (opts.intoRestrictedTrash) {
return this.#findDedupedName(destinationParent, name);
}
if (input.overwrite) {
const folded = holdsFoldedName(
occupant,
childPath(destinationParent.path, name),
);
if (input.overwrite && !folded) {
this.#assertNotReplacingAncestor(source, occupant);
await this.remove(userId, { entry: occupant, recursive: true });
return name;
@@ -5301,13 +5382,19 @@ export class FSService extends PuterService {
if (input.dedupeName) {
return this.#findDedupedName(destinationParent, name);
}
if (folded && input.overwrite) throw foldedNameConflict(name);
// v1 wire contract: clients (the GUI's move/paste flows among them)
// key on `item_with_same_name_exists` + `entry_name` to offer a
// replace/skip prompt.
throw new HttpError(409, `An entry named \`${name}\` already exists`, {
legacyCode: 'item_with_same_name_exists',
fields: { entry_name: name },
});
const shownName = folded ? occupant.name : name;
throw new HttpError(
409,
`An entry named \`${shownName}\` already exists`,
{
legacyCode: 'item_with_same_name_exists',
fields: { entry_name: shownName },
},
);
}
/**
@@ -5369,9 +5456,16 @@ export class FSService extends PuterService {
? `/${name}`
: `${destinationParent.path}/${name}`;
const collision = await this.stores.fsEntry.getEntryByPath(targetPath);
const collision = await this.stores.fsEntry.getEntryByPath(targetPath, {
foldedNames: true,
});
// Overwriting the source would delete it before it is read.
const replacesSource = collision?.uuid === source.uuid;
const replacesCollision =
!!collision &&
!!input.overwrite &&
!replacesSource &&
!holdsFoldedName(collision, targetPath);
// A copy duplicates the bytes for real, so it costs the same against
// the allowance as writing them. Check before the overwrite below
@@ -5387,14 +5481,14 @@ export class FSService extends PuterService {
await this.#assertStorageAllowance(
allowance.ownerId,
await this.#entryStorageSize(source),
collision && input.overwrite && !replacesSource
collision && replacesCollision
? await this.#entryStorageSize(collision)
: 0,
allowance.max,
);
if (collision) {
if (input.overwrite && !replacesSource) {
if (replacesCollision) {
this.#assertNotReplacingAncestor(source, collision);
await this.remove(userId, {
entry: collision,
@@ -5402,18 +5496,23 @@ export class FSService extends PuterService {
});
} else if (input.dedupeName) {
name = await this.#findDedupedName(destinationParent, name);
} else if (input.overwrite && !replacesSource) {
throw foldedNameConflict(name);
} else if (input.overwrite) {
throw new HttpError(400, 'Cannot copy an item over itself', {
legacyCode: 'bad_request',
});
} else {
// v1 wire contract, as in move() above.
const shownName = holdsFoldedName(collision, targetPath)
? collision.name
: name;
throw new HttpError(
409,
`An entry named \`${name}\` already exists`,
`An entry named \`${shownName}\` already exists`,
{
legacyCode: 'item_with_same_name_exists',
fields: { entry_name: name },
fields: { entry_name: shownName },
},
);
}
@@ -399,8 +399,11 @@ export class PuterHomepageService extends PuterService {
// instead of the socket wire.
eventsNotifications: notificationsFoldInEnabled(this.config),
...meta,
// After `meta`, so nothing page-specific can name the opener.
// After `meta`, so nothing page-specific can name the opener or
// the pages a sign-in popup hands a session.
...(popupFrame ? { embed_host_origin: popupFrame.hostOrigin } : {}),
first_party_session_origins:
this.config.first_party_session_origins ?? [],
};
// Only ever this request's; without one the GUI asks for its token as
// it always has.
+13 -5
View File
@@ -82,6 +82,18 @@ export const creditEnforcementExempt = (
return false;
};
/** Whether the credit gate lets this actor through. */
export const actorHasCredits = async (
metering: CreditMetering | undefined,
actor: Actor | undefined,
config: EnforcementConfig,
): Promise<boolean> => {
if (!metering) return true;
if (!enforcementEnabled(config)) return true;
if (creditEnforcementExempt(actor, config)) return true;
return metering.hasAnyUsageCached(actor!);
};
/**
* Reject an actor with nothing left to spend. Same status and code the AI
* surfaces use, so a client that already handles one handles this.
@@ -91,11 +103,7 @@ export const assertActorHasCredits = async (
actor: Actor | undefined,
config: EnforcementConfig,
): Promise<void> => {
if (!metering) return;
if (!enforcementEnabled(config)) return;
if (creditEnforcementExempt(actor, config)) return;
if (!(await metering.hasAnyUsageCached(actor!))) {
if (!(await actorHasCredits(metering, actor, config))) {
throw insufficientCreditsError();
}
};
@@ -520,7 +520,7 @@ describe("NotificationService — 'notif.written'", () => {
const user = await makeUser();
const uid = uuidv4();
const appUid = `app-${uuidv4()}`;
await server.stores.notification.insert({
await server.stores.notification.create({
userId: user.id,
value: { title: 'one', type: 'share.received' },
uid,
@@ -40,6 +40,7 @@ import {
resolveNotificationWrite,
} from './notificationTypes.js';
import type { NotificationTypeName } from './notificationTypes.js';
import type { NotificationRow } from '../../stores/notification/NotificationStore.js';
export {
NOTIFICATION_TYPES,
@@ -52,6 +53,15 @@ export type {
} from './notificationTypes.js';
export { canViewNotification } from './notificationAudience.js';
export type { NotificationRow };
type NotifyOptions = {
type: NotificationTypeName;
appUid?: string | null;
silent?: boolean;
group?: string | null;
};
const RETENTION_SWEEP_JOB = 'notifications.retention';
const RETENTION_SWEEP_EVERY_MS = 60 * 60 * 1000;
const RETENTION_SWEEP_LEASE_MS = 15 * 60 * 1000;
@@ -152,13 +162,18 @@ export class NotificationService extends PuterService {
async notify(
userIds: number[],
notification: Record<string, unknown>,
opts: {
type: NotificationTypeName;
appUid?: string | null;
silent?: boolean;
group?: string | null;
},
opts: NotifyOptions,
): Promise<string | null> {
const [first] = await this.notifyRows(userIds, notification, opts);
return first?.uid ?? null;
}
/** `notify`, returning each row written; failed recipients are left out. */
async notifyRows(
userIds: number[],
notification: Record<string, unknown>,
opts: NotifyOptions,
): Promise<NotificationRow[]> {
const appUid = opts.appUid ?? null;
const registered = resolveNotificationWrite(opts.type, appUid);
const payload = { ...notification, type: registered.type };
@@ -167,8 +182,9 @@ export class NotificationService extends PuterService {
const written = await Promise.all(
userIds.map(async (userId) => {
const uid = uuidv4();
let row: NotificationRow;
try {
await this.stores.notification.insert({
row = await this.stores.notification.create({
userId,
value: payload,
uid,
@@ -200,11 +216,11 @@ export class NotificationService extends PuterService {
{ sendId: uuidv4() },
);
}
return uid;
return row;
}),
);
return written.find((uid) => uid !== null) ?? null;
return written.filter((row) => row !== null);
}
/**
@@ -230,28 +246,70 @@ export class NotificationService extends PuterService {
replace?: boolean;
},
): Promise<boolean> {
const replacedAt = opts.replace ? Date.now() : undefined;
const payload = await this.#rewrite(uid, userId, notification, {
type: opts.type,
silent: opts.silent,
replacedAt,
});
return payload !== null;
}
/** Replaces a row already read; null if it was shown or dismissed since. */
async replaceUnseen(
row: NotificationRow,
userId: number,
notification: Record<string, unknown>,
opts: { type: NotificationTypeName },
): Promise<NotificationRow | null> {
const replacedAt = Date.now();
const payload = await this.#rewrite(row.uid, userId, notification, {
type: opts.type,
replacedAt,
appUid: row.app_uid,
});
if (!payload) return null;
return this.stores.notification.replacedRow(row, payload, replacedAt);
}
/** The payload written, or null; `appUid` saves reading the row for it. */
async #rewrite(
uid: string,
userId: number,
notification: Record<string, unknown>,
opts: {
type: NotificationTypeName;
silent?: boolean;
replacedAt?: number;
appUid?: string | null;
},
): Promise<Record<string, unknown> | null> {
const registered = findNotificationType(opts.type);
if (!registered?.groupable) {
throw new Error(`notification type is not groupable: ${opts.type}`);
}
const payload = { ...notification, type: registered.type };
const replacedAt = opts.replace ? Date.now() : undefined;
const { replacedAt } = opts;
const updated = await this.stores.notification.updateValue(
uid,
userId,
payload,
replacedAt === undefined ? {} : { onlyUnseen: true, replacedAt },
);
if (!updated) return false;
if (opts.silent) return true;
if (!updated) return null;
if (opts.silent) return payload;
// Before the read, so the desktop copy updates even if it fails.
// Before any read, so the desktop copy updates even if that fails.
const desktop = inAccountMailbox(registered.type);
if (desktop) this.#sendRewrite(userId, uid, payload);
const row = await this.stores.notification.getByUid(uid, { userId });
const rowAppUid =
opts.appUid !== undefined
? opts.appUid
: (await this.stores.notification.getByUid(uid, { userId }))
?.app_uid;
const app = notification.app as { uid?: unknown } | undefined;
const appUid =
row?.app_uid ?? (typeof app?.uid === 'string' ? app.uid : null);
rowAppUid ?? (typeof app?.uid === 'string' ? app.uid : null);
await this.emitWritten(
{
userId,
@@ -265,7 +323,7 @@ export class NotificationService extends PuterService {
},
{ sendId: uuidv4(), rewriteSent: desktop },
);
return true;
return payload;
}
/**
@@ -2017,6 +2017,42 @@ describe('PermissionService — scan paths', () => {
).toBe(true);
});
it('scans the user for an app only when an option is implied for it', async () => {
const { row, actor } = await makeGroupedUser();
const app = await makeApp(row.id);
const appActor: Actor = makeActor({
user: actor.user,
app: { uid: app.uid, id: app.id },
});
const scan = vi.spyOn(permService, 'scan');
const userScans = () =>
scan.mock.calls.filter(
([scanned]) => !scanned.app && !scanned.accessToken,
).length;
const scanAsApp = (permission: string) =>
permService.scan(appActor, permission, undefined, {
noCache: true,
});
try {
expect(
await scanAsApp(`zztest:none-${uuidv4()}:ii:read`),
).not.toContainEqual(
expect.objectContaining({ source: 'user-app-implied' }),
);
expect(userScans()).toBe(0);
expect(await scanAsApp('driver:puter-kvstore')).toContainEqual(
expect.objectContaining({
source: 'user-app-implied',
has_terminal: true,
}),
);
expect(userScans()).toBeGreaterThan(0);
} finally {
scan.mockRestore();
}
});
it('resolves a user-to-app grant, and stops once revoked', async () => {
const { row, actor } = await makeGroupedUser();
const app = await makeApp(row.id);
@@ -2214,7 +2250,10 @@ describe('PermissionService — scan paths', () => {
[tokenUid, permission, '{}'],
);
const get = vi.spyOn(server.clients.redis, 'get');
const list = vi.spyOn(
server.stores.permission,
'listAccessTokenPerms',
);
try {
expect(
await permService.check(
@@ -2228,12 +2267,10 @@ describe('PermissionService — scan paths', () => {
),
).toBe(true);
expect(
get.mock.calls.filter(
([key]) => key === `perms:token:${tokenUid}`,
),
list.mock.calls.filter(([uid]) => uid === tokenUid),
).toHaveLength(1);
} finally {
get.mockRestore();
list.mockRestore();
}
});
@@ -735,26 +735,16 @@ export class PermissionService extends PuterService {
// `#scanAccessToken`, bounded by its own rows. Reading the chain here
// would hand it the issuing app's grants unbounded.
if (!actor.app) return;
const issuerActor = userRelatedActor(actor);
const issuerReading = await this.scan(issuerActor, options);
const hasTerminal = readingHasTerminal(issuerReading);
const appUid = actor.app.uid;
const found: Array<{ permission: string; by: string; data: unknown }> =
[];
for (const permission of options) {
const implied = (
default_implicit_user_app_permissions as Record<string, unknown>
)[permission];
if (implied) {
reading.push({
$: 'path',
permission,
has_terminal: hasTerminal,
source: 'user-app-implied',
by: 'user-app-hc-1',
data: implied,
issuer_username: actor.user?.username,
reading: issuerReading,
});
found.push({ permission, by: 'user-app-hc-1', data: implied });
}
// per-app hardcoded overrides
@@ -768,18 +758,30 @@ export class PermissionService extends PuterService {
}
}
if (hits[permission]) {
reading.push({
$: 'path',
found.push({
permission,
has_terminal: hasTerminal,
source: 'user-app-implied',
by: 'user-app-hc-2',
data: hits[permission],
issuer_username: actor.user?.username,
reading: issuerReading,
});
}
}
// The user is scanned only when an option is implied for the app.
if (found.length === 0) return;
const issuerReading = await this.scan(userRelatedActor(actor), options);
const hasTerminal = readingHasTerminal(issuerReading);
for (const { permission, by, data } of found) {
reading.push({
$: 'path',
permission,
has_terminal: hasTerminal,
source: 'user-app-implied',
by,
data,
issuer_username: actor.user?.username,
reading: issuerReading,
});
}
}
async #scanUserApp(
@@ -46,7 +46,12 @@ import {
sharedViewLink,
} from './shareDeepLink';
import { NOTIFY_FANOUT_CAP } from '../../core/teams.js';
import { abuseKey, cleanEmail, isBlockedEmail } from '../../util/email.js';
import {
abuseKey,
cleanEmail,
isBlockedEmail,
mayEmailNotify,
} from '../../util/email.js';
import { blocksAllShares } from './shareBlocks.js';
import type { ResolvedShare } from './ShareService';
import { withRedisLock } from '../../util/redisLock.js';
@@ -1034,6 +1039,18 @@ export class ShareNotificationService extends PuterService {
? (matches.values().next().value ?? null)
: null;
const [{ record: first }] = claimed;
const template =
first.kind === 'holder'
? 'file_shared_with_you'
: 'file_shared_invite';
// Asked at send: an entry may have waited out a window or a retry.
const asked = await mayEmailNotify(this.clients.event, template, [
first.to,
]);
if (!asked.has(first.to)) {
skipped('a listener declined the address', { key });
return;
}
console.log('[share-notify] sending digest:', {
key,
kind: first.kind,
@@ -1065,6 +1082,7 @@ export class ShareNotificationService extends PuterService {
origin: this.#appLink(),
unsubscribe_uuid: first.recipientUuid ?? null,
},
{ stream: 'notify' },
);
} else {
await this.clients.email.send(
@@ -1078,6 +1096,7 @@ export class ShareNotificationService extends PuterService {
shares: digestLines(entries),
link: this.#appLink(),
},
{ stream: 'notify' },
);
}
console.log('[share-notify] digest sent:', {
@@ -1142,12 +1161,26 @@ export class ShareNotificationService extends PuterService {
if (byEmail.size === 0) return;
const issuer = actor.user?.username;
for (const [to, { count, items }] of byEmail) {
const unblocked = [...byEmail.keys()].filter((to) => {
const blocked = isBlockedEmail(to, this.config.blockedEmailDomains);
if (blocked) {
skipped('invite address is on a blocked domain', { to });
}
return !blocked;
});
// One ask for every address, before any of them claims budget.
const allowed = await mayEmailNotify(
this.clients.event,
'file_shared_invite',
unblocked,
);
for (const to of unblocked) {
const { count, items } = byEmail.get(to)!;
// Each address fails alone — one refused send must not cost the
// next invitee their only channel.
try {
if (isBlockedEmail(to, this.config.blockedEmailDomains)) {
skipped('invite address is on a blocked domain', { to });
if (!allowed.has(to)) {
skipped('a listener declined the invite address', { to });
continue;
}
const mayOpen = await this.#claimInviteEmail(issuerId, to);
+389 -31
View File
@@ -32,6 +32,7 @@ import { makeActor, type Actor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { PuterServer } from '../../server.js';
import { createTestUser, setupTestServer } from '../../testUtil.js';
import type { FSEntry } from '../../stores/fs/FSEntry';
import { MANAGE_PERM_PREFIX } from '../permission/consts.js';
describe('ShareService', () => {
@@ -1445,37 +1446,6 @@ describe('ShareService', () => {
server.services.share.listSharedByMe(actor, opts),
);
it('still finds a row written before the owner was recorded', async () => {
const owner = await makeUser();
const delegate = await makeUser();
const third = await makeUser();
const file = await makeFile(owner.user);
await share(owner.actor, {
uid: file.uuid,
recipient: { email: delegate.email },
mode: 'manage',
});
await share(delegate.actor, {
uid: file.uuid,
recipient: { email: third.email },
mode: 'read',
});
// What a writer that predates the column leaves behind.
await server.clients.db.write(
'UPDATE `share` SET `entry_owner_user_id` = NULL',
);
const listed = await listSharedByMe(owner.actor, {
includeTotal: true,
});
expect(listed.items.map((r) => r.holder?.username)).toContain(
third.user.username,
);
expect(listed.total).toBeGreaterThanOrEqual(2);
});
it("includes a delegate's share on an item the caller owns", async () => {
const owner = await makeUser();
const delegate = await makeUser();
@@ -4281,6 +4251,39 @@ describe('ShareService', () => {
expect(audiences.flat()).toContain(recipient.user.id);
});
it("tells the former owner's recipients when the new owner shares nothing", async () => {
const from = await makeUser();
const to = await makeUser();
const recipient = await makeUser();
const { dir, file } = await makeDirWithFile(from.user);
const after = `/${to.user.username}/${file.name}`;
await share(from.actor, {
uid: dir.uuid,
recipient: { email: recipient.email },
mode: 'read',
});
const audiences = await captureAudiences(
'outer.gui.item.removed',
file.uuid,
async () => {
await server.clients.event.emitAndWait(
'fs.move.node',
{
node: { ...file, path: after, userId: to.user.id },
fromPath: file.path,
toPath: after,
fromUserId: from.user.id,
},
{},
);
},
);
expect(audiences.flat()).toContain(recipient.user.id);
});
it('still reports a move within the shared folder as a move', async () => {
const owner = await makeUser();
const recipient = await makeUser();
@@ -6156,6 +6159,361 @@ describe('ShareService', () => {
expect(reachingCalls).toBe(0);
});
it('does not read ancestors for an owner with nothing shared', async () => {
const owner = await makeUser();
const { file } = await makeDirWithFile(owner.user);
const ancestors = vi.spyOn(server.stores.fsEntry, 'getAncestors');
let calls = -1;
try {
await server.clients.event.emitAndWait(
'fs.write.file',
{ node: file },
{},
);
await settle();
calls = ancestors.mock.calls.length;
} finally {
ancestors.mockRestore();
}
expect(calls).toBe(0);
});
describe('an entry stamped with another owner in a shared folder', () => {
/** A file under `dir` whose row names `stamped` as its owner. */
const makeStampedFile = async (
dir: { id: number; uuid: string; path: string },
stamped: { id: number },
) => {
const uuid = uuidv4();
const name = `stamped-${uuid.slice(0, 8)}.txt`;
const path = `${dir.path}/${name}`;
await server.clients.db.write(
'INSERT INTO `fsentries` (`uuid`, `name`, `path`, `user_id`, `is_dir`, `modified`, `parent_id`, `parent_uid`) VALUES (?, ?, ?, ?, ?, ?, ?, ?)',
[
uuid,
name,
path,
stamped.id,
server.clients.db.booleanValue(false),
Math.floor(Date.now() / 1000),
dir.id,
dir.uuid,
],
);
const entry = await server.stores.fsEntry.getEntryByPath(path);
if (!entry) throw new Error('fsentry not created');
return entry;
};
const setup = async () => {
const owner = await makeUser();
const stamped = await makeUser();
const recipient = await makeUser();
const { dir } = await makeDirWithFile(owner.user);
await share(owner.actor, {
uid: dir.uuid,
recipient: { email: recipient.email },
mode: 'read',
});
const file = await makeStampedFile(dir, stamped.user);
return { recipient, file };
};
const heard = async (
event:
| 'outer.gui.item.updated'
| 'outer.gui.item.renamed'
| 'outer.gui.item.removed',
uuid: string,
userId: number,
fn: () => Promise<unknown>,
) => {
let seen = false;
const listener = (_key: string, data: unknown) => {
const payload = data as {
user_id_list?: number[];
response?: { uuid?: string };
};
if (
payload.response?.uuid === uuid &&
payload.user_id_list?.includes(userId)
)
seen = true;
};
server.clients.event.on(event, listener);
try {
await fn();
for (let i = 0; i < 20 && !seen; i++) await settle();
} finally {
server.clients.event.off(event, listener);
}
return seen;
};
it('skips the share lookup when no owner on the chain shares', async () => {
const owner = await makeUser();
const stamped = await makeUser();
const { dir } = await makeDirWithFile(owner.user);
const file = await makeStampedFile(dir, stamped.user);
const reaching = vi.spyOn(server.stores.share, 'listReaching');
let calls = -1;
try {
await server.clients.event.emitAndWait(
'fs.write.file',
{ node: file },
{},
);
await settle();
calls = reaching.mock.calls.length;
} finally {
reaching.mockRestore();
}
expect(calls).toBe(0);
});
it('still tells the folder recipient about a write', async () => {
const { recipient, file } = await setup();
expect(
await heard(
'outer.gui.item.updated',
file.uuid,
recipient.user.id,
() =>
server.clients.event.emitAndWait(
'fs.write.file',
{ node: file },
{},
),
),
).toBe(true);
});
it('still tells the folder recipient about a rename', async () => {
const { recipient, file } = await setup();
const renamed = `${file.path}.renamed`;
expect(
await heard(
'outer.gui.item.renamed',
file.uuid,
recipient.user.id,
() =>
server.clients.event.emitAndWait(
'fs.rename',
{
node: {
...file,
path: renamed,
name: `${file.name}.renamed`,
},
old_path: file.path,
},
{},
),
),
).toBe(true);
});
it('still tells the folder recipient about a delete', async () => {
const { recipient, file } = await setup();
expect(
await heard(
'outer.gui.item.removed',
file.uuid,
recipient.user.id,
() =>
server.clients.event.emitAndWait(
'fs.remove.node',
{ node: file, entry: file, target: file },
{},
),
),
).toBe(true);
});
});
it("looks from a hand-over's destination, new owner's cache warm", async () => {
const from = await makeUser();
const to = await makeUser();
const recipient = await makeUser();
const file = await makeFile(from.user);
await share(from.actor, {
uid: file.uuid,
recipient: { email: recipient.email },
mode: 'read',
});
// Caches "nothing shared" for the new owner.
await server.services.fs.touch(to.user.id, {
path: `/${to.user.username}/warm.txt`,
});
await settle();
// What FSService.move writes before it emits the event.
const toHome = await server.stores.fsEntry.getRootEntryForUser(
to.user.id,
);
const after = `/${to.user.username}/${file.name}`;
await server.stores.fsEntry.updateEntry(file.uuid, {
path: after,
userId: to.user.id,
parentId: toHome!.id,
parentUid: toHome!.uuid,
});
await server.stores.share.reassignEntryOwnerUnder(
to.user.id,
after,
);
const moved = await server.stores.fsEntry.getEntryByPath(after);
const ancestors = vi.spyOn(server.stores.fsEntry, 'getAncestors');
let calls = -1;
try {
await server.clients.event.emitAndWait(
'fs.move.node',
{
node: moved!,
fromPath: file.path,
toPath: after,
fromUserId: from.user.id,
},
{},
);
await settle();
calls = ancestors.mock.calls.filter(
([path]) => path === `/${to.user.username}`,
).length;
} finally {
ancestors.mockRestore();
}
// The share's revoke runs alongside, so no event is promised.
expect(calls).toBeGreaterThan(0);
});
describe("when the path may not be the owner's own home", () => {
/** Whether a write of `node` read the ancestors above it. */
const readsAncestors = async (node: FSEntry) => {
const ancestors = vi.spyOn(
server.stores.fsEntry,
'getAncestors',
);
const parent = node.path.slice(0, node.path.lastIndexOf('/'));
let calls = -1;
try {
await server.clients.event.emitAndWait(
'fs.write.file',
{ node },
{},
);
await settle();
calls = ancestors.mock.calls.filter(
([path]) => path === parent,
).length;
} finally {
ancestors.mockRestore();
}
return calls > 0;
};
it('reads ancestors for an owner with no account', async () => {
const owner = await makeUser();
const file = await makeFile(owner.user);
expect(
await readsAncestors({ ...file, userId: 2_000_000_000 }),
).toBe(true);
});
it('reads ancestors when the home differs only in case', async () => {
const owner = await makeUser();
const file = await makeFile(owner.user);
const upper = `/${owner.user.username.toUpperCase()}/${file.name}`;
expect(await readsAncestors({ ...file, path: upper })).toBe(
true,
);
});
it('reads ancestors under the new name after a rename', async () => {
const owner = await makeUser();
const file = await makeFile(owner.user);
// Caches the old name.
expect(await readsAncestors(file)).toBe(false);
const renamed = `${owner.user.username}r`;
await server.stores.user.update(owner.user.id, {
username: renamed,
});
const moved = { ...file, path: `/${renamed}/${file.name}` };
expect(await readsAncestors(moved)).toBe(true);
});
it('looks an unknown owner up once, misses included', async () => {
const owner = await makeUser();
const file = await makeFile(owner.user);
const ghost = 2_000_000_001;
const lookups = vi.spyOn(server.stores.user, 'getById');
let calls = -1;
try {
for (let round = 0; round < 2; round++) {
await Promise.all(
Array.from({ length: 3 }, () =>
server.clients.event.emitAndWait(
'fs.write.file',
{ node: { ...file, userId: ghost } },
{},
),
),
);
await settle();
}
calls = lookups.mock.calls.filter(
([id]) => id === ghost,
).length;
} finally {
lookups.mockRestore();
}
expect(calls).toBe(1);
});
});
it('still reaches a nested entry under a shared folder', async () => {
const owner = await makeUser();
const recipient = await makeUser();
const { dir } = await makeDirWithFile(owner.user);
await share(owner.actor, {
uid: dir.uuid,
recipient: { email: recipient.email },
mode: 'read',
});
const nested = await server.services.fs.touch(owner.user.id, {
path: `${dir.path}/a/b/deep.txt`,
createMissingParents: true,
});
await settle();
const seen: unknown[] = [];
const listener = (_key: string, data: unknown) => {
const payload = data as {
user_id_list?: number[];
response?: { uuid?: string };
};
if (
payload.response?.uuid === nested.uuid &&
payload.user_id_list?.includes(recipient.user.id)
)
seen.push(data);
};
server.clients.event.on('outer.gui.item.updated', listener);
try {
await server.clients.event.emitAndWait(
'fs.write.file',
{ node: nested },
{},
);
for (let i = 0; i < 50 && seen.length === 0; i++)
await settle();
} finally {
server.clients.event.off('outer.gui.item.updated', listener);
}
expect(seen).toHaveLength(1);
});
it('fans out on the next write once something is shared', async () => {
const owner = await makeUser();
const recipient = await makeUser();
+48 -5
View File
@@ -32,6 +32,7 @@ import {
runWithConcurrencyLimitSettled,
} from '../../util/concurrency.js';
import { isUniqueViolation } from '../../util/dbError.js';
import { BoundedTtlMap } from '../../util/boundedTtlMap.js';
import {
abuseKey,
cleanEmail,
@@ -57,6 +58,7 @@ import {
} from '../metering/enforcement.js';
import { carriesClosedAppData } from '../permission/appDataScopes';
import { MANAGE_PERM_PREFIX } from '../permission/consts';
import { isOwnHomePath } from '../permission/fsPathPermission.js';
import { PermissionUtil } from '../permission/permissionUtil.js';
import { SubscriptionCache } from '../events/subscriptionCache.js';
import { KV_SHARE_ACCESS_LEVELS, kvShareRoot } from '../kvShare/kvShares.js';
@@ -292,6 +294,10 @@ export const modeFromEntryPermission = (permission: string): string | null => {
*/
const RETIRE_CHUNK_SIZE = 100;
/** Owner usernames kept per process, so the home check rarely reads a user. */
const USERNAME_CACHE_MAX_ENTRIES = 10_000;
const USERNAME_CACHE_TTL_MS = 60_000;
/** How long after an owner shares the retire reads stay on the primary. */
const RECENT_SHARE_WINDOW_MS = 30_000;
@@ -406,6 +412,11 @@ export class ShareService extends PuterService {
RECENT_SHARE_WINDOW_MS,
);
readonly #ownerLookups = new Map<string, Promise<boolean>>();
readonly #usernames = new BoundedTtlMap<number, string | null>({
maxEntries: USERNAME_CACHE_MAX_ENTRIES,
ttlMs: USERNAME_CACHE_TTL_MS,
});
readonly #usernameLookups = new Map<number, Promise<string | null>>();
/**
* FS mutations only notify the owner, leaving a recipient's open window
@@ -450,6 +461,10 @@ export class ShareService extends PuterService {
fromUserId?: number;
};
if (!node?.uuid) return;
const handedOver =
typeof fromUserId === 'number' && fromUserId !== node.userId;
// The subtree's share rows were re-pointed at its new owner.
if (handedOver) this.#ownerShared(node.userId);
const notify = this.#fanOutMove(node, fromPath).catch(() => {
// A stale window is better than a failed move.
});
@@ -458,11 +473,7 @@ export class ShareService extends PuterService {
// entry into its new owner's tree — leaving the new owner with
// recipients they never agreed to. Awaited alongside the notify so
// an `emitAndWait` caller sees both.
if (typeof fromUserId !== 'number' || fromUserId === node.userId) {
return notify;
}
// The subtree's share rows were re-pointed at its new owner.
this.#ownerShared(node.userId);
if (!handedOver) return notify;
return Promise.all([
notify,
this.onEntryOwnerChanged(node).catch((err: unknown): void => {
@@ -873,6 +884,15 @@ export class ShareService extends PuterService {
rows: ShareIndexRow[];
nodesById: Map<number, FSEntry>;
}> {
// Under another user's home an ancestor may belong to someone else.
const username = await this.#usernameOf(entry.userId);
if (
username !== null &&
isOwnHomePath(realPath, username) &&
!(await this.#ownerHasShares(entry.userId))
) {
return { rows: [], nodesById: new Map() };
}
const ancestors = await this.stores.fsEntry.getAncestors(
pathPosix.dirname(realPath),
);
@@ -894,6 +914,29 @@ export class ShareService extends PuterService {
return { rows: [...direct, ...viaGroup], nodesById };
}
/** `userId`'s username, or null for none; cached per process. */
#usernameOf(userId: number): Promise<string | null> {
const cached = this.#usernames.get(userId);
if (cached !== undefined) return Promise.resolve(cached);
const inFlight = this.#usernameLookups.get(userId);
if (inFlight) return inFlight;
const lookup = (async () => {
try {
const user = await this.stores.user.getById(userId);
const username = user?.username || null;
this.#usernames.set(userId, username);
return username;
} catch {
// Unknown reads as someone else's home: the full lookup runs.
return null;
} finally {
this.#usernameLookups.delete(userId);
}
})();
this.#usernameLookups.set(userId, lookup);
return lookup;
}
/** Whether `ownerUserId` has anything shared, cached per process. */
async #ownerHasShares(ownerUserId: number): Promise<boolean> {
const cached = this.#ownerShares.read(ownerUserId);
@@ -35,6 +35,7 @@ import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi }
import { makeActor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
import type { EmailNotifyValidateEvent } from '../../clients/event/types.js';
const BOOT_TIMEOUT_MS = 120_000;
const SETTLE_MS = 300;
@@ -49,6 +50,7 @@ interface SentEmail {
html: string;
/** The plain-text alternative, which carries its own links. */
text: string;
stream?: string;
}
const uniqueSuffix = (): string =>
@@ -125,12 +127,14 @@ describe('share email', () => {
subject: string;
html?: string;
text?: string;
stream?: string;
}) => {
sent.push({
to: options.to,
subject: options.subject,
html: options.html ?? '',
text: options.text ?? '',
stream: options.stream,
});
return null;
},
@@ -300,6 +304,7 @@ describe('share email', () => {
// account on any other address will not find the share.
expect(mail.html).toContain(invitee);
expect(mail.html).toContain('Create your free account');
expect(mail.stream).toBe('notify');
// The full origin, port included — a rebuilt protocol://domain link
// is dead on any self-host that doesn't run on the default port.
expect(mail.html).toContain(`href="${env.origin}"`);
@@ -370,6 +375,7 @@ describe('share email', () => {
)}&user_uuid=${encodeURIComponent(recipient.uuid)}`,
);
expect(mail.html).toContain(recipient.username);
expect(mail.stream).toBe('notify');
// A second share to the same pair inside the window is one more thing to
// look at, not one more thing to be interrupted by.
@@ -754,6 +760,168 @@ describe('share email', () => {
);
});
describe('email.notify.validate', () => {
type Listener = (key: string, data: EmailNotifyValidateEvent) => void;
let listener: Listener | null = null;
const listen = (fn: Listener) => {
listener = fn;
env.server.clients.event.on('email.notify.validate', fn);
};
const unlisten = () => {
if (listener) {
env.server.clients.event.off('email.notify.validate', listener);
}
listener = null;
};
afterEach(unlisten);
/** The events that name `to`; other tests' late sends name others. */
const about = (seen: EmailNotifyValidateEvent[], to: string) =>
seen.filter((event) => event.recipients.some((r) => r.to === to));
const decline = (data: EmailNotifyValidateEvent, to: string) => {
for (const recipient of data.recipients) {
if (recipient.to === to) recipient.allow = false;
}
};
it('skips only the account holder a listener declines', async () => {
const owner = env.users.user;
const seen: EmailNotifyValidateEvent[] = [];
let declined = '';
listen((_key, data) => {
seen.push(structuredClone(data));
decline(data, declined);
});
// Signed up with the listener on: account mail is not asked about.
const skippedHolder = await signUpAndConfirm(uninvitedAddress());
const keptHolder = await signUpAndConfirm(uninvitedAddress());
declined = skippedHolder.email;
sent = [];
const file = await makeFile(owner, 'notify-holder');
const res = await post('/share', owner.token, {
recipients: [skippedHolder.email, keptHolder.email],
items: [{ uid: file.uid }],
mode: 'read',
});
expect(res.status).toBe(200);
await waitForMail({ to: keptHolder.email });
await sleep(SETTLE_MS);
expect(mailTo(skippedHolder.email)).toHaveLength(0);
// Each holder is asked alone, once, when their digest is sent.
for (const to of [skippedHolder.email, keptHolder.email]) {
expect(about(seen, to)).toEqual([
{
kind: 'file_shared_with_you',
recipients: [{ to, allow: true }],
},
]);
}
// Only the mail is skipped; the share itself stands.
await eventually('the share to be listed anyway', async () =>
(await sharedWithMe(skippedHolder.token)).some(
(item) => item.uid_entry === file.uid,
),
);
});
it('asks once for every invite address, and skips only the declined ones, spending none of their budget', async () => {
const owner = env.users.user;
const skippedInvitee = uninvitedAddress();
const keptInvitee = uninvitedAddress();
const otherKept = uninvitedAddress();
const seen: EmailNotifyValidateEvent[] = [];
listen((_key, data) => {
seen.push(structuredClone(data));
decline(data, skippedInvitee);
});
const first = await makeFile(owner, 'notify-invite-1');
await post('/share', owner.token, {
recipients: [skippedInvitee, keptInvitee, otherKept],
items: [{ uid: first.uid }],
mode: 'read',
});
await waitForMail({ to: keptInvitee });
await waitForMail({ to: otherKept });
await sleep(SETTLE_MS);
expect(mailTo(skippedInvitee)).toHaveLength(0);
// One event carries all three; the declined one never reaches a send.
expect(about(seen, skippedInvitee)).toEqual([
{
kind: 'file_shared_invite',
recipients: [skippedInvitee, keptInvitee, otherKept].map(
(to) => ({ to, allow: true }),
),
},
]);
// The others are asked again, alone, when their digest is sent.
for (const to of [keptInvitee, otherKept]) {
expect(about(seen, to).slice(1)).toEqual([
{
kind: 'file_shared_invite',
recipients: [{ to, allow: true }],
},
]);
}
// The pair window was never claimed, so the next share mails.
unlisten();
const second = await makeFile(owner, 'notify-invite-2');
await shareWith(owner, skippedInvitee, [{ uid: second.uid }]);
const mail = await waitForMail({ to: skippedInvitee });
expect(mail.html).toContain(second.name);
});
it('still emails when a listener throws', async () => {
const owner = env.users.user;
const invitee = uninvitedAddress();
vi.spyOn(console, 'error').mockImplementation(() => {});
const asked: string[] = [];
listen((_key, data) => {
for (const { to } of data.recipients) {
if (to === invitee) asked.push(to);
}
throw new Error('listener failed');
});
const file = await makeFile(owner, 'notify-throw');
await shareWith(owner, invitee, [{ uid: file.uid }]);
const mail = await waitForMail({ to: invitee });
expect(mail.html).toContain(file.name);
// Once before the invite budget is claimed, once at send.
expect(asked).toEqual([invitee, invitee]);
});
it('asks again when the digest is sent, and honours that answer', async () => {
const owner = env.users.user;
const invitee = uninvitedAddress();
const answers: boolean[] = [];
listen((_key, data) => {
for (const recipient of data.recipients) {
if (recipient.to !== invitee) continue;
// Allowed when queued, declined by the time it is sent.
recipient.allow = answers.length === 0;
answers.push(recipient.allow);
}
});
const file = await makeFile(owner, 'notify-at-send');
await shareWith(owner, invitee, [{ uid: file.uid }]);
await eventually(
'the digest to be asked about',
async () => answers.length === 2,
);
await sleep(SETTLE_MS);
expect(answers).toEqual([true, false]);
expect(mailTo(invitee)).toHaveLength(0);
});
});
it('sends nothing to a recipient who has blocked the sender', async () => {
const owner = env.users.user;
const recipient = await signUpAndConfirm(uninvitedAddress());
@@ -810,12 +978,14 @@ describe('share email digest durability', () => {
subject: string;
html?: string;
text?: string;
stream?: string;
}) => {
sent.push({
to: options.to,
subject: options.subject,
html: options.html ?? '',
text: options.text ?? '',
stream: options.stream,
});
return null;
},
+195 -15
View File
@@ -20,6 +20,7 @@
import type { Request } from 'express';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { PuterServer } from '../../server.js';
import type { UserRow } from '../../stores/user/UserStore.js';
import { setupTestServer } from '../../testUtil.js';
let server: PuterServer;
@@ -193,20 +194,21 @@ describe('SignupService.runSignupGate', () => {
describe('SignupService.signup', () => {
it('provisions a form signup and announces it with the form-only fields', async () => {
const username = `su_${slug()}`;
const { result: user, seen } = await capture(
'puter.signup.success',
() =>
server.services.signup.signup({
req,
body: {
username,
email: `${username}@example.com`,
password: 'correct horse battery',
},
isTemp: false,
fingerprint: null,
bonusCode: null,
}),
const {
result: { user },
seen,
} = await capture('puter.signup.success', () =>
server.services.signup.signup({
req,
body: {
username,
email: `${username}@example.com`,
password: 'correct horse battery',
},
isTemp: false,
fingerprint: null,
bonusCode: null,
}),
);
expect(user.username).toBe(username);
@@ -238,7 +240,7 @@ describe('SignupService email gate last', () => {
server.clients.event.on('puter.signup.validate', require);
const username = `egl_${slug()}`;
try {
const user = await server.services.signup.signup({
const { user } = await server.services.signup.signup({
req,
body: {
username,
@@ -284,6 +286,7 @@ describe('SignupService email gate last', () => {
`${user.username}@example.com`,
'email_verification_code',
{ code: stored!.email_confirm_code },
{ stream: 'signup' },
);
} finally {
restore();
@@ -306,6 +309,183 @@ describe('SignupService email gate last', () => {
});
});
describe('SignupService confirmation stream', () => {
/** Signs up with a listener setting `email_stream`, then runs `check`. */
const signupWithStream = async (
emailStream: unknown,
check: (signedUp: {
stored: UserRow;
send: ReturnType<typeof vi.fn>;
}) => Promise<void>,
extra = {},
) => {
const send = vi.fn(async () => true);
const clients = server.clients as unknown as { email: unknown };
const realEmail = clients.email;
clients.email = { send };
const name = (_k: string, data: unknown) => {
Object.assign(data as Record<string, unknown>, {
email_stream: emailStream,
...extra,
});
};
server.clients.event.on('puter.signup.validate', name);
const username = `str_${slug()}`;
try {
const { user } = await server.services.signup.signup({
req,
body: {
username,
email: `${username}@example.com`,
password: 'correct horse battery',
},
isTemp: false,
fingerprint: null,
bonusCode: null,
});
const stored = await server.stores.user.getById(user.id, {
force: true,
});
await check({ stored: stored!, send });
} finally {
server.clients.event.off('puter.signup.validate', name);
clients.email = realEmail;
}
};
it('sends the confirmation on the stream a listener names and remembers it', async () => {
await signupWithStream('review:2', async ({ stored, send }) => {
expect(send).toHaveBeenCalledWith(
stored.email,
'email_verification_code',
{ code: stored.email_confirm_code },
{ stream: 'review:2' },
);
expect(stored.metadata?.emailStream).toBe('review:2');
expect(server.services.signup.confirmationStream(stored)).toBe(
'review:2',
);
await server.services.signup.sendConfirmationEmail(
stored,
'resend',
{ code: '111111' },
);
expect(send).toHaveBeenLastCalledWith(
stored.email,
'email_verification_code',
{ code: '111111' },
{ stream: 'review:2' },
);
});
});
it('ignores a stream name that is not a short lowercase token, and says so', async () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
try {
for (const bad of [
'Upper',
'has space',
7,
'-lead',
'x'.repeat(65),
]) {
await signupWithStream(bad, async ({ stored, send }) => {
expect(send).toHaveBeenCalledWith(
stored.email,
'email_verification_code',
{ code: stored.email_confirm_code },
{ stream: 'signup' },
);
expect(stored.metadata?.emailStream).toBeUndefined();
});
expect(warn).toHaveBeenCalledWith(
'[signup] ignored email_stream:',
bad,
);
}
} finally {
warn.mockRestore();
}
});
it('records the gate order and the stream together', async () => {
await signupWithStream(
'held',
async ({ stored, send }) => {
expect(stored.metadata).toMatchObject({
emailGateLast: true,
emailStream: 'held',
});
expect(send).not.toHaveBeenCalled();
await server.stores.user.update(stored.id, {
requires_phone_verification: 0,
});
await server.services.signup.sendDeferredConfirmationEmail(
stored.id,
);
expect(send).toHaveBeenCalledWith(
stored.email,
'email_verification_code',
{ code: stored.email_confirm_code },
{ stream: 'held' },
);
},
{ requires_phone_verification: true, email_gate_last: true },
);
});
it('drops the stream a claimed row held when the signup names none', async () => {
const username = `clm_${slug()}`;
const email = `${username}@example.com`;
const placeholder = await server.stores.user.create({
username: `ph_${slug()}`,
uuid: crypto.randomUUID(),
password: null,
email,
});
await server.stores.user.updateMetadata(placeholder.id, {
emailStream: 'review',
});
const send = vi.fn(async () => true);
const clients = server.clients as unknown as { email: unknown };
const realEmail = clients.email;
clients.email = { send };
try {
const { user } = await server.services.signup.signup({
req,
body: { username, email, password: 'correct horse battery' },
isTemp: false,
fingerprint: null,
bonusCode: null,
});
expect(user.id).toBe(placeholder.id);
expect(send).toHaveBeenCalledWith(
email,
'email_verification_code',
expect.anything(),
{ stream: 'signup' },
);
const stored = await server.stores.user.getById(user.id, {
force: true,
});
expect(server.services.signup.confirmationStream(stored!)).toBe(
'signup',
);
} finally {
clients.email = realEmail;
}
});
it('falls back to the signup stream for an account without one', () => {
expect(
server.services.signup.confirmationStream({
metadata: {},
} as unknown as UserRow),
).toBe('signup');
});
});
describe('SignupService.announceSignup', () => {
it('leaves out the form-only fields for a source that has none', async () => {
const username = `an_${slug()}`;
+76 -17
View File
@@ -41,6 +41,21 @@ import {
} from '../../util/userProvisioning.js';
import { PuterService } from '../types.js';
const EMAIL_STREAM_NAME = /^[a-z0-9][a-z0-9:_-]{0,63}$/;
const emailStreamName = (value: unknown): string | null => {
if (value == null) return null;
if (typeof value === 'string' && EMAIL_STREAM_NAME.test(value)) {
return value;
}
console.warn('[signup] ignored email_stream:', value);
return null;
};
/** Where a confirmation email was sent from; names its failure alarm. */
export type ConfirmationEmailStage =
'signup' | 'save-account' | 'resend' | 'deferred';
/** What `puter.signup.validate` and the bonus hooks decided about a signup. */
export type SignupGateOutcome =
| {
@@ -51,6 +66,8 @@ export type SignupGateOutcome =
requiresCardVerification: boolean;
/** Card and phone come before email (see `isEmailGateLast`). */
emailGateLast: boolean;
/** Stream the confirmation email is sent on; `null` for the default. */
emailStream: string | null;
}
| { verdict: 'bonus_code_invalid' }
| {
@@ -85,6 +102,12 @@ export interface SignupInput {
bonusCode: string | null;
}
export interface SignupResult {
user: UserRow;
/** Whether the transport accepted a confirmation email sent at signup. */
confirmationEmailSent?: boolean;
}
/** The trusted client address; `req.ip` honors `trust proxy`. */
export const signupClientIp = (req: Request): string | null =>
req.ip || req.socket?.remoteAddress || null;
@@ -222,6 +245,7 @@ export class SignupService extends PuterService {
data: input.data,
ip: clientIp,
user_agent: req?.headers?.['user-agent'] ?? null,
tls_fingerprint: req?.tlsFingerprint ?? null,
email,
// The form `email.validate` was given, so the abuse harness can
// find the verdict that hook cached for this address.
@@ -238,6 +262,7 @@ export class SignupService extends PuterService {
email_gate_last: false,
message: null as string | null,
code: null as string | null,
email_stream: null as string | null,
...(input.fingerprint !== undefined ? { fingerprint } : {}),
// Signup-time reputation, persisted on the user row.
reputation: null as number | null,
@@ -314,6 +339,7 @@ export class SignupService extends PuterService {
emailGateLast:
Boolean(validateEvent.email_gate_last) &&
(requiresPhoneVerification || requiresCardVerification),
emailStream: emailStreamName(validateEvent.email_stream),
};
}
@@ -321,7 +347,7 @@ export class SignupService extends PuterService {
* A form or temp signup. Returns the account to sign in; throws the 4xx a
* refused signup answers with.
*/
async signup(input: SignupInput): Promise<UserRow> {
async signup(input: SignupInput): Promise<SignupResult> {
const { req, body, isTemp, fingerprint, bonusCode } = input;
// Runs before the duplicate checks so a disabled endpoint doesn't
@@ -534,6 +560,7 @@ export class SignupService extends PuterService {
// column; the raw chain stays in `audit_metadata`.
signup_ip_forwarded: clientIp,
signup_user_agent: req.headers?.['user-agent'] ?? null,
signup_tls_fingerprint: req.tlsFingerprint ?? null,
signup_origin:
(req.headers?.origin as string | null) ?? null,
signup_server: this.config.serverId,
@@ -572,10 +599,25 @@ export class SignupService extends PuterService {
}
const emailGateLast = !isTemp && gate.emailGateLast;
if (emailGateLast) user = await this.#markEmailGateLast(user);
// A claimed row may hold the stream of an earlier signup.
const storedStream = user.metadata?.emailStream ?? null;
const metadataPatch = {
...(emailGateLast ? { emailGateLast: true } : {}),
...(gate.emailStream !== storedStream
? { emailStream: gate.emailStream }
: {}),
};
if (Object.keys(metadataPatch).length > 0) {
await this.stores.user.updateMetadata(user.id, metadataPatch);
user = {
...user,
metadata: { ...user.metadata, ...metadataPatch },
};
}
let confirmationEmailSent: boolean | undefined;
if (!isTemp && user.requires_email_confirmation && !emailGateLast) {
await this.sendConfirmationEmail(
confirmationEmailSent = await this.sendConfirmationEmail(
user,
'signup',
(body.send_confirmation_code ?? true)
@@ -592,7 +634,12 @@ export class SignupService extends PuterService {
isTemp: user.password === null && user.email === null,
saveAccount: !isTemp,
});
return user;
return {
user,
...(confirmationEmailSent !== undefined
? { confirmationEmailSent }
: {}),
};
}
/**
@@ -649,10 +696,10 @@ export class SignupService extends PuterService {
}
}
/** Record that this account clears card and phone before email. */
async #markEmailGateLast(user: UserRow): Promise<UserRow> {
await this.stores.user.updateMetadata(user.id, { emailGateLast: true });
return { ...user, metadata: { ...user.metadata, emailGateLast: true } };
/** The stream this account's confirmation emails are sent on. */
confirmationStream(user: UserRow): string {
const stored = user.metadata?.emailStream;
return typeof stored === 'string' ? stored : 'signup';
}
/**
@@ -685,13 +732,15 @@ export class SignupService extends PuterService {
/**
* Send the confirmation code (or link) for the account's address. A lost
* message is alarmed rather than thrown: the account exists either way.
* Resolves to whether the transport accepted the message.
*/
async sendConfirmationEmail(
user: UserRow,
stage: 'signup' | 'resend' | 'deferred',
stage: ConfirmationEmailStage,
message: { code: string } | { token: string },
): Promise<void> {
if (!this.clients.email || !user.email) return;
): Promise<boolean> {
if (!this.clients.email || !user.email) return false;
const stream = this.confirmationStream(user);
try {
const sent =
'code' in message
@@ -699,6 +748,7 @@ export class SignupService extends PuterService {
user.email,
'email_verification_code',
{ code: message.code },
{ stream },
)
: await this.clients.email.send(
user.email,
@@ -706,11 +756,17 @@ export class SignupService extends PuterService {
{
link: `${this.config.origin ?? ''}/confirm-email-by-token?token=${message.token}&user_uuid=${user.uuid}`,
},
{ stream },
);
// `null` = dropped for want of a transport; silent otherwise.
if (sent === null) this.#confirmationEmailFailed(stage, user, null);
if (sent === null) {
this.#confirmationEmailFailed(stage, user, null);
return false;
}
return true;
} catch (e) {
this.#confirmationEmailFailed(stage, user, e);
return false;
}
}
@@ -862,7 +918,7 @@ export class SignupService extends PuterService {
* phone/card gate is outstanding either.
*/
#confirmationEmailFailed(
stage: 'signup' | 'resend' | 'deferred',
stage: ConfirmationEmailStage,
user: UserRow,
cause: unknown,
): void {
@@ -873,10 +929,13 @@ export class SignupService extends PuterService {
: cause === null
? 'no transport configured (message dropped)'
: String(cause);
console.warn(
`[${stage === 'resend' ? 'send-confirm-email' : 'signup'}] ` +
`confirmation email not delivered: ${detail}`,
);
const route =
stage === 'resend'
? 'send-confirm-email'
: stage === 'save-account'
? stage
: 'signup';
console.warn(`[${route}] confirmation email not delivered: ${detail}`);
// Best-effort: failing to alarm must not fail the signup.
try {
this.clients.alarm?.create(
@@ -219,13 +219,13 @@ describe('handing a delivery out', () => {
await store.leaseToSocket(subId, entryId);
await expect(
store.takeFromSocket(subId, entryId),
store.takeFromSocket(subId, entryId, null),
).resolves.toMatchObject({
entryId,
socketAttempts: 1,
});
// A second decline for the same lease changes nothing.
await expect(store.takeFromSocket(subId, entryId)).resolves.toBeNull();
await expect(store.takeFromSocket(subId, entryId, null)).resolves.toBeNull();
// Held past the socket lease: the worker has it now.
vi.useFakeTimers({ toFake: ['Date'] });
@@ -241,27 +241,77 @@ describe('handing a delivery out', () => {
vi.useFakeTimers({ toFake: ['Date'] });
vi.setSystemTime(Date.now() + EVENTS_SOCKET_LEASE_MS + 1_000);
await expect(store.claim(subId)).resolves.toMatchObject({ entryId });
await expect(store.takeFromSocket(subId, entryId)).resolves.toBeNull();
await expect(store.takeFromSocket(subId, entryId, null)).resolves.toBeNull();
await store.settle(subId, entryId);
await expect(store.takeFromSocket(subId, entryId)).resolves.toBeNull();
await expect(store.takeFromSocket(subId, entryId, null)).resolves.toBeNull();
});
it('takes a decline only from the region the current attempt went to', async () => {
const { entryId } = await store.enqueue(subId, event('a'));
await store.claim(subId);
await store.leaseToSocket(subId, entryId, { region: 'east' });
await expect(
store.takeFromSocket(subId, entryId, null),
).resolves.toBeNull();
await expect(
store.takeFromSocket(subId, entryId, 'south'),
).resolves.toBeNull();
await expect(
store.takeFromSocket(subId, entryId, 'east'),
).resolves.toMatchObject({ entryId, triedRegions: ['east'] });
});
it('leaves a later attempt alone when an earlier region declines again', async () => {
const { entryId } = await store.enqueue(subId, event('a'));
await store.claim(subId);
await store.leaseToSocket(subId, entryId, { region: 'east' });
await store.takeFromSocket(subId, entryId, 'east');
await store.leaseToSocket(subId, entryId, {
region: 'south',
reoffer: true,
});
// East's second client, declining the attempt east already gave back.
await expect(
store.takeFromSocket(subId, entryId, 'east'),
).resolves.toBeNull();
await expect(
store.takeFromSocket(subId, entryId, 'south'),
).resolves.toMatchObject({ triedRegions: ['east', 'south'] });
});
it('keeps a re-offer out of reach of a decline that reads only the first lease', async () => {
const { entryId } = await store.enqueue(subId, event('a'));
await store.claim(subId);
await store.leaseToSocket(subId, entryId, { region: 'east' });
await store.takeFromSocket(subId, entryId, 'east');
await store.leaseToSocket(subId, entryId, {
region: 'south',
reoffer: true,
});
// What a server without re-offers does with a decline: release the
// lease it finds stamped as `socketLeaseUntil`.
const raw = await redis.hget(`ev:q:{${subId}}`, entryId);
const { socketLeaseUntil } = JSON.parse(String(raw));
const moved = await redis.zscore(`ev:ql:{${subId}}`, entryId);
expect(Number(moved)).not.toBe(socketLeaseUntil);
});
it('ignores a decline for a delivery never handed to a socket', async () => {
const { entryId } = await store.enqueue(subId, event('a'));
await store.claim(subId);
await expect(store.takeFromSocket(subId, entryId)).resolves.toBeNull();
await expect(store.takeFromSocket(subId, entryId, null)).resolves.toBeNull();
});
it('records remote regions tried separately from socket attempts', async () => {
const { entryId } = await store.enqueue(subId, event('a'));
await store.claim(subId, { leaseMs: 1 });
await store.leaseToSocket(subId, entryId);
await expect(
store.recordRemoteAttempt(subId, entryId, 'east'),
).resolves.toEqual(['east']);
await store.leaseToSocket(subId, entryId, { region: 'east' });
vi.useFakeTimers({ toFake: ['Date'] });
vi.setSystemTime(Date.now() + EVENTS_SOCKET_LEASE_MS + 1_000);
@@ -275,8 +325,7 @@ describe('handing a delivery out', () => {
it('gives a delivery a fresh attempt budget on request', async () => {
const { entryId } = await store.enqueue(subId, event('a'));
await store.claim(subId, { leaseMs: 1 });
await store.leaseToSocket(subId, entryId);
await store.recordRemoteAttempt(subId, entryId, 'east');
await store.leaseToSocket(subId, entryId, { region: 'east' });
await store.resetSocketAttempts(subId, entryId);
@@ -552,7 +601,7 @@ describe('finding the work', () => {
const { entryId } = await store.enqueue(subId, event('a'));
await store.claim(subId);
await store.leaseToSocket(subId, entryId);
await store.takeFromSocket(subId, entryId);
await store.takeFromSocket(subId, entryId, null);
await store.settle(subId, entryId);
await store.enqueue(subId, event('b'));
await store.purge(subId);
Loaded 100 of 198 files, more files were not shown because too many files have changed in this diff. Show more